Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations

# Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Source: [https://arxiv.org/html/2609.10866](https://arxiv.org/html/2609.10866)
Tong LiEmail:[tli31@CougarNet\.UH\.EDU](mailto:[email protected])Affiliation:Department of Industrial and Systems Engineering, University of Houston, 4302 University Dr, Houston, 77004, TX, USASaunak Kumar PandaEmail:[spanda@CougarNet\.UH\.EDU](mailto:[email protected])Affiliation:Department of Decision and Information Science, University of Houston, 4302 University Dr, Houston, 77004, TX, USAYisha XiangEmail:[yxiang4@Central\.UH\.EDU](mailto:[email protected])Affiliation:Department of Industrial and Systems Engineering, University of Houston, 4302 University Dr, Houston, 77004, TX, USA

###### Abstract

Reinforcement learning \(RL\) agents deployed in real\-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety\-critical applications\. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards\. Existing certification methods, however, mainly focus on risk\-neutral objectives\. In this paper, we extend certification methods to risk\-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards underlpl\_\{p\}\-norm–bounded state adversarial perturbations \(1≤p<∞1\\leq p<\\infty\)\. By introducing aϕ\\phi\-divergence relaxation of the perturbation set, we formulate the risk\-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound\. We further propose an empirical method that improves certified lower bounds by selecting the training risk\-aversion parameterβ\\betaindependently of the risk level used during evaluation\. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk\-neutral training, risk\-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets\. Moreover, under both risk\-neutral and risk\-averse evaluation settings, increasing risk aversion during training leads to non\-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies\.

###### keywords

lower bound certificate, risk\-sensitive RL, state adversarial attack, stochastic observations

### 1Introduction

In real\-world deployments, reinforcement learning \(RL\) agents frequently observe perturbed states due to a variety of factors such as sensor noise, environmental uncertainty, and external interference\. Empirical evidence has shown that many state\-of\-the\-art RL algorithms are highly sensitive to minor changes in the input space, making the algorithms unreliable in real\-world scenarios\([Behzadan and Munir, 2017a](https://arxiv.org/html/2609.10866#bib.bib4);[Huang et al\., 2017](https://arxiv.org/html/2609.10866#bib.bib5);[Pattanaik et al\., 2017](https://arxiv.org/html/2609.10866#bib.bib6)\)\. Even small perturbations, such as minor shifts in observed state variables due to environmental disturbances, can significantly alter the agent’s behavior\([Zhang et al\., 2020](https://arxiv.org/html/2609.10866#bib.bib11)\)\. These vulnerabilities are especially critical in safety\-sensitive domains, including autonomous driving, healthcare systems, and robotic control, where decision failures may lead to serious consequences\([Shalev\-Shwartz et al\., 2016](https://arxiv.org/html/2609.10866#bib.bib7);[Sallab et al\., 2017](https://arxiv.org/html/2609.10866#bib.bib8);[Liang et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib9);[He et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib10)\)\.

To mitigate the impact of adversarial perturbations in state observations, several empirical defense mechanisms have been developed\. Two widely used approaches are adversarial training and regularization\-based methods\. Adversarial training approaches can significantly enhance policy robustness by training agents on perturbed state observations during the learning process\([Pattanaik et al\., 2017](https://arxiv.org/html/2609.10866#bib.bib6);[Zhang et al\., 2020](https://arxiv.org/html/2609.10866#bib.bib11);[Kos and Song, 2017](https://arxiv.org/html/2609.10866#bib.bib3);[Behzadan and Munir, 2017b](https://arxiv.org/html/2609.10866#bib.bib16)\)\. Regularization\-based methods typically incorporate various forms of constraints into the RL optimization objective to improve the policy robustness\([Shen et al\., 2020](https://arxiv.org/html/2609.10866#bib.bib12);[Zhang et al\., 2020](https://arxiv.org/html/2609.10866#bib.bib11)\)\.

While these empirical defense methods have demonstrated improved robustness against adversarial perturbations, several adaptive attacks have been proposed against these defense strategies\. It is important to provide a theoretical guarantee \(e\.g\., lower bound\) for a trained policy to disrupt the repeated games between attackers and defenders, which is referred to asrobustness certification\([Wu et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib1);[Kumar et al\., 2021](https://arxiv.org/html/2609.10866#bib.bib13);[Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\)\. Robustness certification was initially developed in the context of classification tasks, aiming to provide a certified lower bound on classification accuracy under bounded adversarial perturbations\([Wu et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib1);[Kumar et al\., 2021](https://arxiv.org/html/2609.10866#bib.bib13)\)\. However, certifying robustness in RL involves sequential decision\-making, making it substantially more challenging than the one\-step prediction in classification tasks\. Building upon the foundational work on robustness certification in classification, several studies have extended this framework to RL by developing methods that certify the robustness of smoothed policies with respect to cumulative rewards\([Wu et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib1);[Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\)\.

Existing certification methods primarily focus on providing lower bounds for expected cumulative rewards, which represents a risk\-neutral perspective\. Safety\-critical applications often require risk\-averse evaluation that incorporates decision\-makers’ risk preferences into the objective function rather than optimizing expected performance\. In this paper, we consider the setting where the observed state is adversarially perturbed within a bounded region, while the underlying true state of the RL environment remains unchanged\. We extend robust certification torisk\-sensitiveobjectives, developing lower bound certificates for the expected utilities of cumulative rewards under bounded adversarial state perturbations\. Specifically, our formulation builds upon the ReCePS framework\([Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\)and extends it to support exponential utility for risk\-averse performance evaluation\. We formulate the lower bound certificate problem underlpl\_\{p\}\-norm bounded perturbations\(1≤p<∞\)\(1\\leq p<\\infty\)as a convex optimization problem through aϕ\\phi\-divergence\-based relaxation, and derive its dual formulation to obtain tractable and risk\-aware lower bounds on the expected exponential utility of smoothed policies\. This approach bridges the gap between risk\-neutral certification frameworks and the practical need for risk\-aware performance guarantees\.

We further develop a novel empirical method to improve the certified lower bounds of RL policies\. Our results show that risk\-averse training generally leads to policies with higher certified lower bounds than risk\-neutral training, particularly when the perturbation budget is large\. Moreover, under both risk\-neutral and risk\-averse evaluation settings, increasing the training\-time risk aversion initially enhances certified lower bound but eventually leads to degradation due to overly conservative policies\. This non\-monotonic relationship between the training\-time risk\-aversion level and the certified robustness highlights the importance of appropriately selecting the risk\-aversion parameter during training to enhance the provable guarantees of RL policies\. To the best of our knowledge, this is the first effort to enhance the robust certificates of RL policies, rather than just establishing theoretical lower bounds on their expected total rewards\.

The rest of the paper is organized as follows\. Section[2](https://arxiv.org/html/2609.10866#S2)reviews related work on adversarial perturbations and robustness certificates in RL\. Section[3](https://arxiv.org/html/2609.10866#S3)introduces the RL environment model for MDPs with adversarial state perturbations and formulates the optimization problem for lower bound certificates\. Section[4](https://arxiv.org/html/2609.10866#S4)presents the convex optimization approach for solving the lower bound certification problem and the algorithms for computing the lower bounds efficiently\. Section[5](https://arxiv.org/html/2609.10866#S5)presents experimental results on OpenAI environments and a practical machine replacement problem, demonstrating that appropriate risk\-averse training parameters can effectively improve lower bound performance under adversarial state perturbations\.

### 2Literature Review

In this section, we review relevant literature on RL robustness under adversarial perturbations\. We first summarize empirical defense mechanisms that improve policy robustness through training strategies\. We then discuss recent advances in robustness certification methods that provide theoretical guarantees for policy performance under perturbations\.

#### 2\.1Empirical Defense Mechanisms for Adversarial Attacks

Various empirical defense methods have been developed to enhance the robustness of RL agents against adversarial perturbations in state observations\. Adversarial training approaches are one of the most widely studied categories of defense mechanisms\.[Pattanaik et al\. \(2017\)](https://arxiv.org/html/2609.10866#bib.bib6)propose a robust adversarial RL algorithm that trains agents on adversarially perturbed state observations during the learning process, demonstrating improved robustness against test\-time perturbations\. In their study, an adversarial attack is modeled as a perturbation that increases the probability of the agent selecting the worst possible action\. The adversarial states are used during training to help the agent learn policies that are robust to observation noise and misperceptions\.[Behzadan and Munir \(2017a\)](https://arxiv.org/html/2609.10866#bib.bib4)propose an adversarial training strategy based on the deep Q\-learning \(DQN\) framework that periodically augments the experience replay buffer with adversarially perturbed states\. These perturbations are crafted to maximize the temporal\-difference \(TD\) error, encouraging the agent to learn policies that are more robust to observation noise and adversarial manipulation\.

Regularization\-based methods have also been developed for improving training stability and enhancing robustness against strong adversarial attacks\.[Zhang et al\. \(2020\)](https://arxiv.org/html/2609.10866#bib.bib11)develop various regularizers such as Kullback–Leibler \(KL\) divergence\-based,l2l\_\{2\}\-distance\-based and hinge\-loss\-based regularizers for classical RL algorithms\. All these three types of regularizers employ a minimax optimization structure solved via convex relaxation or Stochastic Gradient Langevin Dynamics\.

In addition to these methods,[Shen et al\. \(2020\)](https://arxiv.org/html/2609.10866#bib.bib12)propose the Smooth Regularized RL \(SR2L\) framework, where the regularizer is modeled as a local smoothness penalty measuring the divergence between policy outputs at neighboring states\. Depending on the policy type, this penalty is defined using either the squaredl2l\_\{2\}norm for deterministic policies or Jeffrey’s divergence for stochastic policies, encouraging the policy to produce similar actions for similar inputs\.

#### 2\.2Robustness Certification Methods

The development of robustness certification techniques for RL has gained significant attention as a means of providing theoretical guarantees on policy performance under adversarial perturbations\.[Lütjens et al\. \(2020\)](https://arxiv.org/html/2609.10866#bib.bib14)propose CARRL \(Certified Adversarially\-Robust Reinforcement Learning\) that adapts robustness certification techniques from computer vision domain to RL domain\. The method computes certified lower bounds onQQ\-values by propagating interval bounds through neural network layers and handling ReLU activations through linear approximations\. CARRL defines anϵ\\epsilon\-ball around each observed state and selects the action with the highest guaranteed worst\-caseQQ\-value within this bounded region, ensuring robust performance under adversarial perturbations\.

[Wu et al\. \(2022\)](https://arxiv.org/html/2609.10866#bib.bib1)propose the CROP \(Certifying Robust Policies for RL\) framework to provide robustness guarantees for deep RL against adversarial perturbations in state observations\. This framework addresses two certification criteria through distinct approaches\. For per\-state action certification, the method creates a smoothedQQ\-function by averagingQQ\-values over Gaussian\-perturbed input states, establishing Lipschitz continuity\. A certified radius is then computed by analyzing the margin between the top twoQQ\-values and applying inverse cumulative distribution functions to derive the maximum perturbation bound that guarantees unchanged action selection\. For cumulative reward certification, they use the global smoothing method that samples multiple noisy trajectories to compute statistical lower bounds on rewards, and local smoothing that uses adaptive search to systematically enumerate possible action changes under perturbations, yielding tighter deterministic lower bounds\.

Based on randomized smoothing techniques,[Kumar et al\. \(2021\)](https://arxiv.org/html/2609.10866#bib.bib13)develop a certification framework that provides robustness guarantees for cumulative rewards in RL deployment\. In contrast to the CROP framework\([Wu et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib1)\), which supports per\-state action certification and incorporates multiple smoothing strategies, their approach is specifically designed to certify the expected total reward using a single global smoothing mechanism\. To support this framework, they propose an adaptive variant of the Neyman–Pearson lemma that reduces general probabilistic adversaries to deterministic ones\. They further show that any such deterministic adversary can be transformed into a structured form that concentrates the entire perturbation budget on the first coordinate at the initial timestep\. This theoretical reduction enables direct certification through isometric Gaussian smoothing, yielding provable lower bounds on cumulative rewards under norm\-bounded adversarial attacks with adaptively allocated budgets\.

More recent advances have focused on developing more general and efficient certification frameworks\. Mu et al\. consider the problem of certifying the expected cumulative reward under adversarial perturbations and formulate an optimization model that seeks a convex reformulation based onff\-divergence measures to quantify distributional differences between original and perturbed trajectories\([Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\)\. Unlike previous approaches that rely on threshold\-based probability estimation or Lipschitz continuity bounds, ReCePS directly certifies cumulative reward lower bounds by solving dual optimization problems by adapting theff\-divergence relaxation of the infinite\-dimensional distribution space into finite\-dimensional tractable convex programs\. The method extends certification beyondl2l\_\{2\}\-norm tol1l\_\{1\}\-norm andl0l\_\{0\}\-norm constraints using appropriate divergence measures such as total variation distance and Rényi divergence\. This framework provides a unified approach for certifying robustness across different perturbation types while maintaining the theoretical guarantees of direct cumulative reward certification\.

Our work contributes to the growing literature on robust certification for RL policies by extending certification guarantees to risk\-sensitive settings\. Specifically, we establish certified lower bounds for the exponential utility of cumulative rewards under bounded adversarial state perturbations\. Experimental results show that risk\-averse training generally leads to improved certified lower bounds compared to risk\-neutral training, especially under stronger state adversarial perturbations, although excessive risk aversion can eventually degrade certification performance\.

### 3Problem Formulation

This section develops the robustness certification with adversarial state perturbations in the RL deployment\. We first introduce the MDP under both risk\-neutral and risk\-sensitive frameworks, along with the smoothed RL policies with Gaussian noise\. We further formulate lower bound certification problems that quantify the worst\-case performance guarantees of a given policy under bounded adversarial perturbations in the state space\.

#### 3\.1Preliminaries

MDP and risk\-sensitiveQQ\-function\.A finite\-horizon Markov decision process \(MDP\) is defined by the tuple\(𝒮,𝒜,P,r,T,γ\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,T,\\gamma\), where𝒮⊆ℝd\\mathcal\{S\}\\subseteq\\mathbb\{R\}^\{d\}is the continuous state space,𝒜\\mathcal\{A\}is the action space,P:𝒮×𝒜→Δ⁡\(𝒮\)P:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\Delta\(\\mathcal\{S\}\)is the transition kernel,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}is a bounded reward function,γ∈\[0,1\]\\gamma\\in\[0,1\]is the discount factor, andT<∞T<\\inftyis the finite time horizon\. Policyπ∈Π\\pi\\in\\Piis a mapping from the state space𝒮\\mathcal\{S\}to a probability distribution over𝒜\\mathcal\{A\}, whereΠ\\Piis the set of all possible policies\. At each time stept∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, the agent observes the current statests\_\{t\}, selects an actionat∼π\(⋅\|st\)a\_\{t\}\\sim\\pi\(\\cdot\|s\_\{t\}\), receives a rewardr⁡\(st,at\)r\(s\_\{t\},a\_\{t\}\), and transitions to the next statest\+1∼P\(⋅\|st,at\)s\_\{t\+1\}\\sim P\(\\cdot\|s\_\{t\},a\_\{t\}\)\. In a risk\-neutral environment, the performance of a policy is typically evaluated by the expected cumulative reward𝔼π​\[∑t=1Tγt​r​\(st,at\)∣s1=s\]\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{1\}=s\\right\]\.

RL algorithms often rely on theQQ\-function \(or action\-value function\), defined at each time stept∈\{1,…,T\}t\\in\\\{1,\\ldots,T\\\}as:

Qtπ\(s,a\)=𝔼π\[∑k=tTγk−tr\(sk,ak\)∣st=s,at=a\]\.Q\_\{t\}^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\(s\_\{k\},a\_\{k\}\)\\mid s\_\{t\}=s,a\_\{t\}=a\\right\]\.\(1\)Theoptimal Q\-functionis defined byQt∗​\(s,a\)=maxπ∈Π⁡Qtπ​\(s,a\)Q\_\{t\}^\{\*\}\(s,a\)=\\max\_\{\\pi\\in\\Pi\}Q\_\{t\}^\{\\pi\}\(s,a\), which satisfies theBellman optimality equation:

Qt∗\(s,a\)=r\(s,a\)\+γ𝔼s′∼P\(⋅∣s,a\)\[maxa′∈𝒜Qt\+1∗\(s′,a′\)\],Q\_\{t\}^\{\*\}\(s,a\)=r\(s,a\)\+\\gamma\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\left\[\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}Q\_\{t\+1\}^\{\*\}\(s^\{\\prime\},a^\{\\prime\}\)\\right\],fort=1,…,T−1t=1,\\dots,T\-1\. The optimal policyπt∗\\pi^\{\*\}\_\{t\}is the greedy policy with respect toQt∗Q\_\{t\}^\{\*\}, i\.e\.,πt∗​\(s\)∈arg⁡maxa∈𝒜​Qt∗​\(s,a\)\\pi^\{\*\}\_\{t\}\(s\)\\in\\arg\\max\_\{a\\in\\mathcal\{A\}\}Q\_\{t\}^\{\*\}\(s,a\), for alls∈𝒮s\\in\\mathcal\{S\}andt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\.

To account for decision\-makers’ risk preferences, a risk\-sensitive RL objective is usually needed\. Among various risk\-sensitive\-RL modeling approaches, exponential utility of the cumulative reward is often used for its mathematical tractability and its interpretation as a robust control criterion under model uncertainty\([Jaquette, 1976](https://arxiv.org/html/2609.10866#bib.bib18);[Howard and Matheson, 1972](https://arxiv.org/html/2609.10866#bib.bib19);[Hansen and Sargent, 2001](https://arxiv.org/html/2609.10866#bib.bib23)\)\. For each state\-action pair\(s,a\)\(s,a\), the risk\-sensitiveQQ\-function at timettunder a given policyπ\\piis defined as

Qtπ\(s,a\):=1βlog\{𝔼π\[exp\(β∑k=tTγk−tr\(st,at\)\)\|s,a\]\},Q^\{\\pi\}\_\{t\}\(s,a\):=\\frac\{1\}\{\\beta\}\\log\\Big\\\{\\mathbb\{E\}\_\{\\pi\}\\Big\[\\exp\\Big\(\\beta\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\(s\_\{t\},a\_\{t\}\)\\Big\)\\Big\|s,a\\Big\]\\Big\\\},\(2\)whereβ≠0\\beta\\neq 0denotes the risk sensitivity parameter\. The Taylor expansion of the exponential utility function:

1β​log⁡𝔼⁡\[exp⁡\(β​X\)\]=𝔼⁡\[X\]\+β2​Var​\[X\]\+o⁡\(β\)\\frac\{1\}\{\\beta\}\\log\\mathbb\{E\}\[\\exp\(\\beta X\)\]=\\mathbb\{E\}\[X\]\+\\frac\{\\beta\}\{2\}\\mathrm\{Var\}\[X\]\+o\(\\beta\)indicates that the utility function incorporates both the mean and the variance of returns\. The sign ofβ\\betadetermines whether the variance term contributes positively or negatively: negativeβ\\betaimplies risk aversion preference, positiveβ\\betaindicates risk seeking attitude, and theQQ\-function \([2](https://arxiv.org/html/2609.10866#S3.E2)\) reduces to the standard risk\-neutral form \([1](https://arxiv.org/html/2609.10866#S3.E1)\) asβ→0\\beta\\to 0\. Larger\|β\|\|\\beta\|implies a stronger risk attitude and higher sensitivity to risk\.

Analogous to the risk\-neutral case, the optimal policyπt∗\\pi^\{\*\}\_\{t\}selects actions that maximize the risk\-sensitive value function, i\.e\.,πt∗​\(s\)=arg⁡maxa∈𝒜​Qt∗​\(s,a\)\\pi^\{\*\}\_\{t\}\(s\)=\\arg\\max\_\{a\\in\\mathcal\{A\}\}Q\_\{t\}^\{\*\}\(s,a\), for alls∈𝒮s\\in\\mathcal\{S\}andt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}, whereQt∗​\(s,a\)=maxπ∈Π⁡Qtπ​\(s,a\)Q\_\{t\}^\{\*\}\(s,a\)=\\max\_\{\\pi\\in\\Pi\}Q\_\{t\}^\{\\pi\}\(s,a\)represents the optimal risk\-sensitiveQQ\-function\. Throughout this paper, we focus on the risk\-averseQQ\-function corresponding to the case whereβ<0\\beta<0\.

Smoothed RL policy\.The smoothed policy is defined by injecting Gaussian noise into the input state and computing the action that maximizes the expectedQQ\-value over the perturbed inputs\. Specifically, for each statests\_\{t\}, a noise vectorΔt∼𝒩⁡\(0,σ2​Id\)\\Delta\_\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)is added, and the agent selects the actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}that maximizes the smoothedQQ\-function evaluated at the perturbed state\. The smoothed policyπ~\\tilde\{\\pi\}is defined as

π~​\(st\):=π⁡\(st\+Δt\),\\tilde\{\\pi\}\(s\_\{t\}\):=\\pi\(s\_\{t\}\+\\Delta\_\{t\}\),\(3\)and the correspondingQQ\-value with the smoothed policyπ~\\tilde\{\\pi\}can be expressed as the expectedQQ\-value under random input noise, and is given by

Qπ~​\(s,a\)=𝔼Δ​\[Qπ​\(s\+Δ,a\)\]\.Q^\{\\tilde\{\\pi\}\}\(s,a\)=\\mathbb\{E\}\_\{\\Delta\}\[Q^\{\\pi\}\(s\+\\Delta,a\)\]\.\(4\)This approach facilitates more stable policy behavior and supports certified robustness guarantees\.

#### 3\.2Lower bound certification

##### Risk\-sensitive RL with smoothed policy\.

We consider a finite\-horizon MDP with adversarial state perturbations\. In this setting, at each time steptt, the agent observes a perturbed statest\+δts\_\{t\}\+\\delta\_\{t\}, whereδt∈ℝd\\delta\_\{t\}\\in\\mathbb\{R\}^\{d\}is an additive perturbation introduced by an adversary\. We assume that the perturbation sequence to time horizonTT,δ=\(δ1,…,δT\)\\mathbb\{\\delta\}=\(\\delta\_\{1\},\\ldots,\\delta\_\{T\}\)is constrained inppnorm, such that‖δ‖p=\(∑t=1T‖δt‖pp\)1p≤ϵ\|\|\\delta\|\|\_\{p\}=\(\\sum\_\{t=1\}^\{T\}\\\|\\delta\_\{t\}\\\|\_\{p\}^\{p\}\)^\{\\frac\{1\}\{p\}\}\\leq\\epsilon, for some1≤p<∞1\\leq p<\\inftyand perturbation budgetϵ\>0\\epsilon\>0\. The set of admissible perturbations is given by

ℬϵ=\{δ∈ℝd×T:\(∑t=1T‖δt‖pp\)1p≤ϵ\}\.\\mathcal\{B\}^\{\\epsilon\}=\\left\\\{\\delta\\in\\mathbb\{R\}^\{d\\times T\}:\\left\(\\sum\_\{t=1\}^\{T\}\\\|\\delta\_\{t\}\\\|\_\{p\}^\{p\}\\right\)^\{\\frac\{1\}\{p\}\}\\leq\\epsilon\\right\\\}\.
In the context of RL with state adversarial perturbation, the action is sampled according toat∼π⁡\(st\+δt\)a\_\{t\}\\sim\\pi\(s\_\{t\}\+\\delta\_\{t\}\), that is, the action is chosen based on perturbed statest\+δt\{s\}\_\{t\}\+\\delta\_\{t\}\. After executingata\_\{t\}, the system transitions to the next statest\+1∼P\(⋅\|st,at\)s\_\{t\+1\}\\sim P\(\\cdot\|s\_\{t\},a\_\{t\}\), and the agent receives a rewardr⁡\(st,at\)r\(s\_\{t\},a\_\{t\}\)\.

The presence of perturbations can degrade the performance of the policy\([Everett et al\., 2022](https://arxiv.org/html/2609.10866#bib.bib24);[Kumar et al\., 2021](https://arxiv.org/html/2609.10866#bib.bib13)\)\. The objective of robustness certification is to evaluate how well a given policy performs under worst\-case perturbations\. In this paper, we consider a risk\-aversion objective and incorporate risk attitude through the exponential utility function, where the agent’s preference over returns is governed by a risk aversion levelβ<0\\beta<0\. Instead of directly evaluating the expected return, we formulate a risk\-sensitive objective that accounts for worst\-case perturbations by optimizing over all admissible perturbation sequences constrained by thelpl\_\{p\}\-norm:

minδ⁡1β​log⁡\(𝔼⁡\(exp⁡β⁡\(∑t=1Tγt​r​\(st,at\)\)\)\),s\.t\.​‖δ‖p≤ϵ\.\\min\_\{\\mathbb\{\\delta\}\}\\frac\{1\}\{\\beta\}\\log\\left\(\\mathbb\{E\}\\left\(\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\)\\right\)\\right\),\\quad\\text\{s\.t\. \}\|\|\\delta\|\|\_\{p\}\\leq\\epsilon\.\(5\)withat∼π⁡\(st\+δt\)a\_\{t\}\\sim\\pi\(s\_\{t\}\+\\delta\_\{t\}\)\.

Given a policyπ\\pi, the goal of robust certification is to certify a lower bound on the cumulative reward under all feasible perturbation sequencesδ∈ℬϵ\\delta\\in\\mathcal\{B\}^\{\\epsilon\}, thus providing robustness guarantees on policy performance in adversarial settings\.

To facilitate analytical tractability in the lower bound analysis of \([5](https://arxiv.org/html/2609.10866#S3.E5)\), we adopt the smoothed policyπ~\\tilde\{\\pi\}defined in \([3](https://arxiv.org/html/2609.10866#S3.E3)\), which is constructed via Gaussian noise injection\. Given the smoothed policyπ~\\tilde\{\\pi\}and initial states0s\_\{0\}, the lower bound certificate can be obtained by minimizing the exponential utility of the expected cumulative reward over thelpl\_\{p\}bounded adversarial perturbationsδ=\(δ1,…,δT\)\\delta=\(\\delta\_\{1\},\\ldots,\\delta\_\{T\}\):

minδ⁡1β​log⁡\(𝔼τ∼q⁡\(τ\)​\(exp⁡β⁡\(∑t=1Tγt​r​\(st,π~​\(st\+δt\)\)\)\)\),\\displaystyle\\min\_\{\\delta\}\\frac\{1\}\{\\beta\}\\log\\left\(\\mathbb\{E\}\_\{\\tau\\sim q\(\\tau\)\}\\left\(\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\tilde\{\\pi\}\(s\_\{t\}\+\\delta\_\{t\}\)\)\\right\)\\right\)\\right\),\(6\)s\.t\.‖δ‖p≤ϵ,\\displaystyle\\text\{s\.t\.\}\\qquad\|\|\\delta\|\|\_\{p\}\\leq\\epsilon,whereq⁡\(τ\)q\(\\tau\)denotes the joint distribution of the perturbed trajectoryτ=\(s1,a1,…,sT,aT\)\\tau=\(s\_\{1\},a\_\{1\},\\ldots,s\_\{T\},a\_\{T\}\)withat∼π~​\(st\+δt\)a\_\{t\}\\sim\\tilde\{\\pi\}\(s\_\{t\}\+\\delta\_\{t\}\)andst\+1∼P\(⋅\|st,at\)s\_\{t\+1\}\\sim P\(\\cdot\|s\_\{t\},a\_\{t\}\), for allt=1,…,T−1t=1,\\ldots,T\-1\.

##### Risk\-sensitive RL with stochastic observations\.

Solving the non\-convex optimization problems \([6](https://arxiv.org/html/2609.10866#S3.E6)\) is generally challenging, as the objective functions involve both adversarial perturbations and stochastic smoothing noise\. To address this, we adopt the approach proposed by[Mu et al\. \(2024\)](https://arxiv.org/html/2609.10866#bib.bib2)and[Kumar et al\. \(2021\)](https://arxiv.org/html/2609.10866#bib.bib13), which considers a more general MDP setting where the agent interacts with the environment through stochastic observations\. The essential idea of this approach is that the smoothed policyπ~​\(st\)\\tilde\{\\pi\}\(s\_\{t\}\)can be equivalently interpreted as acting on stochastic observations sampled from a distributionμ⁡\(st\)\\mu\(s\_\{t\}\)\. Rather than injecting Gaussian noiseΔt∼𝒩⁡\(0,σ2​Id\)\\Delta\_\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)into the state and evaluating the policy onst\+Δts\_\{t\}\+\\Delta\_\{t\}, the same effect can be represented by sampling an observationot∈𝒪⊂ℝdo\_\{t\}\\in\\mathcal\{O\}\\subset\\mathbb\{R\}^\{d\}fromμ\(⋅\|st\)\\mu\(\\cdot\|s\_\{t\}\), whereμ\(⋅\|st\)\\mu\(\\cdot\|s\_\{t\}\)represents the Gaussian distribution𝒩⁡\(st,σ2​Id\)\\mathcal\{N\}\(s\_\{t\},\\sigma^\{2\}I\_\{d\}\)and𝒪\\mathcal\{O\}denotes the continuous observation space\. The policyπ\\piis then defined on the observation space, and the agent selects actions by samplingat∼π⁡\(ot\)a\_\{t\}\\sim\\pi\(o\_\{t\}\)based on observationsot∼μ\(⋅\|st\)o\_\{t\}\\sim\\mu\(\\cdot\|s\_\{t\}\)\.

To build on this observation\-based formulation, we now rewrite the optimization problem in \([6](https://arxiv.org/html/2609.10866#S3.E6)\)\. In this setting, the smoothed policyπ~​\(st\+δt\)\\tilde\{\\pi\}\(s\_\{t\}\+\\delta\_\{t\}\)over the perturbed statest\+δts\_\{t\}\+\\delta\_\{t\}can be equivalently interpreted as sampling an actionata\_\{t\}according toπ⁡\(ot′\)\\pi\(o^\{\\prime\}\_\{t\}\), where the perturbed observationot′o^\{\\prime\}\_\{t\}follows the Gaussian distributionμ\(⋅\|st\+δt\)=𝒩\(st\+δt,σ2Id\)\\mu\(\\cdot\|s\_\{t\}\+\\delta\_\{t\}\)=\\mathcal\{N\}\(s\_\{t\}\+\\delta\_\{t\},\\sigma^\{2\}I\_\{d\}\)\. This leads to the following optimization formulation:

minq⁡\(τ\)∈𝒟ϵ⁡1β​log⁡\(𝔼τ∼q​\(exp⁡β⁡\(∑t=1Tγt​r​\(st,π⁡\(ot\)\)\)\)\),\\min\_\{q\(\\tau\)\\in\\mathcal\{D\}^\{\\epsilon\}\}\\frac\{1\}\{\\beta\}\\log\\left\(\\mathbb\{E\}\_\{\\tau\\sim q\}\\left\(\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\pi\(o\_\{t\}\)\)\\right\)\\right\)\\right\),\(7\)where the perturbation sequenceδ=\(δ1,…,δT\)\\delta=\(\\delta\_\{1\},\\ldots,\\delta\_\{T\}\)satisfies‖δ‖p≤ϵ\\\|\\delta\\\|\_\{p\}\\leq\\epsilon, and each observationoto\_\{t\}is sampled from the Gaussian distribution𝒩⁡\(st\+δt,σ2​Id\)\\mathcal\{N\}\(s\_\{t\}\+\\delta\_\{t\},\\sigma^\{2\}I\_\{d\}\), for allt=1,…,Tt=1,\\ldots,T\. The admissible perturbation set withpp\-norm bounded constraint \(1≤p<∞1\\leq p<\\infty\) is defined as:

𝒟ϵ:=\{q⁡\(τ\)∈\(𝒫⁡\(𝒮\)×𝒫⁡\(𝒪\)×𝒫⁡\(𝒜\)\)T:‖δ‖p≤ϵ\},\\mathcal\{D\}^\{\\epsilon\}:=\\left\\\{q\(\\tau\)\\in\\left\(\\mathcal\{P\}\(\\mathcal\{S\}\)\\times\\mathcal\{P\}\(\\mathcal\{O\}\)\\times\\mathcal\{P\}\(\\mathcal\{A\}\)\\right\)^\{T\}:\\\|\\delta\\\|\_\{p\}\\leq\\epsilon\\right\\\},\(8\)which represents the collection of joint distributions over trajectoryτ=\(s1,o1,a1,…,sT,oT,aT\)\\tau=\(s\_\{1\},o\_\{1\},a\_\{1\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\)\. Here,𝒫⁡\(𝒮\)\\mathcal\{P\}\(\\mathcal\{S\}\),𝒫⁡\(𝒪\)\\mathcal\{P\}\(\\mathcal\{O\}\), and𝒫⁡\(𝒜\)\\mathcal\{P\}\(\\mathcal\{A\}\)denote the sets of probability distributions over the state, observation, and action spaces, respectively\.

With a slight abuse of notation, we useτ\\tauto denote the full trajectory consisting of states, observations, and actions under adversarial perturbations, where each observation is sampled from a Gaussian distributionμ⁡\(ot∣st\+δt\)=𝒩⁡\(st\+δt,σ2​Id\)\\mu\(o\_\{t\}\\mid s\_\{t\}\+\\delta\_\{t\}\)=\\mathcal\{N\}\(s\_\{t\}\+\\delta\_\{t\},\\sigma^\{2\}I\_\{d\}\), and each action is selected according to the policyat∼π⁡\(ot\)a\_\{t\}\\sim\\pi\(o\_\{t\}\)\. Given an initial states0∈𝒮s\_\{0\}\\in\\mathcal\{S\}, and assumingo0∼μ⁡\(o0∣s0\+δ0\)o\_\{0\}\\sim\\mu\(o\_\{0\}\\mid s\_\{0\}\+\\delta\_\{0\}\)anda0∼π⁡\(a0∣o0\)a\_\{0\}\\sim\\pi\(a\_\{0\}\\mid o\_\{0\}\), the joint distribution over the trajectoryτ=\(s1,o1,a1,…,sT,oT,aT\)\\tau=\(s\_\{1\},o\_\{1\},a\_\{1\},\\dots,s\_\{T\},o\_\{T\},a\_\{T\}\)is

q⁡\(τ\)=∏t=1TP⁡\(st∣st−1,at−1\)⋅μ⁡\(ot∣st\+δt\)⋅π⁡\(at∣ot\)\.q\(\\tau\)=\\prod\_\{t=1\}^\{T\}P\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\\cdot\\mu\(o\_\{t\}\\mid s\_\{t\}\+\\delta\_\{t\}\)\\cdot\\pi\(a\_\{t\}\\mid o\_\{t\}\)\.\(9\)
Note that function1β​log⁡\(⋅\)\\frac\{1\}\{\\beta\}\\log\(\\cdot\)is decreasing ifβ<0\\beta<0\. The minimization problem in \([7](https://arxiv.org/html/2609.10866#S3.E7)\) can be expressed as maximization of the expected cumulative reward with the risk\-averse utility measure\. To facilitate solving problem \([7](https://arxiv.org/html/2609.10866#S3.E7)\), we reformulate it into the following standard form of convex optimization:

minq⁡\(τ\)∈𝒟ϵ−𝔼τ∼q​\(exp⁡β⁡\(∑t=1Tγt​r​\(st,π⁡\(ot\)\)\)\),β<0,\\min\_\{q\(\\tau\)\\in\\mathcal\{D\}^\{\\epsilon\}\}\-\\mathbb\{E\}\_\{\\tau\\sim q\}\\left\(\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\pi\(o\_\{t\}\)\)\\right\)\\right\),\\quad\\beta<0,\(10\)and the optimal value of \([7](https://arxiv.org/html/2609.10866#S3.E7)\) can be calculated by1β​log⁡\(−v⋆\)\\frac\{1\}\{\\beta\}\\log\(\-v^\{\\star\}\), wherev⋆v^\{\\star\}denotes the optimal solution of problem \([10](https://arxiv.org/html/2609.10866#S3.E10)\)\.

In the following section, we first derive a tractable reformulation of the optimization problem \([10](https://arxiv.org/html/2609.10866#S3.E10)\), and then present efficient algorithms for solving the lower\-bound certification problem\.

### 4Reformulation of the lower bound certification

In this section, we first reformulate the lower bound certification problem \([10](https://arxiv.org/html/2609.10866#S3.E10)\) into an equivalent optimization problem and derive its tractable form\. We then develop algorithms to estimate the certified lower bound under state adversarial perturbations during RL deployment\.

#### 4\.1Optimization problems of the lower bound certification

Convex relaxation\.Solving the optimization problem \([10](https://arxiv.org/html/2609.10866#S3.E10)\) directly over the admissible perturbation set𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}\([8](https://arxiv.org/html/2609.10866#S3.E8)\) is challenging due to its potential non\-convex property and infinite\-dimensional nature\. In this section, we consider a convex relaxation of𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}by using aϕ\\phi\-divergence\-based set𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}, which facilitates tractable optimization and satisfies𝒟ϵ⊆𝒟~ϵ′\\mathcal\{D\}^\{\\epsilon\}\\subseteq\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\.

Aϕ\\phi\-divergence between two probability distributionsqqandppis defined as:

Dϕ\(q∥p\)=∫ϕ\(d​qd​p\)dp,D\_\{\\phi\}\(q\\\|p\)=\\int\\phi\\left\(\\frac\{dq\}\{dp\}\\right\)dp,whereϕ:ℝ\+→ℝ\\phi:\\mathbb\{R\}\_\{\+\}\\to\\mathbb\{R\}is a convex function withϕ⁡\(1\)=0\\phi\(1\)=0\. This formulation includes many widely\-used divergences, including the KL divergence, total variation \(TV\) distance, and theχ2\\chi^\{2\}\-divergence\([Ben\-Tal et al\., 2013](https://arxiv.org/html/2609.10866#bib.bib20)\)\.

We construct the relaxed convex uncertainty set𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}as aϕ\\phi\-divergence ball centered at a reference distribution

p⁡\(τ\)=∏t=1TP⁡\(st∣st−1,at−1\)⋅μ⁡\(ot∣st\)⋅π⁡\(at∣ot\),p\(\\tau\)=\\prod\_\{t=1\}^\{T\}P\(s\_\{t\}\\mid s\_\{t\-1\},a\_\{t\-1\}\)\\cdot\\mu\(o\_\{t\}\\mid s\_\{t\}\)\\cdot\\pi\(a\_\{t\}\\mid o\_\{t\}\),\(11\)which represents the joint distribution over the trajectoryτ\\tauinduced by the agent’s policy, transition dynamics and observation distributions\. Specifically, the relaxed set is given by

𝒟~ϵ′:=\{q∈\(𝒫\(𝒮\)×𝒫\(𝒪\)×𝒫\(𝒜\)\)T:Dϕ\(q∥p\)≤ϵ′\},\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}:=\\left\\\{q\\in\(\\mathcal\{P\}\(\\mathcal\{S\}\)\\times\\mathcal\{P\}\(\\mathcal\{O\}\)\\times\\mathcal\{P\}\(\\mathcal\{A\}\)\)^\{T\}:D\_\{\\phi\}\(q\\\|p\)\\leq\\epsilon^\{\\prime\}\\right\\\},whereϵ′\\epsilon^\{\\prime\}is the threshold determined by the perturbation budgetϵ\\epsilonof the constraint set𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}and theϕ\\phi\-divergence function\. This formulation provides a tractable relaxation of𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}that retains key adversarial deviations from the nominal distribution\.

Therefore, based on the relaxation set𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}, the risk\-averse certification \([10](https://arxiv.org/html/2609.10866#S3.E10)\) can be expressed as

minq∈𝒟~ϵ′−𝔼τ∼q​\(exp⁡β⁡\(∑t=1Tγt​r​\(st,π⁡\(ot\)\)\)\),β<0,\\min\_\{q\\in\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\}\-\\mathbb\{E\}\_\{\\tau\\sim q\}\\left\(\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\pi\(o\_\{t\}\)\)\\right\)\\right\),\\quad\\beta<0,\(12\)which admits a convex formulation and can be efficiently solved using duality techniques\. By leveraging the Lagrangian duality method\([Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\), the convex problem in \([12](https://arxiv.org/html/2609.10866#S4.E12)\) admits an equivalent dual representation, as formalized in the following theorem\.

###### Theorem 1\.

Letτ=\(s1,o1,a1,…,sT,oT,aT\)\\tau=\(s\_\{1\},o\_\{1\},a\_\{1\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\)denote the trajectory up to time horizonTT\. Under theϕ\\phi\-divergence\-based constraintDϕ\(q\|\|p\)≤ϵ′D\_\{\\phi\}\(q\|\|p\)\\leq\\epsilon^\{\\prime\}, the optimization problem in \([12](https://arxiv.org/html/2609.10866#S4.E12)\) is equivalent to solving the following convex optimization problem:

maxξ\>0,η∈R\{ξ\(η−𝔼τ∼p\(ϕ⋆\(\(η\+ϵ′\+exp⁡β⁡\(∑t=1Tγt​r​\(st,π⁡\(ot\)\)\)ξ\)\)\},\\max\_\{\\xi\>0,\\eta\\in R\}\\Big\\\{\\xi\\Big\(\\eta\-\\mathbb\{E\}\_\{\\tau\\sim p\}\\Big\(\\phi^\{\\star\}\\Big\(\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{\\exp\\beta\\left\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\pi\(o\_\{t\}\)\)\\right\)\}\{\\xi\}\\Big\)\\Big\)\\Big\\\},\(13\)whereϕ∗​\(x\)=maxy\>0⁡\(x​y−ϕ⁡\(y\)\)\\phi^\{\*\}\(x\)=\\max\_\{y\>0\}\(xy\-\\phi\(y\)\)denotes the conjugate ofϕ\\phi, andϕ\\phiis a convex function satisfyingϕ⁡\(1\)=0\\phi\(1\)=0\.

###### Proof\.

For each trajectoryτ=\(s1,o1,a1,…,sT,oT,aT\)\\tau=\(s\_\{1\},o\_\{1\},a\_\{1\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\), we defineR⁡\(τ\):=exp⁡β⁡\(∑t=1Tγt​r​\(st,π⁡\(ot\)\)\)R\(\\tau\):=\\exp\\beta\(\\sum\_\{t=1\}^\{T\}\\gamma^\{t\}r\(s\_\{t\},\\pi\(o\_\{t\}\)\)\)as the exponential utility of the cumulative reward with risk\-aversion levelβ<0\\beta<0\. Therefore, the optimization problem \([12](https://arxiv.org/html/2609.10866#S4.E12)\) can be expressed as

minq−𝔼τ∼q\[R\(τ\)\],s\.t\.Dϕ\(q\|\|p\)≤ϵ′\.\\min\_\{q\}\-\\mathbb\{E\}\_\{\\tau\\sim q\}\[R\(\\tau\)\],\\quad\\text\{s\.t\.\}\\,D\_\{\\phi\}\(q\|\|p\)\\leq\\epsilon^\{\\prime\}\.\(14\)The Lagrangian dual of the optimization problem \([14](https://arxiv.org/html/2609.10866#S4.E14)\) is given by:

maxξ\>0minq\{−Eτ∼q\[R\(τ\)\]\+ξ\(Dϕ\(q\|\|p\)−ϵ′\)\}\\displaystyle\\max\_\{\\xi\>0\}\\min\_\{q\}\\left\\\{\-E\_\{\\tau\\sim q\}\[R\(\\tau\)\]\+\\xi\(D\_\{\\phi\}\(q\|\|p\)\-\\epsilon^\{\\prime\}\)\\right\\\}=maxξ\>0\{ξ\[minq\[−Eτ∼q\[R⁡\(τ\)ξ\)\]\+Dϕ\(q\|\|p\)−ϵ′\]\]\}\\displaystyle=\\max\_\{\\xi\>0\}\\left\\\{\\xi\\left\[\\min\_\{q\}\\left\[\-E\_\{\\tau\\sim q\}\[\\frac\{R\(\\tau\)\}\{\\xi\}\)\]\+D\_\{\\phi\}\(q\|\|p\)\-\\epsilon^\{\\prime\}\\right\]\\right\]\\right\\\}=\(15\)​maxξ\>0​\{ξ⁡\[maxt∈ℝ⁡\{t−Eτ∼p​\[ϕ∗​\(t\+R⁡\(τ\)ξ\)\]−ϵ′\}\]\}\\displaystyle\\overset\{\(15\)\}\{=\}\\max\_\{\\xi\>0\}\\left\\\{\\xi\\left\[\\max\_\{t\\in\\mathbb\{R\}\}\\left\\\{t\-E\_\{\\tau\\sim p\}\[\\phi^\{\*\}\(t\+\\frac\{R\(\\tau\)\}\{\\xi\}\)\]\-\\epsilon^\{\\prime\}\\right\\\}\\right\]\\right\\\}=maxξ\>0,η∈ℝ⁡\{ξ⁡\[η\+ϵ′−Eτ∼p​\[ϕ∗​\(η\+ϵ′\+R⁡\(τ\)ξ\)\]−ϵ′\]\}\\displaystyle=\\max\_\{\\xi\>0,\\eta\\in\\mathbb\{R\}\}\\left\\\{\\xi\\left\[\\eta\+\\epsilon^\{\\prime\}\-E\_\{\\tau\\sim p\}\[\\phi^\{\*\}\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{R\(\\tau\)\}\{\\xi\}\)\]\-\\epsilon^\{\\prime\}\\right\]\\right\\\}=maxξ\>0,η∈ℝ⁡\{ξ⁡\[η−Eτ∼p​\[ϕ∗​\(η\+ϵ′\+R⁡\(τ\)ξ\)\]\]\},\\displaystyle=\\max\_\{\\xi\>0,\\eta\\in\\mathbb\{R\}\}\\left\\\{\\xi\\left\[\\eta\-E\_\{\\tau\\sim p\}\[\\phi^\{\*\}\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{R\(\\tau\)\}\{\\xi\}\)\]\\right\]\\right\\\},where \([4\.1](https://arxiv.org/html/2609.10866#S4.Ex9)\) is due to the duality formula of the optimization withϕ\\phidivergence\([Ben\-Tal and Teboulle, 2007](https://arxiv.org/html/2609.10866#bib.bib17)\):minq\{−𝔼τ∼q\[R\(τ\)\]\+Dϕ\(q\|\|p\)\}=maxη∈ℝ\{η−𝔼τ∼p\[ϕ∗\(η\+R\(τ\)\)\]\}\\min\_\{q\}\\\{\-\\mathbb\{E\}\_\{\\tau\\sim q\}\[R\(\\tau\)\]\+D\_\{\\phi\}\(q\|\|p\)\\\}=\\max\_\{\\eta\\in\\mathbb\{R\}\}\\\{\\eta\-\\mathbb\{E\}\_\{\\tau\\sim p\}\[\\phi^\{\*\}\(\\eta\+R\(\\tau\)\)\]\\\}\.

Therefore, minimizing the optimization problem \([12](https://arxiv.org/html/2609.10866#S4.E12)\) with risk\-aversion levelβ<0\\beta<0can be solved by maximizing its dual reformulation \([13](https://arxiv.org/html/2609.10866#S4.E13)\)\. The dual problem is a finite\-dimensional convex program involving two variables,ξ\>0\\xi\>0andη∈ℝ\\eta\\in\\mathbb\{R\}, where the objective is defined via the convex conjugateϕ∗\\phi^\{\*\}of the divergence functionϕ\\phi\. ∎

Theorem[1](https://arxiv.org/html/2609.10866#Thmtheorem1)provides a tractable reformulation of the optimization problem \([12](https://arxiv.org/html/2609.10866#S4.E12)\) that holds for any choice ofϕ\\phi\-divergence\. Given a reference distributionpp, the original infinite\-dimensional problem \([12](https://arxiv.org/html/2609.10866#S4.E12)\) can be equivalently reformulated as the finite\-dimensional convex program \([13](https://arxiv.org/html/2609.10866#S4.E13)\)\.

In practice, selecting an appropriateϕ\\phi\-divergence is crucial, as it ensures the validity and tractability of the convex reformulation in Theorem[1](https://arxiv.org/html/2609.10866#Thmtheorem1)\. In our setting, the uncertainty set𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}includes trajectory distributions induced bylpl\_\{p\}\-norm bounded state perturbations\. To ensure a valid relaxation, theϕ\\phi\-divergence should induce a divergence ball𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}that contains𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}, whereϵ′\\epsilon^\{\\prime\}is a divergence threshold determined by the original perturbation budgetϵ\\epsilonand the choice ofϕ\\phi\. The normppdetermines the geometric structure of𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}, thereby guiding the selection of an appropriate divergenceϕ\\phiand thresholdϵ′\\epsilon^\{\\prime\}to guarantee the inclusion𝒟ϵ⊆𝒟~ϵ′\\mathcal\{D\}^\{\\epsilon\}\\subseteq\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\.

Due to the sequential dependencies in trajectory distributions, it is challenging to directly derive the relationship betweenϕ\\phi\-divergence of joint distributions and thelpl\_\{p\}\-norm bounded perturbations sequencesδ=\(δ1,…,δT\)\\delta=\(\\delta\_\{1\},\\ldots,\\delta\_\{T\}\)\. To address this, we adopt the assumption that adversarial perturbation occurs only at the initial state\([Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2);[Kumar et al\., 2021](https://arxiv.org/html/2609.10866#bib.bib13)\)\. This assumption is justified as the initial perturbation affects the future observations and actions along the trajectory, thereby capturing long\-term adversarial effects while allowing for tractable divergence control\. Under this assumption, given the initial perturbationδ1∈ℬϵ\\delta\_\{1\}\\in\\mathcal\{B\}^\{\\epsilon\}and the initial states1s\_\{1\}, theϕ\\phi\-divergence between the distributionsq⁡\(τ\)q\(\\tau\)andp⁡\(τ\)p\(\\tau\)reduces to theϕ\\phi\-divergence the observation distributionsμ\(⋅\|s1\)\\mu\(\\cdot\|s\_\{1\}\)andμ\(⋅\|s1\+δ1\)\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)\. The detailed derivation is provided in Appendix[7](https://arxiv.org/html/2609.10866#S7)\.

Based on the Gaussian smoothing formulation described in \([3](https://arxiv.org/html/2609.10866#S3.E3)\), the distributions of the initial observation can be modeled asμ\(⋅\|s1\)=𝒩\(s1,σ2Id\)\\mu\(\\cdot\|s\_\{1\}\)=\\mathcal\{N\}\(s\_\{1\},\\sigma^\{2\}I\_\{d\}\)andμ\(⋅\|s1\+δ1\)=𝒩\(s1\+δ1,σ2Id\)\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)=\\mathcal\{N\}\(s\_\{1\}\+\\delta\_\{1\},\\sigma^\{2\}I\_\{d\}\), respectively\. Under this formulation, we identify suitableϕ\\phi\-divergences for different values ofpp\(1≤p<∞1\\leq p<\\infty\), such thatDϕ\(μ\(o1\|s1\+δ1\)\|\|μ\(o1\|s1\)\)≤ϵ′,∀\|\|δ1\|\|p≤ϵD\_\{\\phi\}\(\\mu\(o\_\{1\}\|s\_\{1\}\+\\delta\_\{1\}\)\|\|\\mu\(o\_\{1\}\|s\_\{1\}\)\)\\leq\\epsilon^\{\\prime\},\\forall\|\|\\delta\_\{1\}\|\|\_\{p\}\\leq\\epsilon\. That is, given a perturbation budgetϵ\\epsilon, one can select appropriateϕ\\phi\-divergence functions and thresholdϵ′\\epsilon^\{\\prime\}to ensure that𝒟ϵ⊆𝒟~ϵ′\\mathcal\{D\}^\{\\epsilon\}\\subseteq\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\.

The following proposition characterizes the divergence between the observation distributionsμ\(⋅\|s1\+δ1\)\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)andμ\(⋅\|s1\)\\mu\(\\cdot\|s\_\{1\}\), under adversarial perturbations bounded inl1l\_\{1\}\-norm\. Under the Gaussian smoothing formulation, it establishes a relationship between the perturbation budgetϵ\\epsilonand the corresponding TV thresholdϵ′\\epsilon^\{\\prime\}, enabling a tractable convex relaxation of the original constraint set𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}\.

###### Proposition 1\.

Given the initial states1∈𝒮s\_\{1\}\\in\\mathcal\{S\}and two Gaussian distributionsμ\(⋅\|s1\+δ1\)=𝒩\(s1\+δ1,σ2Id\)\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)=\\mathcal\{N\}\(s\_\{1\}\+\\delta\_\{1\},\\sigma^\{2\}I\_\{d\}\)andμ\(⋅\|s1\)=𝒩\(s1,σ2Id\)\\mu\(\\cdot\|s\_\{1\}\)=\\mathcal\{N\}\(s\_\{1\},\\sigma^\{2\}I\_\{d\}\), for any adversarial perturbation‖δ1‖1≤ϵ\|\|\\delta\_\{1\}\|\|\_\{1\}\\leq\\epsilon, we have

DT​V\(μ\(o1\|s1\+δ1\)\|\|μ\(o1\|s1\)\)≤2Φ\(ϵ2​σ\)−1,D\_\{TV\}\(\\mu\(o\_\{1\}\|s\_\{1\}\+\\delta\_\{1\}\)\|\|\\mu\(o\_\{1\}\|s\_\{1\}\)\)\\leq 2\\Phi\\left\(\\frac\{\\epsilon\}\{2\\sigma\}\\right\)\-1,\(16\)whereΦ\\Phiis the cumulative distribution function \(CDF\) of the standard normal distribution𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\.

###### Proof\.

See details in Appendix[8](https://arxiv.org/html/2609.10866#S8)\. ∎

Given the original constraint set𝒟ϵ\\mathcal\{D\}^\{\\epsilon\}with anl1l\_\{1\}\-norm bounded budget, Proposition[1](https://arxiv.org/html/2609.10866#Thmprop1)shows that when theϕ\\phi\-divergence is defined as the TV distance, i\.e\.,ϕ⁡\(x\)=12​\|x−1\|\\phi\(x\)=\\tfrac\{1\}\{2\}\|x\-1\|, the relaxed constraint set𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}with perturbation budgetϵ′=2​Φ​\(ϵ2​σ\)−1\\epsilon^\{\\prime\}=2\\Phi\\\!\\left\(\\tfrac\{\\epsilon\}\{2\\sigma\}\\right\)\-1satisfies𝒟ϵ⊆𝒟~ϵ′\\mathcal\{D\}^\{\\epsilon\}\\subseteq\\widetilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\.

It has been shown that for any‖δ1‖2≤ϵ\|\|\\delta\_\{1\}\|\|\_\{2\}\\leq\\epsilon, the provably tight relaxation𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}can be selected as

𝒟~ϵ′=\{μ\(⋅\|s1\+δ1\):DH​S,θ\(μ\(⋅\|s1\+δ1\|\|μ\(⋅\|s1\)\)\)≤ϵ′\},∀θ≥0,\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}=\\\{\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\):D\_\{HS,\\theta\}\(\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\|\|\\mu\(\\cdot\|s\_\{1\}\)\)\)\\leq\\epsilon^\{\\prime\}\\\},\\forall\\theta\\geq 0,\(17\)whereDH​S,θ\(μ\(⋅\|s1\+δ1\|\|μ\(⋅\|s1\)\)\)D\_\{HS,\\theta\}\(\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\|\|\\mu\(\\cdot\|s\_\{1\}\)\)\)denotes the Hockey\-Stick divergence between Gaussian distributionsμ\(⋅\|s1\+δ1\)\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)andμ\(⋅\|s1\)\\mu\(\\cdot\|s\_\{1\}\)withϕ⁡\(x\)=max⁡\(x−θ,0\)−max⁡\(1−θ,0\)\\phi\(x\)=\\max\(x\-\\theta,0\)\-\\max\(1\-\\theta,0\)\([Dvijotham et al\., 2020](https://arxiv.org/html/2609.10866#bib.bib21);[Mu et al\., 2024](https://arxiv.org/html/2609.10866#bib.bib2)\)and the thresholdϵ′\\epsilon^\{\\prime\}is given by\([Balle and Wang, 2018](https://arxiv.org/html/2609.10866#bib.bib22)\)

Φ⁡\(ϵ2​σ−log⁡\(θ\)2​ϵ\)−λ​Φ​\(−ϵ2​σ−log⁡\(θ\)2​ϵ\)−max⁡\(1−θ,0\)\.\\Phi\\left\(\\frac\{\\epsilon\}\{2\\sigma\}\-\\frac\{\\log\(\\theta\)\}\{2\\epsilon\}\\right\)\-\\lambda\\Phi\\left\(\-\\frac\{\\epsilon\}\{2\\sigma\}\-\\frac\{\\log\(\\theta\)\}\{2\\epsilon\}\\right\)\-\\max\(1\-\\theta,0\)\.\(18\)
In the subsequent subsection, we develop algorithms to solve the maximization problem \([13](https://arxiv.org/html/2609.10866#S4.E13)\), aiming to certify a lower bound for RL deployment under state adversarial perturbations\.

#### 4\.2Lower bound certificate algorithms

In this subsection, we develop algorithms to approximate the lower bound certificate defined in \([12](https://arxiv.org/html/2609.10866#S4.E12)\) under a convex relaxation set𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}\. Theorem[1](https://arxiv.org/html/2609.10866#Thmtheorem1)provides a tractable dual formulation \([13](https://arxiv.org/html/2609.10866#S4.E13)\) for problem \([12](https://arxiv.org/html/2609.10866#S4.E12)\) by introducing the Lagrange multipliersξ\\xiandη\\eta\. The algorithm for solving the dual problem \([13](https://arxiv.org/html/2609.10866#S4.E13)\) is detailed in Algorithm[1](https://arxiv.org/html/2609.10866#alg1)\.

To solve the convex optimization problem in \([13](https://arxiv.org/html/2609.10866#S4.E13)\), it is necessary to evaluate the expectation

𝔼τ∼p​\(ϕ⋆​\(\(η\+ϵ′\+R⁡\(τ\)ξ\)\)CLOSE\.\\mathbb\{E\}\_\{\\tau\\sim p\}\(\\phi^\{\\star\}\(\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{R\(\\tau\)\}\{\\xi\}\)\)\.\(19\)It is challenging to calculate the exact value of the expected return due to the sequential nature of RL\. We adopt a Monte Carlo sampling method to approximate the expectation of the exponential utility return\. Specifically, the Monte Carlo estimate of the above expectation is given byμ^=1M​∑i=1Mϕ∗​\(η\+ϵ′\+R⁡\(τi\)ξ\)\\hat\{\\mu\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\phi^\{\\ast\}\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{R\(\\tau\_\{i\}\)\}\{\\xi\}\), whereτ1,…,τM\\tau\_\{1\},\\dots,\\tau\_\{M\}are independent trajectory samples drawn fromp⁡\(τ\)p\(\\tau\)\.

To quantify the estimation uncertainty, we construct a confidence interval for the true expectation at a confidence level ofα=0\.01\\alpha=0\.01\. According to Hoeffding’s inequality\([Vershynin, 2018](https://arxiv.org/html/2609.10866#bib.bib25)\), givenMMindependent samples, the expectation \([19](https://arxiv.org/html/2609.10866#S4.E19)\) is bounded above by

μ^upper=μ^\+K​ln⁡\(2/α\)2​M\\hat\{\\mu\}\_\{\\text\{upper\}\}=\\hat\{\\mu\}\+K\\sqrt\{\\frac\{\\ln\(2/\\alpha\)\}\{2M\}\}\(20\)with probability at least1−α1\-\\alpha, whereKKdenotes the range ofϕ∗​\(η\+ϵ′\+R⁡\(τ\)ξ\)\\phi^\{\\ast\}\(\\eta\+\\epsilon^\{\\prime\}\+\\frac\{R\(\\tau\)\}\{\\xi\}\)\. In this paper, we adopt the upper confidence boundμupper\\mu\_\{\\text\{upper\}\}as a conservative estimation of expectation \([19](https://arxiv.org/html/2609.10866#S4.E19)\), ensuring that the resulting certificate remains a valid lower bound on policy performance with high probability\.

Algorithm[1](https://arxiv.org/html/2609.10866#alg1)performs this estimation procedure and consists of two main components: solving the convex optimization problem in \([13](https://arxiv.org/html/2609.10866#S4.E13)\) and evaluating the exponential utility of the cumulative reward under a smoothed policyπ~\\tilde\{\\pi\}\. The overall framework of Algorithm[1](https://arxiv.org/html/2609.10866#alg1)is adapted from[Mu et al\. \(2024\)](https://arxiv.org/html/2609.10866#bib.bib2)with modifications to accommodate our risk\-sensitive certification setting\. These procedures are detailed in Functions[1](https://arxiv.org/html/2609.10866#function1)and[2](https://arxiv.org/html/2609.10866#function2), respectively\.

Algorithm 1Lower bound certificate algorithm1:Trained smoothed policy

π~\\tilde\{\\pi\}, smoothed

QQ\-network

Qπ~Q^\{\\tilde\{\\pi\}\},

ϕ\\phi\-divergence\-based convex relaxation set

𝒟~ϵ′\\tilde\{\\mathcal\{D\}\}^\{\\epsilon^\{\\prime\}\}, risk aversion level

β\\beta, sample sizes

NNand

MM\(

N<MN<M\)

2:

ξ∗,η∗←\\xi^\{\*\},\\eta^\{\*\}\\leftarrowOptimization

\(N,Qπ~,ϵ′,ϕ\)\(N,Q^\{\\tilde\{\\pi\}\},\\epsilon^\{\\prime\},\\phi\)// Calculate the dual optimization problem \([21](https://arxiv.org/html/2609.10866#S4.E21)\) with a relatively small sample size

NN
3:

R=\{R1,⋯,RM\}←R=\\\{R\_\{1\},\\cdots,R\_\{M\}\\\}\\leftarrowGetReward

\(M,Qπ~\)\(M,Q^\{\\tilde\{\\pi\}\}\)// Calculate the exponential utility of the cumulative reward with smoothed policy with a relatively large sample size

MM
4:Compute

ϕ∗​\(η∗\+ϵ′\+Riξ∗\)\\phi^\{\*\}\(\\eta^\{\*\}\+\\epsilon^\{\\prime\}\+\\frac\{R\_\{i\}\}\{\\xi^\{\*\}\}\)for each sample

i=1,…,Mi=1,\\ldots,M, and obtain the Monte Carlo estimate

μ^=1M​∑i=1Mϕi∗​\(η∗\+ϵ′\+Riξ∗\)\\hat\{\\mu\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\phi^\{\*\}\_\{i\}\(\\eta^\{\*\}\+\\epsilon^\{\\prime\}\+\\frac\{R\_\{i\}\}\{\\xi^\{\*\}\}\)along with its upper bound

μ^upper\\hat\{\\mu\}\_\{\\text\{upper\}\}\([20](https://arxiv.org/html/2609.10866#S4.E20)\)

returnCertified lower boundR¯=ξ∗​\(η∗−μ^upper\)\\underline\{R\}=\\xi^\{\*\}\(\\eta^\{\*\}\-\\hat\{\\mu\}\_\{\\text\{upper\}\}\)

Function 1Optimization\(N,Qπ~,ϵ′,ϕ\)\(N,Q^\{\\tilde\{\\pi\}\},\\epsilon^\{\\prime\},\\phi\)1:Sample size

NN, trained smoothed policy

π~\\tilde\{\\pi\}, smoothed

QQ\-network

Qπ~Q^\{\\tilde\{\\pi\}\}
2:

R=\{R1,…,RN\}←GetReward​\(N,Qπ~\)R=\\\{R\_\{1\},\\ldots,R\_\{N\}\\\}\\leftarrow\\text\{GetReward\}\(N,Q^\{\\tilde\{\\pi\}\}\)
3:Solving the convex optimization problem

maxξ\>0,η∈R\{ξ\[η−∑i=1N\(ϕ⋆\(η\+ϵ′\+Riν\)\]\}\\max\_\{\\xi\>0,\\eta\\in R\}\\big\\\{\\xi\\big\[\\eta\-\\sum\_\{i=1\}^\{N\}\\big\(\\phi^\{\\star\}\(\\eta\+\{\\epsilon^\{\\prime\}\}\+\\frac\{R\_\{i\}\}\{\\nu\}\\big\)\\big\]\\big\\\}\(21\)

returnOptimal solutionsξ∗,η∗\\xi^\{\*\},\\eta^\{\*\}

In Algorithm[1](https://arxiv.org/html/2609.10866#alg1), Step 1 callsOptimization\(N,Qπ~,ϵ′,ϕ\)\(N,Q^\{\\tilde\{\\pi\}\},\\epsilon^\{\\prime\},\\phi\)\(Function[1](https://arxiv.org/html/2609.10866#function1)\) to solve the dual optimization problem defined in \([21](https://arxiv.org/html/2609.10866#S4.E21)\)\. In this function, the expectation in \([19](https://arxiv.org/html/2609.10866#S4.E19)\) is approximated using a Monte Carlo average overNNreturn samples, which are obtained by executing the smoothed policyπ~\\tilde\{\\pi\}on perturbed trajectories \(see details in Function[2](https://arxiv.org/html/2609.10866#function2)\)\. Based on this approximation, the original risk‑sensitive certification problem \([13](https://arxiv.org/html/2609.10866#S4.E13)\) is reformulated as the convex optimization problem \([21](https://arxiv.org/html/2609.10866#S4.E21)\), which can be efficiently solved to obtain the optimal Lagrange multipliersξ∗\\xi^\{\*\}andη∗\\eta^\{\*\}\. The optimization problem \([21](https://arxiv.org/html/2609.10866#S4.E21)\) is implemented using the CVXPY modeling framework\([Diamond and Boyd, 2016](https://arxiv.org/html/2609.10866#bib.bib15)\), which enables efficient and reliable solution of the convex objective using standard convex solvers under disciplined convex programming rules\.

In step 2, Algorithm[1](https://arxiv.org/html/2609.10866#alg1)callsGetReward\(M,Qπ~\)\(M,Q^\{\\tilde\{\\pi\}\}\)\(Function[2](https://arxiv.org/html/2609.10866#function2)\) to generate return samplesR=\{R1,…,RM\}R=\\\{R\_\{1\},\\dots,R\_\{M\}\\\}by executing the smoothed policyπ~\\tilde\{\\pi\}overMMindependent trajectories, each perturbed by Gaussian noiseΔ∼𝒩⁡\(0,σ2​Id\)\\Delta\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)\. At each time steptt, the agent selects actions according to the well\-trained smoothedQQ\-networkQπ~Q^\{\\tilde\{\\pi\}\}, and the exponential utility of the cumulative reward is recorded to form the Monte Carlo sample set\. Note that a relatively small sample sizeNNis used in Step 1 to improve computational efficiency when solving the optimization problem \([21](https://arxiv.org/html/2609.10866#S4.E21)\), as each sample increases the complexity of the convex objective\. Step 2 adopts a larger sample sizeMMto generate return samples for a more accurate Monte Carlo estimate of the certification objective in Step 3\.

Function 2GetReward\(M,Qπ~\)\(M,Q^\{\\tilde\{\\pi\}\}\)1:Sample size

MM, trained smoothed policy

π~\\tilde\{\\pi\}, smoothed

QQ\-network

Qπ~Q^\{\\tilde\{\\pi\}\}, risk aversion level

β\\beta
2:for

m←1,…​Mm\\leftarrow 1,\\ldots Mdo

3:

Rm←0R\_\{m\}\\leftarrow 0
4:for

ttfrom 1 to

TTdo

5:Sample random noise

Δ∼𝒩⁡\(0,σ2​Id\)\\Delta\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\_\{d\}\)
6:Inject random noise to the state

st′←st\+Δs^\{\\prime\}\_\{t\}\\leftarrow s\_\{t\}\+\\Delta
7:Select greedy action

at←arg⁡maxa∈A​Qπ~​\(st′,a\)a\_\{t\}\\leftarrow\\arg\\max\_\{a\\in A\}Q^\{\\tilde\{\\pi\}\}\(s^\{\\prime\}\_\{t\},a\)
8:Obtain reward

rtr\_\{t\}and next state

st\+1s\_\{t\+1\}
9:

Rm←exp⁡β⁡\(Rm\+rt\)R\_\{m\}\\leftarrow\\exp\\beta\(R\_\{m\}\+r\_\{t\}\)
10:endfor

11:endfor

returnR=\{R1,…,RM\}R=\\\{R\_\{1\},\\ldots,R\_\{M\}\\\}

Finally, in Step 3, Algorithm[1](https://arxiv.org/html/2609.10866#alg1)computes the Monte Carlo estimate of the dual objective usingMMsampled trajectories and derives its upper confidence boundμ^upper\\hat\{\\mu\}\_\{\\mathrm\{upper\}\}under a confidence level ofα\\alpha\. The certified lower bound is then computed as

R=ξ∗​\(η∗−μ^upper\),R=\\xi^\{\*\}\(\\eta^\{\*\}\-\\hat\{\\mu\}\_\{\\mathrm\{upper\}\}\),and returned as the final output of the algorithm\. This certification procedure provides a computationally tractable estimate of the certified performance guarantee under state adversarial perturbations during RL deployment\.

### 5Numerical Study

In this section, we conduct experiments to demonstrate the proposed methods for computing the risk\-averse lower bound certification \([7](https://arxiv.org/html/2609.10866#S3.E7)\)\. We also investigate how different levels of risk aversion during training affect the certified performance of RL policies under adversarial state perturbations\. We treat the risk aversion levelβ\\betaas a tunable hyperparameter, allowing the training value \(βtrain\\beta\_\{\\text\{train\}\}\) to differ from the evaluation\-time value \(βtest\\beta\_\{\\text\{test\}\}\) used for lower bound certification\.

![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_neutral_l2.png)\(a\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.2_l2.png)\(b\)βtest=−0\.2\\beta\_\{\\text\{test\}\}=\-0\.2\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.4_l2.png)\(c\)βtest=−0\.4\\beta\_\{\\text\{test\}\}=\-0\.4\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.6_l2.png)\(d\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_neutral_l1.png)\(e\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.2_l1.png)\(f\)βtest=−0\.2\\beta\_\{\\text\{test\}\}=\-0\.2\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.4_l1.png)\(g\)βtest=−0\.4\\beta\_\{\\text\{test\}\}=\-0\.4\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/lunar_risk_averse_0.6_l1.png)\(h\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l1l\_\{1\}\-norm

Figure 1:Certified lower bounds for Lunar Lander with risk\-neutral and risk\-averse testing environments underl2l\_\{2\}\-norm \(top two rows\) andl1l\_\{1\}\-norm \(bottom two rows\) perturbations\.We train multiple policies using the risk\-neutral objective \([1](https://arxiv.org/html/2609.10866#S3.E1)\) and the risk\-averse objective \([2](https://arxiv.org/html/2609.10866#S3.E2)\) with varying values ofβtrain\\beta\_\{\\text\{train\}\}\. These policies are then evaluated under both risk\-neutral \(βtest→0\\beta\_\{\\text\{test\}\}\\to 0\) and risk\-averse \(βtest<0\\beta\_\{\\text\{test\}\}<0\) testing\-time objectives \([7](https://arxiv.org/html/2609.10866#S3.E7)\), and their certified lower bounds are compared to assess policy performance across different deployment conditions\. The experiments are designed to identify the training\-time risk aversion levelβtrain\\beta\_\{\\text\{train\}\}that maximizes certified lower bound for each testing deployment condition\. The results suggest that tuningβtrain\\beta\_\{\\text\{train\}\}can influence certified performance, which provides practical guidance for selecting the suitable training hyperparameters based on anticipated deployment scenarios\.

We first conduct experiments in the Lunar Lander and CartPole environments from OpenAI Gym, both of which feature continuous state spaces and discrete action spaces\([Towers and others, 2024](https://arxiv.org/html/2609.10866#bib.bib27)\)\. We then extend our analysis to a stylized machine replacement problem\([Puterman, 1994](https://arxiv.org/html/2609.10866#bib.bib26)\), with the goal of providing interpretability into how risk\-averse training environments contribute to the improvement of lower bound certificates under state adversarial perturbations\. To prevent computational issues due to vanishing exponential values arising from large rewards under negative risk aversion parametersβ\\beta, we normalize the reward scale across all three environments\.

#### 5\.1OpenAI Gym environment

We first consider the Lunar Lander environment from the Box2D suite of OpenAI Gym, where the agent controls a spacecraft to land on a designated pad\([Towers and others, 2024](https://arxiv.org/html/2609.10866#bib.bib27)\)\. Since small variations in state observations \(e\.g\., position or velocity\) can influence the selection of actions and ultimately affect task outcomes, this environment is well\-suited for evaluating the certified lower bounds for RL policies under state adversarial perturbations\. In particular, we apply Gaussian noise with zero mean and a standard deviation of 0\.05 to simulate perturbations in the continuous state observations\. For a range of perturbation budgetsϵ∈\(0,1\)\\epsilon\\in\(0,1\), we compute the corresponding certified lower bounds \([7](https://arxiv.org/html/2609.10866#S3.E7)\) underl2l\_\{2\}\- andl1l\_\{1\}\-norm constraints to assess the performance of different RL policies\. We evaluate the trained policies across four testing risk levels, including a risk\-neutral certification \([7](https://arxiv.org/html/2609.10866#S3.E7)\) withβtest→0\\beta\_\{\\text\{test\}\}\\to 0, and three risk\-averse certifications \([7](https://arxiv.org/html/2609.10866#S3.E7)\) withβtest=−0\.2,−0\.4,−0\.6\\beta\_\{\\text\{test\}\}=\{\-0\.2,\-0\.4,\-0\.6\}\. For each testing scenario, we evaluate and compare the certified lower bounds of RL policies trained under different training objectives and risk aversion levels\.

Figure[1](https://arxiv.org/html/2609.10866#S5.F1)presents the certified lower bounds of RL policies evaluated under multiple testing risk aversion levels in the Lunar Lander environment\. The results include both risk\-neutral and risk\-averse testing scenarios, where certified lower bounds are evaluated under adversarial perturbations bounded by thel2l\_\{2\}norm \(top two rows\) andl1l\_\{1\}norm \(bottom two rows\)\. Each curve represents the certified lower bound of an RL policy trained with a distinctβtrain\\beta\_\{\\text\{train\}\}value\. In all subfigures, we observe that the certified lower bounds are non\-increasing as the perturbation budget increases\.

Figures[1a](https://arxiv.org/html/2609.10866#S5.F1.sf1)and[1e](https://arxiv.org/html/2609.10866#S5.F1.sf5)show the certified lower bounds of policies trained under varying levels of risk aversion, evaluated in a risk\-neutral testing environment withl2l\_\{2\}\- andl1l\_\{1\}\-norm bounded adversarial perturbations, respectively\. We select the curve corresponding to the policy trained with the risk\-neutral objective as the baseline for comparison\. We observe that risk\-averse training generally produces policies with higher certified lower bounds than those obtained through risk\-neutral training, particularly under larger perturbation budgets\. This suggests that incorporating risk aversion during training enhances the certified robustness of policies in risk\-neutral evaluation settings with adversarial perturbations\. Moreover, we also find that as the risk aversion parameterβtrain\\beta\_\{\\text\{train\}\}decreases, the certified lower bounds initially increase and then decrease\. Specifically, asβtrain\\beta\_\{\\text\{train\}\}decreases from−0\.3\-0\.3to−0\.7\-0\.7, the corresponding lower bound curves gradually improve, but atβtrain=−0\.8\\beta\_\{\\text\{train\}\}=\-0\.8, the lower bound decreases, exhibiting a non\-monotonic trend\. This indicates that excessive risk aversion during training can lead to over conservative policies and consequently degrade certified performance under perturbations\.

![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_neutral_l2.png)\(a\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.1_l2.png)\(b\)βtest=−0\.1\\beta\_\{\\text\{test\}\}=\-0\.1\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.4_l2.png)\(c\)βtest=−0\.4\\beta\_\{\\text\{test\}\}=\-0\.4\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.6_l2.png)\(d\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_neutral_l1.png)\(e\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.1_l1.png)\(f\)βtest=−0\.1\\beta\_\{\\text\{test\}\}=\-0\.1\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.4_l1.png)\(g\)βtest=−0\.4\\beta\_\{\\text\{test\}\}=\-0\.4\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/cartpole_risk_averse_0.6_l1.png)\(h\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l1l\_\{1\}\-norm

Figure 2:Certified lower bounds for CartPole with risk\-neutral and risk\-averse testing environments underl2l\_\{2\}\-norm \(top two rows\) andl1l\_\{1\}\-norm \(bottom two rows\) perturbations\.Figures[1b](https://arxiv.org/html/2609.10866#S5.F1.sf2)to[1d](https://arxiv.org/html/2609.10866#S5.F1.sf4)and[1f](https://arxiv.org/html/2609.10866#S5.F1.sf6)to[1h](https://arxiv.org/html/2609.10866#S5.F1.sf8)present the certified lower bounds evaluated in risk\-averse testing scenarios underl2l\_\{2\}andl1l\_\{1\}norm\-bounded perturbations, respectively\. Among these subfigures, we observe that the certified lower bounds initially increase and subsequently decrease asβtrain\\beta\_\{\\text\{train\}\}decreases, which is consistent with the pattern observed in the risk\-neutral testing case, where relatively moderate risk aversion tends to improve robustness, while excessive aversion can lead to performance degradation\.

We also consider the CartPole environment from the classic control suite of OpenAI Gym, in which an agent needs to balance a pole by moving a cart left or right\([Towers and others, 2024](https://arxiv.org/html/2609.10866#bib.bib27)\)\. In the CartPole environment, we perturb the continuous state observations using Gaussian noise with zero mean and a standard deviation of 0\.2 to simulate adversarial state uncertainty\. The CartPole experiment follows the same setup as Lunar Lander\. Specifically, we compare the certified lower bounds of various RL policies under both risk\-neutral and risk\-averse evaluation settings, measured with respect to adversarial perturbations bounded by thel2l\_\{2\}andl1l\_\{1\}norms\. The results are presented in Figure[2](https://arxiv.org/html/2609.10866#S5.F2)\. Figure[2](https://arxiv.org/html/2609.10866#S5.F2)shows similar trends to those observed in Figure[1](https://arxiv.org/html/2609.10866#S5.F1), supporting the consistency of our findings\.

#### 5\.2Machine replacement problem

In order to gain more insights on why increasing risk aversion during training can lead to more robust policies and improved testing performance, we next evaluate our certification framework on the machine replacement problem\([Puterman, 1994](https://arxiv.org/html/2609.10866#bib.bib26)\)\.

We consider a machine with a continuous state space𝒮=\[0,10\)\\mathcal\{S\}=\[0,10\), where smaller state values correspond to healthier operating conditions and larger values indicate increasing levels of deterioration\. Specifically, the interval\[0,1\)\[0,1\)represents a like\-new condition, whereas states closer to the upper bound correspond to severely degraded conditions\. For computational tractability, the continuous state space is uniformly partitioned into 10 intervals:\[0,1\),\[1,2\),…,\[9,10\)\[0,1\),\[1,2\),\\ldots,\[9,10\)\. Each continuous states∈Ss\\in Sis discretized using the floor function⌊s⌋\\lfloor s\\rfloor, which denotes the greatest integer less than or equal toss\. This yields a finite set of discretized degradation levels indexed from 0 to 9\.

At each decision step, the agent makes a binary decision: to do nothing\(a=0\)\(a=0\)or to perform maintenance\(a=1\)\(a=1\)\. The last degradation interval\[9,10\)\[9,10\)is an absorbing state, where the agent is required to perform maintenance\. The cost function is defined as:

c⁡\(s,a\)=\{0,a=0,s∈\[0,9\)2,a=1,s∈\[0,9\)5,a=1,s∈\[9,10\)\.c\(s,a\)=\\begin\{cases\}0,&a=0,s\\in\[0,9\)\\\\ 2,&a=1,s\\in\[0,9\)\\\\ 5,&a=1,\\,s\\in\[9,10\)\.\\end\{cases\}When actiona=1a=1is taken, the machine is fully repaired and returns to the like\-new interval\[0,1\)\[0,1\)with probability 1\. When doing\-nothing action \(a=0a=0\) is selected, the system evolves according to a Poisson deterioration process\. Leti=⌊st⌋i=\\lfloor s\_\{t\}\\rfloordenote the discretized degradation level of the current statests\_\{t\}\.The transition probability to the next discretized statej≥ij\\geq iis given by a truncated Poisson distribution with a state\-dependent rate parameterλi\\lambda\_\{i\}:

ℙ⁡\(⌊st\+1⌋=j∣⌊st⌋=i,at=0\)=λij−i​e−λi\(j−i\)\!​Zi,\\mathbb\{P\}\(\\lfloor s\_\{t\+1\}\\rfloor=j\\mid\\lfloor s\_\{t\}\\rfloor=i,a\_\{t\}=0\)=\\frac\{\\lambda\_\{i\}^\{j\-i\}e^\{\-\\lambda\_\{i\}\}\}\{\(j\-i\)\!Z\_\{i\}\},wherej=i,i\+1,…,8j=i,i\+1,\\dots,8andZi=∑k=08−iλik​e−λik\!Z\_\{i\}=\\sum\_\{k=0\}^\{8\-i\}\\frac\{\\lambda\_\{i\}^\{k\}e^\{\-\\lambda\_\{i\}\}\}\{k\!\}is the normalization constant ensuring a valid probability distribution\. The rate parameterλi\\lambda\_\{i\}varies with the discretized degradation leveliiand increases with the degree of deterioration, taking values\{0\.1,0\.2,0\.3,0\.4,0\.5,0\.6,0\.8,1\.0,1\.5\}\\\{0\.1,0\.2,0\.3,0\.4,0\.5,0\.6,0\.8,1\.0,1\.5\\\}fori=0,1,…,8i=0,1,\\ldots,8, respectively\. The discretized state⌊s⌋=9\\lfloor s\\rfloor=9is treated as an absorbing state under the actiona=0a=0\.

The machine replacement problem traditionally aims to minimize the expected cumulative cost\. To maintain consistency with the previous experiments, we reformulate the objective as maximizing the cumulative negative cost, interpreted as a reward in the RL framework\.

In line with the previous experiments, we assess the certified lower bounds across different testing conditions to investigate the impact of training\-time risk preferences on certified performance\. In these evaluations, the underlying state is perturbed by zero\-mean Gaussian noise with a standard deviation of 0\.5\. Figures[3a](https://arxiv.org/html/2609.10866#S5.F3.sf1)and[3e](https://arxiv.org/html/2609.10866#S5.F3.sf5)present the certified lower bounds of the policies evaluated under a risk\-neutral testing condition, with adversarial perturbations bounded by thel2l\_\{2\}andl1l\_\{1\}norms, respectively\. We observe that policies trained with risk\-averse objectives tend to achieve higher certified lower bounds than risk\-neutral ones, particularly under larger perturbation budgets\. Moreover, as the degree of risk aversion decreases, the certified lower bounds initially increase but decrease atβtrain=−0\.8\\beta\_\{\\text\{train\}\}=\-0\.8, indicating a non\-monotonic relationship between training\-time risk aversion and testing\-time certification performance\. This pattern can be explained by the structural property of the optimal policy in the machine replacement problem\. Specifically, the optimal policy is characterized by a thresholds∗s^\{\*\}: the agent performs maintenance\(a=1\)\(a=1\)when⌊s⌋≥s∗\\lfloor s\\rfloor\\geq s^\{\*\}and does nothing\(a=0\)\(a=0\)otherwise\([Puterman, 1994](https://arxiv.org/html/2609.10866#bib.bib26)\)\. When the policy is trained under a risk\-averse objective, the threshold tends to be lower, leading the agent to select the repair action at earlier stages of degradation\. Such behavior enhances policy robustness under perturbed observations, as the agent is more likely to make the same maintenance decision even when the observed state deviates from the true underlying state\. As a result, the certified lower bounds of risk\-averse policies can outperform those of risk\-neutral ones, particularly under testing conditions with larger perturbation budgets\. However, when the risk\-aversion level becomes excessively strong, the learned policy turns overly conservative, initiating maintenance too early and thereby causing the certified lower bounds to deteriorate during testing\.

![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_neutral_l2.png)\(a\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.1_l2.png)\(b\)βtest=−0\.1\\beta\_\{\\text\{test\}\}=\-0\.1\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.3_l2.png)\(c\)βtest=−0\.3\\beta\_\{\\text\{test\}\}=\-0\.3\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.6_l2.png)\(d\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l2l\_\{2\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_neutral_l1.png)\(e\)βtest→0\\beta\_\{\\text\{test\}\}\\to 0\(risk\-neutral\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.1_l1.png)\(f\)βtest=−0\.1\\beta\_\{\\text\{test\}\}=\-0\.1\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.3_l1.png)\(g\)βtest=−0\.3\\beta\_\{\\text\{test\}\}=\-0\.3\(risk\-averse\),l1l\_\{1\}\-norm
![Refer to caption](https://arxiv.org/html/2609.10866v1/new_figure/mr_risk_averse_0.6_l1.png)\(h\)βtest=−0\.6\\beta\_\{\\text\{test\}\}=\-0\.6\(risk\-averse\),l1l\_\{1\}\-norm

Figure 3:Certified lower bounds for the machine replacement problem with risk\-neutral and risk\-averse testing environments underl2l\_\{2\}\-norm \(top two rows\) andl1l\_\{1\}\-norm \(bottom two rows\) perturbations\.Figures[3b](https://arxiv.org/html/2609.10866#S5.F3.sf2)to[3d](https://arxiv.org/html/2609.10866#S5.F3.sf4)and[3f](https://arxiv.org/html/2609.10866#S5.F3.sf6)to[3h](https://arxiv.org/html/2609.10866#S5.F3.sf8)present the certified lower bounds of policies trained under varying levels of risk aversion, evaluated in testing environments with parametersβtest=−0\.1,−0\.3\\beta\_\{\\text\{test\}\}=\-0\.1,\-0\.3and−0\.6\-0\.6\. We observe that, under bothl2l\_\{2\}\- andl1l\_\{1\}\-bounded perturbations within the same testing environment, the certified lower bounds initially increase and then decrease as the training\-time risk parameterβtrain\\beta\_\{\\text\{train\}\}becomes more negative\. This non\-monotonic trend is consistent with the patterns previously observed in the Lunar Lander and CartPole environments\.

### 6Conclusion

In this paper, we propose a novel certification framework for evaluating the robustness of RL policies under state adversarial perturbations\. Specifically, we formulate the certification objective as the expectation of exponential utility returns to reflect the agent’s risk preference underlpl\_\{p\}\-norm bounded perturbations \(1≤p<∞1\\leq p<\\infty\)\. By introducing aϕ\\phi\-divergence–based constraint on the perturbation budget, we construct a convex relaxation of the risk\-sensitive certification objective and derive its tractable dual formulation\. We conduct extensive experiments on both OpenAI Gym environments and a machine replacement task\. In each setting, we train policies using a range of training\-time risk aversion levels \(βtrain\\beta\_\{\\text\{train\}\}\)\. We then evaluate their certified lower bounds under different evaluation\-time parameters \(βtest\\beta\_\{\\text\{test\}\}\) to analyze how risk preferences influence certified robustness\. Our results show that risk\-averse training generally results in policies with higher certified lower bounds than risk\-neutral training, especially under larger perturbation budgets\. Moreover, in both risk\-neutral and risk\-averse evaluation settings, increasing risk aversion during training tends to improve certified robustness initially, while excessive risk aversion will ultimately degrade certification performance due to overly conservative policies\.

### 7Equivalence Between Trajectory Distribution Divergence and Observation Distribution Divergence

In this appendix, we reduce theϕ\\phi\-divergence between trajectory distributionsq⁡\(τ\)q\(\\tau\)andp⁡\(τ\)p\(\\tau\), withτ=\(s1,o1,a1,…,sT,oT,aT\)\\tau=\(s\_\{1\},o\_\{1\},a\_\{1\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\), to theϕ\\phi\-divergence between observation distributionsμ\(⋅∣s1\+δ1\)\\mu\(\\cdot\\mid s\_\{1\}\+\\delta\_\{1\}\)andμ\(⋅∣s1\)\\mu\(\\cdot\\mid s\_\{1\}\), given a fixed initial states1s\_\{1\}and assuming that onlys1s\_\{1\}is perturbed byδ1\\delta\_\{1\}\.

Dϕ\(q\(τ\)\|\|p\(τ\)\)\\displaystyle D\_\{\\phi\}\(q\(\\tau\)\|\|p\(\\tau\)\)=∫τϕ⁡\(q⁡\(τ\)p⁡\(τ\)\)​𝑑p​\(τ\)\\displaystyle=\\int\_\{\\tau\}\\phi\\left\(\\frac\{q\(\\tau\)\}\{p\(\\tau\)\}\\right\)dp\(\\tau\)=∫τϕ⁡\(μ⁡\(o1\|s1\+δ1\)μ⁡\(o1\|s1\)×π⁡\(a1\|o1\)π⁡\(a1\|o1\)×∏t=2TP⁡\(st\|st−1,at−1\)​μ​\(ot\|st\)​π​\(at\|ot\)OPENP⁡\(st\|st−1,at−1\)​μ​\(ot\|st\)​π​\(at\|ot\)\)\)​𝑑p​\(τ\)\\displaystyle=\\int\_\{\\tau\}\\phi\\left\(\\frac\{\\mu\(o\_\{1\}\|s\_\{1\}\+\\delta\_\{1\}\)\}\{\\mu\(o\_\{1\}\|s\_\{1\}\)\}\\times\\frac\{\\bcancel\{\\pi\(a\_\{1\}\|o\_\{1\}\)\}\}\{\\bcancel\{\\pi\(a\_\{1\}\|o\_\{1\}\)\}\}\\times\\prod\_\{t=2\}^\{T\}\\frac\{\\bcancel\{P\(s\_\{t\}\|s\_\{t\-1\},a\_\{t\-1\}\)\\mu\(o\_\{t\}\|s\_\{t\}\)\\pi\(a\_\{t\}\|o\_\{t\}\)\}\}\{\\bcancel\{P\(s\_\{t\}\|s\_\{t\-1\},a\_\{t\-1\}\)\\mu\(o\_\{t\}\|s\_\{t\}\)\\pi\(a\_\{t\}\|o\_\{t\}\)\)\}\}\\right\)dp\(\\tau\)\(22\)=∫o1ϕ⁡\(μ⁡\(o1\|s1\+δ1\)μ⁡\(o1\|s1\)\)​μ​\(o1\|s1\)\\displaystyle=\\int\_\{o\_\{1\}\}\\phi\\left\(\\frac\{\\mu\(o\_\{1\}\|s\_\{1\}\+\\delta\_\{1\}\)\}\{\\mu\(o\_\{1\}\|s\_\{1\}\)\}\\right\)\\mu\(o\_\{1\}\|s\_\{1\}\)×\(∫a1,s2,o2,a2,…,sT,oT,aTP\(a1,…,sT,oT,aT\|o1,s1\)da1ds2do2da2…dsTdoTdaT\)do1\\displaystyle\\quad\\times\\left\(\\int\_\{a\_\{1\},s\_\{2\},o\_\{2\},a\_\{2\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\}P\(a\_\{1\},\\ldots,s\_\{T\},o\_\{T\},a\_\{T\}\|o\_\{1\},s\_\{1\}\)da\_\{1\}ds\_\{2\}do\_\{2\}da\_\{2\}\\ldots ds\_\{T\}do\_\{T\}da\_\{T\}\\right\)do\_\{1\}=∫o1μ⁡\(o1\|s1\)​ϕ​\(μ⁡\(o1\|s1\+δ1\)μ⁡\(o1\|s1\)\)​d​o1\\displaystyle=\\int\_\{o\_\{1\}\}\\mu\(o\_\{1\}\|s\_\{1\}\)\\phi\\left\(\\frac\{\\mu\(o\_\{1\}\|s\_\{1\}\+\\delta\_\{1\}\)\}\{\\mu\(o\_\{1\}\|s\_\{1\}\)\}\\right\)do\_\{1\}\(23\)=Dϕ\(μ\(⋅\|s1\+δ1\)\|\|μ\(⋅\|s1\)\),\\displaystyle=D\_\{\\phi\}\(\\mu\(\\cdot\|s\_\{1\}\+\\delta\_\{1\}\)\|\|\\mu\(\\cdot\|s\_\{1\}\)\),where \([22](https://arxiv.org/html/2609.10866#S7.E22)\) is due to the expanded form \([9](https://arxiv.org/html/2609.10866#S3.E9)\) and \([11](https://arxiv.org/html/2609.10866#S4.E11)\) of the trajectory distributionsq⁡\(τ\)q\(\\tau\)andp⁡\(τ\)p\(\\tau\)respectively\. \([23](https://arxiv.org/html/2609.10866#S7.E23)\) follows from marginalizing the integral over all trajectory variables to retain only the initial observationo1o\_\{1\}\.

### 8Proof of Proposition[1](https://arxiv.org/html/2609.10866#Thmprop1)

We first prove that the total variation \(TV\) divergence between the two Gaussian distributionsμ=𝒩⁡\(s1\+δ,σ2​Id\)\\mu=\\mathcal\{N\}\(s\_\{1\}\+\\delta,\\sigma^\{2\}I\_\{d\}\)andν=𝒩⁡\(s1,σ2​Id\)\\nu=\\mathcal\{N\}\(s\_\{1\},\\sigma^\{2\}I\_\{d\}\)is equal to the TV divergence between their corresponding standardized distributionsμ0=𝒩⁡\(δ~,Id\)\\mu\_\{0\}=\\mathcal\{N\}\(\\tilde\{\\delta\},I\_\{d\}\)andν0=𝒩⁡\(0,Id\)\\nu\_\{0\}=\\mathcal\{N\}\(0,I\_\{d\}\), whereδ~=δ/σ\\tilde\{\\delta\}=\\delta/\\sigma\.

By definition, the TV divergence between distributionsμ\\muandν\\nuis given by

DT​V\(μ∥ν\)=12∫ℝd\|p\(x\)−q\(x\)\|dx,D\_\{TV\}\(\\mu\\\|\\nu\)=\\frac\{1\}\{2\}\\int\_\{\\mathbb\{R\}^\{d\}\}\|p\(x\)\-q\(x\)\|dx,wherep⁡\(x\)=1\(2​π​σ2\)d/2​exp⁡\(−12​σ2​‖x−\(s1\+δ\)‖2\)p\(x\)=\\frac\{1\}\{\(2\\pi\\sigma^\{2\}\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\\sigma^\{2\}\}\\\|x\-\(s\_\{1\}\+\\delta\)\\\|^\{2\}\\right\)andq⁡\(x\)=1\(2​π​σ2\)d/2​exp⁡\(−12​σ2​‖x−s1‖2\)q\(x\)=\\frac\{1\}\{\(2\\pi\\sigma^\{2\}\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\\sigma^\{2\}\}\\\|x\-s\_\{1\}\\\|^\{2\}\\right\)denote the probability density functions of the Gaussian distributionsμ\\muandν\\nu, respectively\.

Applying the substitutiony=x−s1σy=\\frac\{x\-s\_\{1\}\}\{\\sigma\}withd​x=σd​d​ydx=\\sigma^\{d\}dy, we obtain:

DT​V\(μ∥ν\)\\displaystyle D\_\{TV\}\(\\mu\\\|\\nu\)\(24\)=12∫ℝd\|1\(2​π​σ2\)d/2exp\(−12​σ2∥σy\+s1−\(s1\+δ\)∥2\)\\displaystyle=\\frac\{1\}\{2\}\\int\_\{\\mathbb\{R\}^\{d\}\}\\left\|\\frac\{1\}\{\(2\\pi\\sigma^\{2\}\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\\sigma^\{2\}\}\\\|\\sigma y\+s\_\{1\}\-\(s\_\{1\}\+\\delta\)\\\|^\{2\}\\right\)\\right\.−1\(2​π​σ2\)d/2exp\(−12​σ2∥σy\+s1−s1∥2\)\|σddy\\displaystyle\\quad\\quad\\left\.\-\\frac\{1\}\{\(2\\pi\\sigma^\{2\}\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\\sigma^\{2\}\}\\\|\\sigma y\+s\_\{1\}\-s\_\{1\}\\\|^\{2\}\\right\)\\right\|\\sigma^\{d\}dy=DT​V\(μ0∥ν0\)\.\\displaystyle=D\_\{TV\}\(\\mu\_\{0\}\\\|\\nu\_\{0\}\)\.
To evaluate the TV divergenceDT​V\(μ0∥ν0\)D\_\{TV\}\(\\mu\_\{0\}\\\|\\nu\_\{0\}\), we analyze the sign of the differencep0​\(x\)−q0​\(x\)p\_\{0\}\(x\)\-q\_\{0\}\(x\), wherep0p\_\{0\}andq0q\_\{0\}denotes the probability density functions ofμ0\\mu\_\{0\}andν0\\nu\_\{0\}respectively\. Computing the density ratio yields:

p0​\(x\)q0​\(x\)\\displaystyle\\frac\{p\_\{0\}\(x\)\}\{q\_\{0\}\(x\)\}=1\(2​π\)d/2​exp⁡\(−12​‖x−δ~‖2\)1\(2​π\)d/2​exp⁡\(−12​‖x‖2\)\\displaystyle=\\frac\{\\frac\{1\}\{\(2\\pi\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\}\\\|x\-\\tilde\{\\delta\}\\\|^\{2\}\\right\)\}\{\\frac\{1\}\{\(2\\pi\)^\{d/2\}\}\\exp\\left\(\-\\frac\{1\}\{2\}\\\|x\\\|^\{2\}\\right\)\}=exp⁡\(−12​‖x−δ~‖2\+12​‖x‖2\)\\displaystyle=\\exp\\left\(\-\\frac\{1\}\{2\}\\\|x\-\\tilde\{\\delta\}\\\|^\{2\}\+\\frac\{1\}\{2\}\\\|x\\\|^\{2\}\\right\)=exp⁡\(−12​\(‖x‖2−2​xT​δ~\+‖δ~‖2\)\+12​‖x‖2\)\\displaystyle=\\exp\\left\(\-\\frac\{1\}\{2\}\(\\\|x\\\|^\{2\}\-2x^\{T\}\\tilde\{\\delta\}\+\\\|\\tilde\{\\delta\}\\\|^\{2\}\)\+\\frac\{1\}\{2\}\\\|x\\\|^\{2\}\\right\)=exp⁡\(xT​δ~−12​‖δ~‖2\)\.\\displaystyle=\\exp\\left\(x^\{T\}\\tilde\{\\delta\}\-\\frac\{1\}\{2\}\\\|\\tilde\{\\delta\}\\\|^\{2\}\\right\)\.It follows thatp0​\(x\)\>q0​\(x\)p\_\{0\}\(x\)\>q\_\{0\}\(x\)if and only ifxT​δ~\>12​‖δ~‖2x^\{T\}\\tilde\{\\delta\}\>\\frac\{1\}\{2\}\\\|\\tilde\{\\delta\}\\\|^\{2\}, which defines a half\-spaceH=\{x∈ℝd:xT​δ~\>12​‖δ~‖2\}H=\\\{x\\in\\mathbb\{R\}^\{d\}:x^\{T\}\\tilde\{\\delta\}\>\\frac\{1\}\{2\}\\\|\\tilde\{\\delta\}\\\|^\{2\}\\\}\. As a result, the TV divergence can be expressed as

DT​V\(μ0∥ν0\)\\displaystyle D\_\{TV\}\(\\mu\_\{0\}\\\|\\nu\_\{0\}\)=12​∫ℝd\|p0​\(x\)−q0​\(x\)\|​𝑑x\\displaystyle=\\frac\{1\}\{2\}\\int\_\{\\mathbb\{R\}^\{d\}\}\|p\_\{0\}\(x\)\-q\_\{0\}\(x\)\|dx=12​\[∫H\(p0​\(x\)−q0​\(x\)\)​𝑑x\+∫Hc\(q0​\(x\)−p0​\(x\)\)​𝑑x\]\\displaystyle=\\frac\{1\}\{2\}\\left\[\\int\_\{H\}\(p\_\{0\}\(x\)\-q\_\{0\}\(x\)\)dx\+\\int\_\{H^\{c\}\}\(q\_\{0\}\(x\)\-p\_\{0\}\(x\)\)dx\\right\]=∫Hp0​\(x\)​𝑑x−∫Hq0​\(x\)​𝑑x\\displaystyle=\\int\_\{H\}p\_\{0\}\(x\)dx\-\\int\_\{H\}q\_\{0\}\(x\)dx=P⁡\(H\)−Q⁡\(H\),\\displaystyle=P\(H\)\-Q\(H\),whereP⁡\(H\)P\(H\)andQ⁡\(H\)Q\(H\)denote denote the probabilities of the half\-space underμ0\\mu\_\{0\}andν0\\nu\_\{0\}, respectively\.

We next compute these probabilities by analyzing the distributions of linear combinationsX⊤​δ~X^\{\\top\}\\tilde\{\\delta\}andY⊤​δ~Y^\{\\top\}\\tilde\{\\delta\}respectively, whereX=\(X1,⋯,Xd\)∼𝒩⁡\(δ~,Id\)X=\(X\_\{1\},\\cdots,X\_\{d\}\)\\sim\\mathcal\{N\}\(\\tilde\{\\delta\},I\_\{d\}\)andY=\(Y1,⋯,Yd\)∼𝒩⁡\(0,Id\)Y=\(Y\_\{1\},\\cdots,Y\_\{d\}\)\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\. SinceX⊤​δ~∼𝒩⁡\(‖δ~‖2,‖δ~‖2\)X^\{\\top\}\\tilde\{\\delta\}\\sim\\mathcal\{N\}\(\\\|\\tilde\{\\delta\}\\\|^\{2\},\\\|\\tilde\{\\delta\}\\\|^\{2\}\)andY⊤​δ~∼𝒩⁡\(0,‖δ~‖2\)Y^\{\\top\}\\tilde\{\\delta\}\\sim\\mathcal\{N\}\(0,\\\|\\tilde\{\\delta\}\\\|^\{2\}\), we obtain:

P\(H\)=ℙ\[X⊤δ~\>‖δ~‖22\]=Φ\(‖δ~‖2\),P\(H\)=\\mathbb\{P\}\\left\[X^\{\\top\}\\tilde\{\\delta\}\>\\frac\{\\\|\\tilde\{\\delta\}\\\|^\{2\}\}\{2\}\\right\]=\\Phi\\left\(\\frac\{\\\|\\tilde\{\\delta\}\\\|\}\{2\}\\right\),and

Q\(H\)=ℙ\[Y⊤δ~\>‖δ~‖22\]=1−Φ\(‖δ~‖2\),Q\(H\)=\\mathbb\{P\}\\left\[Y^\{\\top\}\\tilde\{\\delta\}\>\\frac\{\\\|\\tilde\{\\delta\}\\\|^\{2\}\}\{2\}\\right\]=1\-\\Phi\\left\(\\frac\{\\\|\\tilde\{\\delta\}\\\|\}\{2\}\\right\),whereΦ⁡\(⋅\)\\Phi\(\\cdot\)denotes the CDF of the standard normal distribution𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\. Hence, the total variation is given by

DT​V\(μ∥ν\)=DT​V\(μ0∥ν0\)=P\(H\)−Q\(H\)=2Φ\(‖δ~‖2\)−1\.D\_\{TV\}\(\\mu\\\|\\nu\)=D\_\{TV\}\(\\mu\_\{0\}\\\|\\nu\_\{0\}\)=P\(H\)\-Q\(H\)=2\\Phi\\left\(\\frac\{\\\|\\tilde\{\\delta\}\\\|\}\{2\}\\right\)\-1\.\(25\)Substitutingδ~=δσ\\tilde\{\\delta\}=\\frac\{\\delta\}\{\\sigma\}into \([25](https://arxiv.org/html/2609.10866#S8.E25)\) gives that

DT​V\(μ∥ν\)=2Φ\(‖δ‖22​σ\)−1\.D\_\{TV\}\(\\mu\\\|\\nu\)=2\\Phi\\left\(\\frac\{\\\|\\delta\\\|\_\{2\}\}\{2\\sigma\}\\right\)\-1\.Since‖δ‖2≤‖δ‖1≤ϵ\\\|\\delta\\\|\_\{2\}\\leq\\\|\\delta\\\|\_\{1\}\\leq\\epsilonandΦ\\Phiis monotonically increasing, we obtain:

DT​V\(μ∥ν\)=2Φ\(‖δ‖22​σ\)−1≤2Φ\(‖δ‖12​σ\)−1≤2Φ\(ϵ2​σ\)−1\.D\_\{TV\}\(\\mu\\\|\\nu\)=2\\Phi\\left\(\\frac\{\\\|\\delta\\\|\_\{2\}\}\{2\\sigma\}\\right\)\-1\\leq 2\\Phi\\left\(\\frac\{\\\|\\delta\\\|\_\{1\}\}\{2\\sigma\}\\right\)\-1\\leq 2\\Phi\\left\(\\frac\{\\epsilon\}\{2\\sigma\}\\right\)\-1\.This completes the proof\.

## References

- B\. Balle and Y\. WangImproving the gaussian mechanism for differential privacy: analytical calibration and optimal denoising\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 394–403\.Cited by:[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p14.2)\.
- Behzadan and Munir \(2017a\)V\. Behzadan and A\. MunirVulnerability of deep reinforcement learning to policy induction attacks\.InInternational Conference on Machine Learning and Data Mining in Pattern Recognition \(MLDM\),New York, NY, USA,pp\. 262–275\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.10866#S2.SS1.p1.1)\.
- Behzadan and Munir \(2017b\)V\. Behzadan and A\. MunirWhatever does not kill deep reinforcement learning, makes it stronger\.External Links:1712\.09344,[Link](http://arxiv.org/abs/1712.09344)Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p2.1)\.
- Ben\-Talet al\.\(2013\)A\. Ben\-Tal, D\. den Hertog, A\. De Waegenaere, B\. Melenberg, and G\. RennenRobust solutions of optimization problems affected by uncertain probabilities\.Management Science59\(2\),pp\. 341–357\.Cited by:[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p2.2)\.
- Ben\-Tal and Teboulle \(2007\)A\. Ben\-Tal and M\. TeboulleAn old\-new concept of convex risk measures: the optimized certainty equivalent\.Mathematical Finance17\(3\),pp\. 449–476\.External Links:[Link](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9965.2007.00311.x)Cited by:[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p5.4.1)\.
- Diamond and Boyd \(2016\)S\. Diamond and S\. BoydCVXPY: a python\-embedded modeling language for convex optimization\.Journal of Machine Learning Research17\(83\),pp\. 1–5\.Cited by:[§4\.2](https://arxiv.org/html/2609.10866#S4.SS2.p5.1)\.
- Dvijothamet al\.\(2020\)K\. Dvijotham, J\. Hayes, B\. Balle, J\. Z\. Kolter, C\. Qin, A\. György, K\. Xiao, S\. Gowal, and P\. KohliA framework for robustness certification of smoothed classifiers using f\-divergences\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p14.2)\.
- Everettet al\.\(2022\)M\. Everett, B\. Lütjens, and J\. P\. HowCertifiable robustness to adversarial state uncertainty in deep reinforcement learning\.IEEE Transactions on Neural Networks and Learning Systems33\(9\),pp\. 4184–4198\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2021.3058699)Cited by:[§3\.2](https://arxiv.org/html/2609.10866#S3.SS2.SSS0.Px1.p3.1)\.
- Hansen and Sargent \(2001\)L\. P\. Hansen and T\. J\. SargentRobust control and model uncertainty\.American Economic Review91\(2\),pp\. 60–66\.Cited by:[§3\.1](https://arxiv.org/html/2609.10866#S3.SS1.p3.1)\.
- Heet al\.\(2022\)X\. He, B\. Lou, H\. Yang, and C\. LvRobust decision making for autonomous vehicles at highway on\-ramps: a constrained adversarial reinforcement learning approach\.IEEE Transactions on Intelligent Transportation Systems24\(4\),pp\. 4103–4113\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1)\.
- Howard and Matheson \(1972\)R\. A\. Howard and J\. E\. MathesonRisk\-sensitive markov decision processes\.Management Science18\(7\),pp\. 356–369\.Cited by:[§3\.1](https://arxiv.org/html/2609.10866#S3.SS1.p3.1)\.
- Huanget al\.\(2017\)S\. Huang, N\. Papernot, I\. Goodfellow, Y\. Duan, and P\. AbbeelAdversarial attacks on neural network policies\.External Links:1702\.02284Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1)\.
- Jaquette \(1976\)S\. C\. JaquetteA utility criterion for markov decision processes\.Management Science23\(1\),pp\. 43–49\.Cited by:[§3\.1](https://arxiv.org/html/2609.10866#S3.SS1.p3.1)\.
- Kos and Song \(2017\)J\. Kos and D\. SongDelving into adversarial attacks on deep policies\.External Links:1705\.06452Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p2.1)\.
- Kumaret al\.\(2021\)A\. Kumar, A\. Levine, and S\. FeiziPolicy smoothing for provably robust reinforcement learning\.External Links:2106\.11420,[Link](https://api.semanticscholar.org/CorpusID:235593394)Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.10866#S2.SS2.p3.1),[§3\.2](https://arxiv.org/html/2609.10866#S3.SS2.SSS0.Px1.p3.1),[§3\.2](https://arxiv.org/html/2609.10866#S3.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p9.1)\.
- Lianget al\.\(2022\)Y\. Liang, Y\. Sun, R\. Zheng, and F\. HuangEfficient adversarial training without attacking: worst\-case\-aware robust reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 22547–22561\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1)\.
- Lütjenset al\.\(2020\)B\. Lütjens, M\. Everett, and J\. P\. HowCertified adversarial robustness for deep reinforcement learning\.InProceedings of the 4th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.100,pp\. 1328–1337\.External Links:[Link](https://proceedings.mlr.press/v100/lutjens20a.html)Cited by:[§2\.2](https://arxiv.org/html/2609.10866#S2.SS2.p1.1)\.
- Muet al\.\(2024\)R\. Mu, L\. S\. Marcolino, Y\. Zhang, T\. Zhang, X\. Huang, and W\. RuanReward certification for policy smoothed reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 21429–21437\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p3.1),[§1](https://arxiv.org/html/2609.10866#S1.p4.1),[§2\.2](https://arxiv.org/html/2609.10866#S2.SS2.p4.1),[§3\.2](https://arxiv.org/html/2609.10866#S3.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p14.2),[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p4.2),[§4\.1](https://arxiv.org/html/2609.10866#S4.SS1.p9.1),[§4\.2](https://arxiv.org/html/2609.10866#S4.SS2.p4.1)\.
- Pattanaiket al\.\(2017\)A\. Pattanaik, Z\. Tang, S\. Liu, G\. Bommannan, and G\. ChowdharyRobust deep reinforcement learning with adversarial attacks\.External Links:1712\.03632Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1),[§1](https://arxiv.org/html/2609.10866#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.10866#S2.SS1.p1.1)\.
- Puterman \(1994\)M\. L\. PutermanMarkov decision processes: discrete stochastic dynamic programming\.John Wiley & Sons,New York, NY, USA\.Cited by:[§5\.2](https://arxiv.org/html/2609.10866#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2609.10866#S5.SS2.p5.1),[§5](https://arxiv.org/html/2609.10866#S5.p3.1)\.
- Sallabet al\.\(2017\)A\. E\. Sallab, M\. Abdou, E\. Perot, and S\. YogamaniDeep reinforcement learning framework for autonomous driving\.External Links:1704\.02532Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1)\.
- Shalev\-Shwartzet al\.\(2016\)S\. Shalev\-Shwartz, S\. Shammah, and A\. ShashuaSafe, multi\-agent, reinforcement learning for autonomous driving\.External Links:1610\.03295Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1)\.
- Shenet al\.\(2020\)Q\. Shen, Y\. Li, H\. Jiang, Z\. Wang, and T\. ZhaoDeep reinforcement learning with robust and smooth policy\.InInternational Conference on Machine Learning \(ICML\),pp\. 8707–8718\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.10866#S2.SS1.p3.1)\.
- Towerset al\.\(2024\)M\. Towerset al\.Gymnasium: a standard interface for reinforcement learning environments\.External Links:2407\.17032Cited by:[§5\.1](https://arxiv.org/html/2609.10866#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2609.10866#S5.SS1.p5.1),[§5](https://arxiv.org/html/2609.10866#S5.p3.1)\.
- Vershynin \(2018\)R\. VershyninHigh\-dimensional probability: an introduction with applications in data science\.Cambridge University Press,Cambridge\.Cited by:[§4\.2](https://arxiv.org/html/2609.10866#S4.SS2.p3.1)\.
- Wuet al\.\(2022\)F\. Wu, L\. Li, Z\. Huang, Y\. Vorobeychik, D\. Zhao, and B\. LiCROP: certifying robust policies for reinforcement learning through functional smoothing\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.10866#S2.SS2.p2.1),[§2\.2](https://arxiv.org/html/2609.10866#S2.SS2.p3.1)\.
- Zhanget al\.\(2020\)H\. Zhang, H\. Chen, C\. Xiao, B\. Li, M\. Liu, D\. Boning, and C\. HsiehRobust deep reinforcement learning against adversarial perturbations on state observations\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 21024–21037\.Cited by:[§1](https://arxiv.org/html/2609.10866#S1.p1.1),[§1](https://arxiv.org/html/2609.10866#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.10866#S2.SS1.p2.1)\.