Quantile Geometry Regularization for Distributional Reinforcement Learning

# Quantile Geometry Regularization for Distributional Reinforcement Learning
Source: [https://arxiv.org/html/2605.08182](https://arxiv.org/html/2605.08182)
Zhaofan Zhang, Minghao Yang, Rufeng Chen, Sihong Xie, Hui Xiong Information Hub, AI Thrust Hong Kong University of Science and Technology \(Guangzhou\) Guangzhou, Guangdong, China

###### Abstract

Quantile\-based distributional reinforcement learning methods learn return distributions through sampled quantile regression, but their bootstrapped target quantiles may induce distorted or degenerate distribution estimates\. We propose Robust Quantile\-based Implicit Quantile Networks \(RQIQN\), a lightweight Wasserstein distributionally robust enhancement boosted from a quantile estimation perspective\. We first reinterpret a snapshot of IQN loss as a collection of local empirical quantile estimation problems over sampled current fractions\. We then robustify each local slot with a Wasserstein distributionally robust quantile estimation formulation, yielding a closed\-form, fraction\-dependent correction to the Bellman target\. This correction directly addresses distributional degeneration: its median\-antisymmetry preserves the risk\-neutral quantile average, while its monotonicity enlarges upper–lower quantile gaps and counteracts collapsed distributional spread\. RQIQN thus regularizes quantile geometry without changing the underlying value objective or requiring additional sample\-set reconstruction\. Finally, we empirically show that the proposed RQIQN outperforms other existing quantile\-based distributional reinforcement learning algorithms in risk\-sensitive navigation and Atari games\.

## 1Introduction

Distributional reinforcement learning \(DistRL\) models the full distribution of discounted returns rather than only its expectation, providing a richer basis for learning under stochasticity and risk\. A useful way to understand practical DistRL algorithms is through the distinction between return distributions and statistics of those distributions\. The distributional Bellman operator acts on full return distributions, whereas scalable algorithms usually propagate finite\-dimensional summaries, such as atoms, quantiles, or expectiles\. Existing DistRL algorithms differ mainly in how they represent and update return distributions, including categorical atoms, quantile functions, expectiles, and sample\-set representations\. Among these choices, quantile\-based methods are particularly appealing because quantile functions naturally align with Wasserstein geometry, do not require a fixed value support, and provide direct access to different regions of the return distribution\. Within this scope, Implicit Quantile Networks \(IQN\)Dabneyet al\.\([2018a](https://arxiv.org/html/2605.08182#bib.bib3)\)further extends QR\-DQNDabneyet al\.\([2018b](https://arxiv.org/html/2605.08182#bib.bib4)\)with fixed\-grid quantiles by learning a continuous quantile function through sampled quantile fractionsτ∼U​\(\[0,1\]\)\\tau\\sim U\(\[0,1\]\)\. Generally, a set of statistics is Bellman\-closed if the statistics of the Bellman\-updated distribution can be computed solely from the same statistics of the next\-state distribution and the reward, without reconstructing the full distribution\. This nature is crucial since it allows recursive dynamic programming \(Eq\. \([1](https://arxiv.org/html/2605.08182#S2.E1)\)\) in the chosen statistic space\. However, finite quantile statistics are not Bellman\-closedRowlandet al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib7)\), which can lead to distribution degeneration and biased estimates of distributional spread \(Figure[1](https://arxiv.org/html/2605.08182#S1.F1)\(a\)\)\.

In quantile\-based DistRL, deep Q\-learningMnihet al\.\([2013](https://arxiv.org/html/2605.08182#bib.bib1)\)is extended from scalar value regression to distributional quantile fitting\. For each transition, the agent forms pairwise TD errors between current quantile estimates and bootstrapped target quantiles, and minimizes a quantile regression loss \(Eq\. \([3](https://arxiv.org/html/2605.08182#S2.E3)\)\)\. Thus, the TD update has an explicit quantile\-estimation interpretation, with each sampled fraction corresponding to a local quantile fitting problem\. In this work, we revisit IQN through the lens of local quantile estimation\. Specifically, the IQN loss \(Eq\.\([3](https://arxiv.org/html/2605.08182#S2.E3)\)\) can be interpreted exactly at the empirical\-loss level as a collection of empirical quantile\-estimation slots, one for each sampled current quantile fractionτi\\tau\_\{i\}\. Each slot fits theτi\\tau\_\{i\}located quantile of the bootstrapped Bellman target induced by target quantile fractions\{τj′\}j=1N′\\\{\\tau^\{\\prime\}\_\{j\}\\\}\_\{j=1\}^\{N^\{\\prime\}\}\. This view reveals that IQN performs local quantile estimation over a model\-generated empirical target distribution formed by finitely many bootstrapped target quantile values\. From this perspective, distribution degeneration can be understood as accumulated bias in the learned quantile values, which distorts the geometry of the represented return distribution\.

We propose RQIQN \(Figure[1](https://arxiv.org/html/2605.08182#S1.F1)\(b\)\) to address the distribution degeneration problem, namely biased quantile representation, by robustifying each local quantile fitting problem against worst\-case perturbations of the empirical Bellman target distribution within a Wasserstein ambiguity set\. The resulting closed\-form quantile correction regularizes the learned return distribution by stabilizing finite\-sample quantile estimates\. Instead of introducing computationally intensive schemes by modifying the distributional Bellman operator or distributional representation in existing workRowlandet al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib7)\); Jullienet al\.\([2023](https://arxiv.org/html/2605.08182#bib.bib5)\); Nguyenet al\.\([2020](https://arxiv.org/html/2605.08182#bib.bib17)\), RQIQN replaces each local quantile regression slot with a Wasserstein distributionally robust quantile estimation problem around the empirical Bellman target law\. For the check\-loss formulation111Namely, quantile loss\. It’s different from quantile Huber loss \(Huberized quantile regression loss\)\., this robust slot admits a closed\-form fraction\-dependent correctionΔp​\(τ;ϵ\)\\Delta\_\{p\}\(\\tau;\\epsilon\), whereppis the Wasserstein order, andϵ\\epsilonis the robustness radius that decays over time\. Geometrically, the correction is median\-antisymmetric and monotone inτ\\tau\. Hence it preserves the uniform quantile average while expanding upper–lower quantile gaps, providing a mean\-neutral regularization of distributional spread\.

![Refer to caption](https://arxiv.org/html/2605.08182v1/x1.png)Figure 1:An illustration of\(a\)distribution degeneration at state0and\(b\)how the proposed RQIQN correction modulates quantile geometry\. The samples visualization of fitted return distributions are from a four\-state chain MDP with deterministic transitions under a unique action\. State transitions are directional and sequential, progressing from state0to state33\. Rewards are zero except at the terminal state, wherer∼12​𝒩​\(−2,1\)\+12​𝒩​\(2,1\)r\\sim\\frac\{1\}\{2\}\\mathcal\{N\}\(\-2,1\)\+\\frac\{1\}\{2\}\\mathcal\{N\}\(2,1\)\. In\(b\), each agent uses its default training loss\.### 1\.1Related Work

DistRL has developed rapidly as a powerful alternative to expectation\-based value learning, with representative approaches modeling return distributions using categorical atoms, quantiles, expectiles, or sample\-based representationsBellemareet al\.\([2017](https://arxiv.org/html/2605.08182#bib.bib2)\); Dabneyet al\.\([2018b](https://arxiv.org/html/2605.08182#bib.bib4),[a](https://arxiv.org/html/2605.08182#bib.bib3)\); Rowlandet al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib7)\); Yanget al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib8)\)\. Among them, IQNDabneyet al\.\([2018a](https://arxiv.org/html/2605.08182#bib.bib3)\)has become particularly influential due to its flexible implicit quantile\-function representation, which supports arbitrary quantile sampling and naturally enables risk\-sensitive policies through distortion risk measures\. IQN has been widely applied in tasks that require risk\-sensitive decision\-making, such as autonomous surface navigationLinet al\.\([2023](https://arxiv.org/html/2605.08182#bib.bib13)\); Zhanget al\.\([2025](https://arxiv.org/html/2605.08182#bib.bib11)\)and quadrupedal locomotionShiet al\.\([2024](https://arxiv.org/html/2605.08182#bib.bib14)\)\. However, IQN still inherits a limitation of quantile\-based DistRL: finite quantile statistics are not Bellman\-closed\.

Several lines of work have addressed the difficulty caused by non\-Bellman\-closed distributional statistics by either changing the learned statistics or introducing an explicit statistics\-to\-samples interface\. Expectiles, introduced as asymmetric least\-squares location statisticsNewey and Powell \([1987](https://arxiv.org/html/2605.08182#bib.bib18)\), are attractive because they provide smootherL2L\_\{2\}\-based fitting than check\-loss quantile regression, and their symmetric caseτ=1/2\\tau=1/2coincides with the mean\. This property makes expectiles, useful for stable value estimation, since the mean is the statistic used for risk\-neutral action selection\. However, a finite set of expectiles is still not Bellman\-closed in general, and expectile values cannot be directly interpreted as samples from the return distribution\. To make expectiles usable for Bellman backups, ER\-DQNRowlandet al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib7)\)recover Bellman target samples through an imputation step, which requires solving costly nonlinear systems\. IEQNJullienet al\.\([2023](https://arxiv.org/html/2605.08182#bib.bib5)\)instead jointly learns expectiles and quantiles to retain efficientL2L\_\{2\}\-based learning while avoiding explicit imputation, at the cost of additional prediction heads and auxiliary coupling losses\. Sample\-set methods, such as MMDNguyenet al\.\([2020](https://arxiv.org/html/2605.08182#bib.bib17)\)and MWGZhanget al\.\([2024](https://arxiv.org/html/2605.08182#bib.bib16)\), take a different route by operating on explicit return samples, thereby bypassing the need to propagate non\-closed statistics as samples\. However, deterministic sample sets may limit stochastic target diversity, while Gaussian\-mixture variants rely on EM\-style projectionDempsteret al\.\([1977](https://arxiv.org/html/2605.08182#bib.bib19)\)and sample augmentation, introducing additional computational and algorithmic complexity\.

## 2Problem Setup and Main Analysis

### 2\.1Distributional Reinforcement Learning

We consider a Markov decision process \(MDP\)ℳ=\(𝒮,𝒜,𝒫,ℝ,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{P\},\\mathbb\{R\},\\gamma\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,𝒫\(⋅∣s,a\)\\mathcal\{P\}\(\\cdot\\mid s,a\)is the transition kernel,ℝ\(⋅∣s,a\)\\mathbb\{R\}\(\\cdot\\mid s,a\)denotes the reward distribution, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. For a policyπ\\pi, the state\-action return is the random variableZπ​\(s,a\)​=𝐷​∑t=0∞γt​RtZ^\{\\pi\}\(s,a\)\\overset\{D\}\{=\}\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{t\},S0=sS\_\{0\}=s,A0=aA\_\{0\}=a, where=𝐷\\overset\{D\}\{=\}denotes equality in distribution and the transition process can be explained bySt\+1∼𝒫\(⋅∣St,At\)S\_\{t\+1\}\\sim\\mathcal\{P\}\(\\cdot\\mid S\_\{t\},A\_\{t\}\),Rt∼ℝ\(⋅∣St,At\)R\_\{t\}\\sim\\mathbb\{R\}\(\\cdot\\mid S\_\{t\},A\_\{t\}\),At∼π\(⋅∣St\)A\_\{t\}\\sim\\pi\(\\cdot\\mid S\_\{t\}\)fort≥1t\\geq 1\. Standard reinforcement learning typically optimizes only the first moment of this return distribution\. In particular, the action\-value function is defined asQπ​\(s,a\)=𝔼Z​\[Zπ​\(s,a\)\]\.Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{Z\}\\\!\\left\[Z^\{\\pi\}\(s,a\)\\right\]\.The optimal action\-value function satisfies the Bellman optimality equationQ\(s,a\)=𝔼R∼ℝ\(⋅∣s,a\),S′∼𝒫\(⋅∣s,a\)\[R\+γmaxa′∈𝒜Q\(S′,a′\)\|S=s,A=a\]Q\(s,a\)=\\mathbb\{E\}\_\{R\\sim\\mathbb\{R\}\(\\cdot\\mid s,a\),\\,S^\{\\prime\}\\sim\\mathcal\{P\}\(\\cdot\\mid s,a\)\}\\\!\\left\[R\+\\gamma\\max\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}Q\(S^\{\\prime\},a^\{\\prime\}\)\\,\\middle\|\\,S=s,A=a\\right\]\. This expectation\-based formulation compresses the full return distribution into a single scalar, discarding distributional information induced by stochastic transitions and rewards\.

DistRL extends standard RL by modeling the*return distribution*ZZinstead of only its expectation\. Formally,ZZcan be updated by dynamic programming, with distributional Bellman optimality operator222Uppercase lettersSSandAAdenote random variables, and lowercase letters denote their realized values\.defined by

𝒯π​Z​\(s,a\)=DR​\(s,a\)\+γ​Z​\(S′,A′\)\.\\mathcal\{T\}^\{\\pi\}Z\(s,a\)\\stackrel\{\{\\scriptstyle D\}\}\{\{=\}\}R\(s,a\)\+\\gamma Z\(S^\{\\prime\},A^\{\\prime\}\)\.\(1\)We can compute the optimal return distribution by using the distributional Bellman optimality operator𝒯\\mathcal\{T\}defined as

𝒯​Z​\(s,a\)=DR​\(s,a\)\+γ​Z​\(S′,a∗\),a∗=argmaxa′𝔼Z​\[Z​\(S′,a′\)\]\.\\mathcal\{T\}Z\(s,a\)\\stackrel\{\{\\scriptstyle D\}\}\{\{=\}\}R\(s,a\)\+\\gamma Z\\left\(S^\{\\prime\},a^\{\*\}\\right\),\\qquad a^\{\*\}=\\operatorname\*\{argmax\}\_\{a^\{\\prime\}\}\\mathbb\{E\}\_\{Z\}\[Z\(S^\{\\prime\},a^\{\\prime\}\)\]\.\(2\)As a representative quantile\-based DistRL method, Implicit Quantile Networks \(IQN\)Dabneyet al\.\([2018a](https://arxiv.org/html/2605.08182#bib.bib3)\)provides a flexible distributional representation through the quantile function of the return distribution\. LetFZ​\(s,a\)−1​\(τ\)F^\{\-1\}\_\{Z\(s,a\)\}\(\\tau\)denote the quantile function of the random returnZ​\(s,a\)Z\(s,a\)at quantile fractionτ\\tau\. For notational simplicity, we writeZτ​\(s,a\):=FZ​\(s,a\)−1​\(τ\)Z\_\{\\tau\}\(s,a\):=F^\{\-1\}\_\{Z\(s,a\)\}\(\\tau\), so that whenτ∼U​\(\[0,1\]\)\\tau\\sim U\(\[0,1\]\), the sampled quantile valueZτ​\(s,a\)Z\_\{\\tau\}\(s,a\)follows the return distributionZ​\(s,a\)Z\(s,a\)\. Based on this implicit quantile representation, IQN parameterizes this quantile function with a neural networkZτ​\(x,a;θ\)≈Zτ​\(s,a\)Z\_\{\\tau\}\(x,a;\\theta\)\\approx Z\_\{\\tau\}\(s,a\)\. For brevity, we useZτ​\(s,a\)Z\_\{\\tau\}\(s,a\)directly\. Since IQN samples quantile fractions explicitly, risk\-sensitive behavior can be induced by modifying the quantile sampling range\. A common choice is the conditional value\-at\-risk \(CVaR\) distortion, which maps a uniformly sampled fractionτ∼U​\(\[0,1\]\)\\tau\\sim U\(\[0,1\]\)toτ~=η​τ\\tilde\{\\tau\}=\\eta\\tau, thereby restricting sampling to the lower\-tail quantile regionU​\(\[0,η\]\)U\(\[0,\\eta\]\)for risk\-averse control\. In this work, unless explicitly stated, fractions are sampled without distortion\.

Throughout this work, we consider two quantile regression losses related to DistRL\. The first is the check loss,

ρτ​\(u\)=u​\(τ−𝟙\{u<0\}\)=\|τ−𝟙\{u<0\}\|​\|u\|,\\rho\_\{\\tau\}\(u\)=u\\left\(\\tau\-\\mathbbm\{1\}\_\{\\\{u<0\\\}\}\\right\)=\\left\|\\tau\-\\mathbbm\{1\}\_\{\\\{u<0\\\}\}\\right\|\\,\|u\|,which gives the exact quantile\-regression objective\. The second is the standard quantile HuberHuber \([1992](https://arxiv.org/html/2605.08182#bib.bib15)\)loss used in practical QR\-DQN/IQN implementations:

ρτκ​\(u\)=\|τ−𝟙\{u<0\}\|​ℋκ​\(u\)κ,ℋκ​\(u\)=\{12​u2,\|u\|≤κ,κ​\(\|u\|−12​κ\),\|u\|\>κ\.\\rho\_\{\\tau\}^\{\\kappa\}\(u\)=\\left\|\\tau\-\\mathbbm\{1\}\_\{\\\{u<0\\\}\}\\right\|\\frac\{\\mathcal\{H\}\_\{\\kappa\}\(u\)\}\{\\kappa\},\\qquad\\mathcal\{H\}\_\{\\kappa\}\(u\)=\\begin\{cases\}\\frac\{1\}\{2\}u^\{2\},&\|u\|\\leq\\kappa,\\\\\[2\.84526pt\] \\kappa\\left\(\|u\|\-\\frac\{1\}\{2\}\\kappa\\right\),&\|u\|\>\\kappa\.\\end\{cases\}Here,κ\>0\\kappa\>0is the Huber threshold\. The check loss preserves the exact quantile\-estimation interpretation, while the Huberized loss provides a smoother surrogate for numerical stability and obtaining higher return\.

### 2\.2Distribution Degeneration with Quantile\-based Representation

In quantile\-based DistRL, the distributional Bellman update is implemented through TD target fitting, where current quantile estimates are regressed toward bootstrapped target quantiles\. However, finite quantile sets are not Bellman\-closed in general, and quantiles are statistics rather than samples\. Thus, unless the Bellman\-updated distribution is explicitly projected back to the quantile representation or reconstructed through an imputed target distribution, approximation bias may accumulate and distort the learned return distributionRowlandet al\.\([2019](https://arxiv.org/html/2605.08182#bib.bib7)\)\. In practice, this issue is exacerbated by the Huberized quantile regression loss, which no longer preserves the exact guarantees of quantile regression\. The learned return distribution may consequently collapse toward its mean, discarding tail information\.

The degeneration risk of return distribution motivates a robustness\-oriented view of quantile\-based distributional learning, in which mitigating degeneration amounts to stabilizing the quantile estimates produced by the proceeding of Bellman updates\. In Figure[1](https://arxiv.org/html/2605.08182#S1.F1), IQN \(depicted in blue\) suffers from worse variance degradation, whereas the RQIQN under two loss settings demonstrates robustness in terms of distribution geometry\. The detailed formulation of RQIQN is provided in Section[3](https://arxiv.org/html/2605.08182#S3)\.

### 2\.3The Quantile Estimation Inspired Loss

Given a transition\(s,a,r,s′\)\(s,a,r,s^\{\\prime\}\), IQN samples current fractions\{τi\}i=1N\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}for the quantile levels to be fitted at\(s,a\)\(s,a\)and target fractions\{τj′\}j=1N′\\\{\\tau^\{\\prime\}\_\{j\}\\\}\_\{j=1\}^\{N^\{\\prime\}\}for bootstrapped next\-state quantile targets, both fromU​\(\[0,1\]\)U\(\[0,1\]\), and defines the IQN loss with TD errorδi​j\\delta\_\{ij\}as

ℒIQN​\(s,a,r,s′\)=1N′​∑i=1N∑j=1N′ρτiκ​\(δi​j\)=∑i=1N𝔼τ′​\[ρτiκ​\(δi​j\)\],δi​j=r\+γ​Zτj′​\(s′,a′\)−Zτi​\(s,a\)\.\\mathcal\{L\}\_\{\\mathrm\{IQN\}\}\(s,a,r,s^\{\\prime\}\)=\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\rho^\{\\kappa\}\_\{\\tau\_\{i\}\}\\bigl\(\\delta\_\{ij\}\\bigr\)=\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\_\{\\tau^\{\\prime\}\}\\left\[\\rho\_\{\\tau\_\{i\}\}^\{\\kappa\}\\left\(\\delta\_\{ij\}\\right\)\\right\],\\quad\\delta\_\{ij\}=r\+\\gamma Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\\prime\}\)\-Z\_\{\\tau\_\{i\}\}\(s,a\)\.\(3\)Similar to the TD loss in deep Q\-learningMnihet al\.\([2013](https://arxiv.org/html/2605.08182#bib.bib1)\), it is mentioned in QR\-DQNDabneyet al\.\([2018b](https://arxiv.org/html/2605.08182#bib.bib4)\)that a nature in the loss function of quantile\-based DistRL is to employ quantile regression temporal difference learning \(QRTD\)\. The update is computed for all pairs of\(Zτj′​\(s′,a′\),Zτi​\(s,a\)\)\(Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\\prime\}\),Z\_\{\\tau\_\{i\}\}\(s,a\)\)\. At its essence, quantile estimation seeks the value corresponding to a given quantile levelτ\\taufrom samples drawn from a distribution\. With respect to the TD error in Eq\. \([3](https://arxiv.org/html/2605.08182#S2.E3)\), if the quantile fractionτi\\tau\_\{i\}is fixed, the update target is regarded as a sample, then the update can be interpreted as estimating the quantile at levelτi\\tau\_\{i\}\. Therefore, the overall IQN loss performs quantile estimation over all sampled quantile levels during the distributional Bellman update, which in turn yields a representation of the return distribution\.

###### Lemma 1\(Snapshot of Implicit Quantile Network loss as quantile estimation\)\.

For any transition\(s,a,r,s′\)\(s,a,r,s^\{\\prime\}\), consider a snapshot𝔼τ′​\[ρτiκ​\(δi​j\)\]\\mathbb\{E\}\_\{\\tau^\{\\prime\}\}\\left\[\\rho\_\{\\tau\_\{i\}\}^\{\\kappa\}\\left\(\\delta\_\{ij\}\\right\)\\right\]of IQN loss in Eq\. \([3](https://arxiv.org/html/2605.08182#S2.E3)\) with quantile fraction samplesτj′∼U​\(\[0,1\]\)\\tau^\{\\prime\}\_\{j\}\\sim U\(\[0,1\]\)and a fixed oneτi∈\(0,1\)\\tau\_\{i\}\\in\(0,1\):

δi​j=yj−qi,yj:=r\+γ​Zτj′​\(s′,a′\),qi:=Zτi​\(s,a\)\.\\delta\_\{ij\}=y\_\{j\}\-q\_\{i\},\\qquad y\_\{j\}:=r\+\\gamma Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\\prime\}\),\\qquad q\_\{i\}:=Z\_\{\\tau\_\{i\}\}\(s,a\)\.
Let the empirical Bellman target beμ^s,a=1N′​∑j=1N′δyj\\widehat\{\\mu\}\_\{s,a\}=\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\delta\_\{y\_\{j\}\}, whereδyj\\delta\_\{y\_\{j\}\}denotes a Dirac atyjy\_\{j\}\. The empirical quantile regression slot of Eq\. \([3](https://arxiv.org/html/2605.08182#S2.E3)\) atτi\\tau\_\{i\}is

qτi0∈arg⁡minq∈ℝ⁡𝔼Y∼μ^s,a​\[ρτi​\(Y−q\)\],q\_\{\\tau\_\{i\}\}^\{0\}\\in\\arg\\min\_\{q\\in\\mathbb\{R\}\}\\mathbb\{E\}\_\{Y\\sim\\widehat\{\\mu\}\_\{s,a\}\}\\left\[\\rho\_\{\\tau\_\{i\}\}\(Y\-q\)\\right\],\(4\)whereρτ​\(u\)=u​\(τ−𝟙\{u<0\}\)\\rho\_\{\\tau\}\(u\)=u\(\\tau\-\\mathbbm\{1\}\_\{\\\{u<0\\\}\}\)and any minimizerqτi0q\_\{\\tau\_\{i\}\}^\{0\}satisfies the empirical coverage condition

ℙY∼μ^s,a​\(Y<qτi0\)≤τi≤ℙY∼μ^s,a​\(Y≤qτi0\)\.\\mathbb\{P\}\_\{Y\\sim\\hat\{\\mu\}\_\{s,a\}\}\(Y<q\_\{\\tau\_\{i\}\}^\{0\}\)\\leq\\tau\_\{i\}\\leq\\mathbb\{P\}\_\{Y\\sim\\hat\{\\mu\}\_\{s,a\}\}\(Y\\leq q\_\{\\tau\_\{i\}\}^\{0\}\)\.

Summarized by Lemma[1](https://arxiv.org/html/2605.08182#Thmlemma1), the IQN loss minimization can be interpreted as simultaneously solving a collection of local quantile regression problems over sampled fractions\{τi\}i=1N\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}\.

### 2\.4Robustness of Quantile Geometry Regularization

Building on the degeneration analysis in Section[2\.2](https://arxiv.org/html/2605.08182#S2.SS2), we reinterpret the robustness issue of the return distribution as a sequence of local robust numerical quantile estimation problems in Lemma[1](https://arxiv.org/html/2605.08182#Thmlemma1)\. Specifically, once the Bellman target distribution is fixed, each sampled quantile levelτ∈\(0,1\)\\tau\\in\(0,1\)asks for a scalar estimateqτq\_\{\\tau\}that locally represents theτ\\tau\-quantile of the target return law\.

Robust quantile estimationTzavidiset al\.\([2010](https://arxiv.org/html/2605.08182#bib.bib20)\); John \([2015](https://arxiv.org/html/2605.08182#bib.bib21)\); Galarza Moraleset al\.\([2017](https://arxiv.org/html/2605.08182#bib.bib22)\)has been extensively studied in statistics literature, and is closely related to the robustness of IQN loss at a specific snapshot\. More recently, Wasserstein distributionally robust quantile regression \(WDRQR\)Zhanget al\.\([2026](https://arxiv.org/html/2605.08182#bib.bib9)\)shows that, under a type\-ppWasserstein ambiguity set, distributional robustness induces an exact regularized reformulation for the check\-loss quantile regression objective\. More importantly, this robustness effect is not merely a global penalty\. The robust solution differs from the corresponding regularized nominal solution through a radius\-dependent location adjustment, while the check loss is essentially the unique convex loss class that admits such an additive Wasserstein regularization phenomenon\. Next, we present Theorem[1](https://arxiv.org/html/2605.08182#Thmtheorem1)in a concise form based on WDRQR, making its connection to the statistical intuition more explicit\.

###### Theorem 1\(Wasserstein\-robust local quantile correction\)\.

Forϵ≥0\\epsilon\\geq 0,p∈\(1,∞\]p\\in\(1,\\infty\], consider the Wasserstein distributionally robust quantile estimation slot atτi\\tau\_\{i\}from Eq\. \([4](https://arxiv.org/html/2605.08182#S2.E4)\)

qτiϵ∈arg⁡minq∈ℝ​supν∈Bp​\(μ^s,a,ϵ\)𝔼Y∼ν​\[ρτi​\(Y−q\)\],q\_\{\\tau\_\{i\}\}^\{\\epsilon\}\\in\\arg\\min\_\{q\\in\\mathbb\{R\}\}\\sup\_\{\\nu\\in B\_\{p\}\(\\hat\{\\mu\}\_\{s,a\},\\epsilon\)\}\\mathbb\{E\}\_\{Y\\sim\\nu\}\\\!\\left\[\\rho\_\{\\tau\_\{i\}\}\(Y\-q\)\\right\],\(5\)whereBp​\(μ^s,a,ϵ\)B\_\{p\}\(\\hat\{\\mu\}\_\{s,a\},\\epsilon\)denotes the type\-ppWasserstein ambiguity setBp​\(μ^s,a,ϵ\):=\{ν∈𝒫​\(ℝ\):Wp​\(ν,μ^s,a\)≤ϵ\}\.B\_\{p\}\(\\hat\{\\mu\}\_\{s,a\},\\epsilon\):=\\\{\\nu\\in\\mathcal\{P\}\(\\mathbb\{R\}\):W\_\{p\}\(\\nu,\\hat\{\\mu\}\_\{s,a\}\)\\leq\\epsilon\\\}\.For each sampled fractionτi\\tau\_\{i\}, any robust minimizer admits the explicit location correction

qτiϵ=qτi0\+Δp​\(τi,ϵ\),q\_\{\\tau\_\{i\}\}^\{\\epsilon\}=q\_\{\\tau\_\{i\}\}^\{0\}\+\\Delta\_\{p\}\(\\tau\_\{i\},\\epsilon\),\(6\)where the robust termΔp\\Delta\_\{p\}can be expressed as

Δp​\(τ,ϵ\)=ϵq​\(τq−\(1−τ\)q\)​cτ,p1−q,1p\+1q=1,\\Delta\_\{p\}\(\\tau,\\epsilon\)=\\frac\{\\epsilon\}\{q\}\\big\(\\tau^\{q\}\-\(1\-\\tau\)^\{q\}\\big\)\\,c\_\{\\tau,p\}^\{\\,1\-q\},\\qquad\\frac\{1\}\{p\}\+\\frac\{1\}\{q\}=1,\(7\)and

cτ,p=\{\(τq​\(1−τ\)\+τ​\(1−τ\)q\)1/q,p∈\(1,∞\),2​τ​\(1−τ\),p=∞\.c\_\{\\tau,p\}=\\begin\{cases\}\\big\(\\tau^\{q\}\(1\-\\tau\)\+\\tau\(1\-\\tau\)^\{q\}\\big\)^\{1/q\},&p\\in\(1,\\infty\),\\\\\[4\.0pt\] 2\\tau\(1\-\\tau\),&p=\\infty\.\\end\{cases\}\(8\)

For each sampled fractionτi\\tau\_\{i\}, the Wasserstein distributionally robust quantile estimation problem in Theorem[1](https://arxiv.org/html/2605.08182#Thmtheorem1)can be written asqτiϵ∈arg⁡minq∈ℝ⁡1N′​∑j=1N′ρτi​\(yj\+Δp​\(τi,ϵ\)−q\)\.q\_\{\\tau\_\{i\}\}^\{\\epsilon\}\\in\\arg\\min\_\{q\\in\\mathbb\{R\}\}\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\rho\_\{\\tau\_\{i\}\}\\\!\\left\(y\_\{j\}\+\\Delta\_\{p\}\(\\tau\_\{i\},\\epsilon\)\-q\\right\)\.

Standard IQN estimates this quantity under the empirical target induced by transition samples and corresponding target\. In contrast, our goal is to estimate a quantile that remains stable under local distributional perturbations of the Bellman target\. Thus, instead of directly robustifying the infinite\-dimensional return distribution, we robustify the scalar quantile estimator that represents each local slice of the learned quantile function\.

## 3Algorithm

In this section, we propose the practical algorithm forRQIQN\. It can be readily adapted to other variants of QRTD based DistRL\.

###### Corollary 1\(The RQIQN Loss\)\.

Following the form of Eq\. \([6](https://arxiv.org/html/2605.08182#S2.E6)\), robust quantile\-regression\-based loss function for RQIQN is given by

ℒRQIQN​\(s,a,r,s′\)=1N′​∑i=1N∑j=1N′ρτi​\(r\+γ​Zτj′​\(s′,a∗\)\+Δp​\(τi,ϵ\)−Zτi​\(s,a\)\)\.\\mathcal\{L\}\_\{\\mathrm\{RQIQN\}\}\(s,a,r,s^\{\\prime\}\)=\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\rho\_\{\\tau\_\{i\}\}\\\!\\left\(r\+\\gamma Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\*\}\)\+\\Delta\_\{p\}\(\\tau\_\{i\},\\epsilon\)\-Z\_\{\\tau\_\{i\}\}\(s,a\)\\right\)\.\(9\)

A key issue arises when correction\-related termcτ,pc\_\{\\tau,p\}in Eq\. \([8](https://arxiv.org/html/2605.08182#S2.E8)\) is applied to continuously sampled quantile fractions in RQIQN\. Forp∈\(1,∞\)p\\in\(1,\\infty\), the closed\-form correction may become unbounded asτ→0\\tau\\rightarrow 0or11\. This is undesirable in deep distributional Q\-learning, where quantile fractions are sampled from a continuous space and bootstrapped targets are further affected by non\-stationary network updates\. Moreover, since Q\-learning itself is prone to overestimation, directly amplifying extreme target quantiles may further destabilize TD learning\.

As discussed in Remark[1](https://arxiv.org/html/2605.08182#Thmremark1), the robust correction provides stronger correction intensity near the distributional tails, where quantiles correspond to extreme return outcomes and are more sensitive to finite\-sample uncertainty\. In our implementation, we focus on two representative cases,p=2p=2andp=∞p=\\infty\. The casep=∞p=\\inftyyields the bounded correctionΔ∞​\(τ;ϵ\)=ϵ​\(2​τ−1\)\\Delta\_\{\\infty\}\(\\tau;\\epsilon\)=\\epsilon\(2\\tau\-1\)\. Forp=2p=2, the raw correction contains a denominator that vanishes near the endpoints\. We therefore use a bounded adaptation by reversing the endpoint behavior of the denominator:

Δ2​\(τ;ϵ\)=ϵ2​1−2​ττ2\+\(1−τ\)2\.\\Delta\_\{2\}\(\\tau;\\epsilon\)=\\frac\{\\epsilon\}\{2\}\\frac\{1\-2\\tau\}\{\\sqrt\{\\tau^\{2\}\+\(1\-\\tau\)^\{2\}\}\}\.This adapted term remains finite for allτ∈\[0,1\]\\tau\\in\[0,1\], while preserving the median\-antisymmetry and monotonicity required by the geometric interpretation in Remark[1](https://arxiv.org/html/2605.08182#Thmremark1)\. To avoid directly increasing bootstrapped target values,Δ2​\(τ;ϵ\)\\Delta\_\{2\}\(\\tau;\\epsilon\)provides a subtractive prediction\-side correction in the TD residual\. Under this sign convention, the effective corrected quantile geometry still satisfies the same mean\-neutral spread\-modulation property\.

Statistically, WDRQRZhanget al\.\([2026](https://arxiv.org/html/2605.08182#bib.bib9)\)suggests a power\-law shrinkage of the Wasserstein radius with the empirical sample size, i\.e\.,ϵN=O​\(N−1/2\)\\epsilon\_\{N\}=O\(N^\{\-1/2\}\), deep distributional Q\-learning induces non\-stationary empirical targets through replay sampling, bootstrapping, and network updates\. We therefore decoupleϵ\\epsilonfrom the nominal batch size and use a decay schedule, preserving strong early\-stage robustness and gradually annealing the correction as training stabilizes\. Specifically, to implement adaptive robustness during training rather than using a fixed radiusϵ\\epsilon, we adopt a time\-dependent reverse\-logistic decay schedule

ϵt=ϵ01\+exp⁡\(k​\(t−t0\)\),\\epsilon\_\{t\}=\\frac\{\\epsilon\_\{0\}\}\{1\+\\exp\(k\(t\-t\_\{0\}\)\)\},wherettis the training step,ϵ0\\epsilon\_\{0\}denotes the initial robustness scale,k\>0k\>0controls the decay sharpness, andt0t\_\{0\}specifies the midpoint of the reverse\-logistic transition\.

Input:

\(s,a,r,s′\)\(s,a,r,s^\{\\prime\}\),

γ∈\[0,1\)\\gamma\\in\[0,1\),

N,N′,KN,N^\{\\prime\},K, timestep

t\>0t\>0,

ϵt≥0\\epsilon\_\{t\}\\geq 0,

p∈\(1,∞\]p\\in\(1,\\infty\],

β\\beta,

ZZ
τ~k∼β​\(⋅\),k=1,…,K\\textstyle\\tilde\{\\tau\}\_\{k\}\\sim\\beta\(\\cdot\),\\quad k=1,\\ldots,K

//Sample fractions for action selection

a∗←arg​maxa′∈𝒜⁡1K​∑k=1KZτ~k​\(s′,a′\)\\textstyle a^\{\*\}\\leftarrow\\operatorname\*\{arg\\,max\}\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}Z\_\{\\tilde\{\\tau\}\_\{k\}\}\(s^\{\\prime\},a^\{\\prime\}\)

//Select greedy next action

τi,τj′∼𝒰​\(\[0,1\]\),i=1,…,N,j=1,…,N′\\textstyle\\tau\_\{i\},\\tau^\{\\prime\}\_\{j\}\\sim\\mathcal\{U\}\(\[0,1\]\),\\quad i=1,\\ldots,N,\\;j=1,\\ldots,N^\{\\prime\}

//Sample quantile fractions

Δi←Δp​\(τi;ϵt\)\\textstyle\\Delta\_\{i\}\\leftarrow\\Delta\_\{p\}\(\\tau\_\{i\};\\epsilon\_\{t\}\)

//Compute robust local correction

δ~i​j←r\+γ​Zτj′​\(s′,a∗\)\+Δi−Zτi​\(s,a\),∀i,j\\textstyle\\widetilde\{\\delta\}\_\{ij\}\\leftarrow r\+\\gamma Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\*\}\)\+\\Delta\_\{i\}\-Z\_\{\\tau\_\{i\}\}\(s,a\),\\quad\\forall i,j

//Compute robust TD error

Output:

1N′​∑i=1N∑j=1N′ρτi​\(δ~i​j\)\\textstyle\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\rho\_\{\\tau\_\{i\}\}\\left\(\\widetilde\{\\delta\}\_\{ij\}\\right\)

Algorithm 1Robust Quantile Implicit Quantile Network Loss#### Check loss versus Huberized loss\.

Recent WDRQR theory shows that, forp\>1p\>1, the check loss is essentially the unique convex loss that admits an exact additive Wasserstein regularization under a location\-adjusted objective\. The quantile Huber loss, although standard in quantile\-based methods like QR\-DQN and IQN, is not an affine transformation of the check loss because it is locally quadratic and globally linear\. Therefore, the exact distributionally robust quantile regression equivalence is not feasible to extend to the Huber variant\. Instead, RQIQN\-Huber is used as a practical smooth surrogate that preserves the same quantile\-dependent perturbation structure\.

While Corollary[1](https://arxiv.org/html/2605.08182#Thmcorollary1)characterizes RQIQN under the original check loss, our implementation adopts a Huberized quantile regression objective for improved optimization stability\. Specifically, givenNNcurrent quantile fractions andN′N^\{\\prime\}target quantile fractions, the RQIQN\-Huber loss is defined as

ℒRQIQN​\-​Huber=1N′​∑i=1N∑j=1N′ρτiκ​\(δ~i​j\),δ~i​j=r\+γ​Zτj′​\(s′,a∗\)\+Δi−Zτi​\(s,a\)\.\\mathcal\{L\}\_\{\\mathrm\{RQIQN\\text\{\-\}Huber\}\}=\\frac\{1\}\{N^\{\\prime\}\}\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{N^\{\\prime\}\}\\rho\_\{\\tau\_\{i\}\}^\{\\kappa\}\\left\(\\widetilde\{\\delta\}\_\{ij\}\\right\),\\quad\\widetilde\{\\delta\}\_\{ij\}=r\+\\gamma Z\_\{\\tau^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\*\}\)\+\\Delta\_\{i\}\-Z\_\{\\tau\_\{i\}\}\(s,a\)\.

## 4Empirical Results

In this section, we report preliminary experimental results\. RQIQN is implemented on top of the standard IQN architecture\. Unless otherwise specified, we use the check loss for theoretical consistency and set more generalp=2p=2for the closed\-form Wasserstein correction\. The initial robustness radiusϵ0=1\\epsilon\_\{0\}=1is natural and stable across the considered tasks\. For Atari experiments, we use a reverse\-logistic decay schedule with midpointt0=3\.75×106t\_\{0\}=3\.75\\times 10^\{6\}training steps and sharpnessk=1\.2×10−6k=1\.2\\times 10^\{\-6\}\. For the navigation experiments, which run for3×1063\\times 10^\{6\}total training steps, we use the same sharpnessk=1\.2×10−6k=1\.2\\times 10^\{\-6\}and set the midpoint tot0=5\.9×105t\_\{0\}=5\.9\\times 10^\{5\}\.

### 4\.1Risk\-Sensitive Control: Autonomous Surface Vehicles Navigation

IQN has emerged as a powerful paradigm for robotics tasksZhanget al\.\([2025](https://arxiv.org/html/2605.08182#bib.bib11)\)such as uncertainty\-aware autonomous navigation\. We evaluate RQIQN in a representative safety\-critical marine navigation environmentLinet al\.\([2023](https://arxiv.org/html/2605.08182#bib.bib13)\), where bounded workspaces, static obstacles, and vortex\-induced flow disturbances are simulated according to physically motivated environmental settingsAcheson \([1990](https://arxiv.org/html/2605.08182#bib.bib30)\)\. Following the original learning\-based evaluation protocol, we deploy the learned policy for unmanned surface vehicle control and compare RQIQN against IQN and DQNMnihet al\.\([2015](https://arxiv.org/html/2605.08182#bib.bib26)\)\.

![Refer to caption](https://arxiv.org/html/2605.08182v1/x2.png)Figure 2:Qualitative trajectory results of RL agents\. Theyellow circledenotes the start position, and theyellow stardenotes the goal\.Magenta circlesindicate static obstacles, while the background vector field represents vortex\-induced flow disturbances\. Trajectories for IQN and RQIQN are shown under the adaptive setting, where both achieve stronger performance\.Specifically, the RL agent receives observations encoding the vessel state and LiDAR\-based frontal obstacle distances without environmental prior knowledge, and outputs discrete motion\-control actions for goal\-directed navigation\. The reward encourages goal\-reaching progress and safe navigation\. The objective is to learn a policy that reaches the target efficiently while avoiding collisions and maintaining robust behavior under invisible flow\-induced disturbances\.

Table 1:Evaluation under the*hard\-mode*navigation setting with 10 static obstacles and 8 vortex cores\. Results are reported as mean±\\pmstandard deviation over 5 independently trained seeds, with each seed evaluated on 500 episodes using different environment layouts\.MethodSuccess\(%\)↑\\uparrowCollision\(%\)↓\\downarrowTimeout\(%\)↓\\downarrowReturn↑\\uparrowTimesucc\(s\)↓\\downarrowEnergysucc↓\\downarrowDQN67\.24±\\pm4\.7932\.72±\\pm4\.770\.04±\\pm0\.0912\.97±\\pm3\.6533\.48±\\pm0\.3398\.95±\\pm5\.76IQN64\.80±\\pm22\.6621\.92±\\pm9\.5413\.28±\\pm29\.47\-13\.17±\\pm51\.7839\.08±\\pm1\.22122\.56±\\pm17\.29\\rowcolorgray\!12 \+Adaptive71\.04±\\pm23\.3515\.44±\\pm6\.1313\.52±\\pm29\.11\-12\.19±\\pm51\.2042\.24±\\pm1\.55133\.31±\\pm17\.04RQIQN83\.24±\\pm1\.1916\.48±\\pm1\.140\.28±\\pm0\.2719\.20±\\pm1\.7337\.55±\\pm1\.25111\.24±\\pm6\.34\\rowcolorgray\!12 \+Adaptive85\.64±\\pm2\.2714\.20±\\pm2\.020\.16±\\pm0\.2618\.57±\\pm3\.9039\.19±\\pm1\.87115\.83±\\pm6\.41

In this study, the default IQN and RQIQN agents use the natural full quantile support, i\.e\.,τ∼U​\(\[0,1\]\)\\tau\\sim U\(\[0,1\]\)\. Following the existing navigation task, we additionally evaluate the*adaptive variants*in Table[1](https://arxiv.org/html/2605.08182#S4.T1), where the CVaR thresholdη\\etais adjusted according to the perceived distance to the nearest obstacle\. Smaller obstacle distances induce more conservative lower\-tail sampling and RQIQN under both sampling reaches improved navigation results and safety\. As shown in Figure[2](https://arxiv.org/html/2605.08182#S4.F2), RQIQN remains robust in dense\-obstacle scenarios where vortex\-induced flows perturb actions and state transitions\.

### 4\.2Atari Games

We compare our algorithm against several representative quantile\-based DistRL baselines that have demonstrated strong performance on standard RL benchmarks\. DQN serves as the expectation\-based baseline that learns only scalar action values\. C51Bellemareet al\.\([2017](https://arxiv.org/html/2605.08182#bib.bib2)\)represents the return distribution with a categorical distribution over fixed supports\. QR\-DQNDabneyet al\.\([2018b](https://arxiv.org/html/2605.08182#bib.bib4)\)learns a fixed set of quantile locations via quantile regression, while IQN extends this idea by sampling quantile fractions to approximate an implicit quantile function\. RainbowHesselet al\.\([2018](https://arxiv.org/html/2605.08182#bib.bib27)\)is included as a strong aggregate baseline that combines C51 with several orthogonal improvements to DQN\. RQIQN is evaluated as our robust extension of IQN, using the implicit quantile representation with proposed Wasserstein\-robust local correction\.

![Refer to caption](https://arxiv.org/html/2605.08182v1/x3.png)Figure 3:Performance comparison on 9 Atari games\. RQIQN results are averaged over 3 random seeds and compared against reference values from Castro et al\.Castroet al\.\([2018](https://arxiv.org/html/2605.08182#bib.bib28)\)\.We evaluate on several Atari games, including the Atari\-5 subbenchmark\(Aitchisonet al\.,[2023](https://arxiv.org/html/2605.08182#bib.bib25)\), a compact subset of the Arcade Learning Environment \(ALE\) designed to approximate trends on the full Atari\-57 suite with representativeness and substantially lower computational cost\. For reproducible evaluation, all baselines and our method are implemented in the Dopamine frameworkCastroet al\.\([2018](https://arxiv.org/html/2605.08182#bib.bib28)\)and trained for 200M frames under the*sticky\-actions protocol*Machadoet al\.\([2018](https://arxiv.org/html/2605.08182#bib.bib29)\)\. Under this protocol, the ALE repeats the previously executed action with probability0\.250\.25instead of always applying the agent’s newly selected action, injecting controlled stochasticity and reducing reliance on deterministic Atari dynamics\.

As shown in Figure[3](https://arxiv.org/html/2605.08182#S4.F3), RQIQN improves over IQN on most Atari games, with the exception ofNameThisGame\. Notably, it achieves stronger performance onAlien,BattleZone,Frostbite, andQbertcompared with the evaluated baselines\. These results suggest that the proposed robust quantile correction can further enhance IQN, indicating that quantile\-level robustness remains beneficial even when the quantile Huber loss already provides a stronger practical surrogate than the pure check loss\.

![Refer to caption](https://arxiv.org/html/2605.08182v1/x4.png)Figure 4:Performance of RQIQN variants\.#### Variants of RQIQN\.

To examine the sensitivity of RQIQN to key implementation choices, we further conduct ablation studies onQbert\. Specifically, we compare the check\-loss and Huberized variants, and evaluate two representative type\-ppWasserstein ambiguity sets, corresponding top=2p=2andp=∞p=\\infty\. As shown in Figure[4](https://arxiv.org/html/2605.08182#S4.F4), RQIQN achieves the best performance under the default configuration, using the check loss with a type\-22Wasserstein ambiguity set\. The Huberized variant withp=2p=2performs comparably, with only a small performance gap, and both variants consistently outperform IQN\. In contrast, when the ambiguity set is defined withp=∞p=\\infty, both the check\-loss and Huberized variants exhibit performance degradation\. These results suggest that RQIQN is more sensitive to the choice of Wasserstein geometry than to the choice between check and Huberized losses\.

#### The Suitability of Distribution Representation\.

In Figure[3](https://arxiv.org/html/2605.08182#S4.F3), we observe that onNameThisGame, methods with more advanced schemes exhibit unexpected performance fluctuations and converge to suboptimal final returns\. In contrast, C51 maintains a relatively stable and high training return throughout learning\. This indicates that RQIQN, despite its overall gains, may still be vulnerable to performance degradation in specific environments, as observed for other advanced DistRL methods\.

In the standard Atari setting, rewards are clipped to\[−1,1\]\[\-1,1\], and C51 represents the return distribution using 51 fixed atoms on a bounded support, typically\[Vmin,Vmax\]=\[−10,10\]\[V\_\{\\min\},V\_\{\\max\}\]=\[\-10,10\]\. Bellman targets are projected back onto this fixed support, which can suppress the influence of extreme high\-return estimates and induce a conservative bias in action selection\. This may prevent the agent from over\-committing to aggressive strategies driven by unstable tail values\.

In contrast, IQN and RQIQN use a more flexible quantile\-function representation without a fixed finite support\. While this flexibility improves distributional expressiveness, it also makes the estimated mean return more sensitive to fluctuations in the distributional tails when the target distribution is not well fitted\. The observation is informative: in principle, a sufficiently accurate quantile model should exploit this flexibility without suffering from tail instability, but in practice there is a trade\-off between representational capacity and estimation error\. This trade\-off is consistent with the motivation of IQN and RQIQN\. Rather than imposing a fixed support, one can exploit the flexible quantile representation through distortion\-based risk control, such as CVaR sampling, to emphasize different regions of the return distribution\. This also aligns with our navigation results, where adaptive CVaR distortion improves risk\-aware control under hazardous conditions\.

## 5Conclusion

In this work, we introduce RQIQN, which improves IQN by making local quantile fitting robust to finite\-sample perturbations of the bootstrapped Bellman target distribution, without introducing computationally intensive schemes\. We further develop a Wasserstein\-robust quantile correction mechanism for mitigating degeneration problem\. This correction modulates the distribution geometry during Bellman updates, leading to improved performance on Atari games and risky navigation\.

## References

- Elementary fluid dynamics\.Oxford University Press\.Cited by:[§4\.1](https://arxiv.org/html/2605.08182#S4.SS1.p1.1)\.
- M\. Aitchison, P\. Sweetser, and M\. Hutter \(2023\)Atari\-5: distilling the arcade learning environment down to five games\.InInternational Conference on Machine Learning,pp\. 421–438\.Cited by:[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p2.1)\.
- M\. G\. Bellemare, W\. Dabney, and R\. Munos \(2017\)A distributional perspective on reinforcement learning\.InInternational conference on machine learning,pp\. 449–458\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p1.1)\.
- P\. S\. Castro, S\. Moitra, C\. Gelada, S\. Kumar, and M\. G\. Bellemare \(2018\)Dopamine: a research framework for deep reinforcement learning\.arXiv preprint arXiv:1812\.06110\.Cited by:[Figure 3](https://arxiv.org/html/2605.08182#S4.F3),[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p2.1)\.
- W\. Dabney, G\. Ostrovski, D\. Silver, and R\. Munos \(2018a\)Implicit quantile networks for distributional reinforcement learning\.InInternational conference on machine learning,pp\. 1096–1105\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§1](https://arxiv.org/html/2605.08182#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.08182#S2.SS1.p2.15)\.
- W\. Dabney, M\. Rowland, M\. Bellemare, and R\. Munos \(2018b\)Distributional reinforcement learning with quantile regression\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§1](https://arxiv.org/html/2605.08182#S1.p1.1),[§2\.3](https://arxiv.org/html/2605.08182#S2.SS3.p1.10),[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p1.1)\.
- A\. P\. Dempster, N\. M\. Laird, and D\. B\. Rubin \(1977\)Maximum likelihood from incomplete data via the em algorithm\.Journal of the royal statistical society: series B \(methodological\)39\(1\),pp\. 1–22\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3)\.
- C\. Galarza Morales, V\. Lachos Davila, C\. Barbosa Cabral, and L\. Castro Cepero \(2017\)Robust quantile regression using a generalized class of skewed distributions\.Stat6\(1\),pp\. 113–130\.Cited by:[§2\.4](https://arxiv.org/html/2605.08182#S2.SS4.p2.1)\.
- M\. Hessel, J\. Modayil, H\. Van Hasselt, T\. Schaul, G\. Ostrovski, W\. Dabney, D\. Horgan, B\. Piot, M\. Azar, and D\. Silver \(2018\)Rainbow: combining improvements in deep reinforcement learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p1.1)\.
- P\. J\. Huber \(1992\)Robust estimation of a location parameter\.InBreakthroughs in statistics: Methodology and distribution,pp\. 492–518\.Cited by:[§2\.1](https://arxiv.org/html/2605.08182#S2.SS1.p3.3)\.
- O\. O\. John \(2015\)Robustness of quantile regression to outliers\.American Journal of Applied Mathematics and Statistics3\(2\),pp\. 86–88\.Cited by:[§2\.4](https://arxiv.org/html/2605.08182#S2.SS4.p2.1)\.
- S\. Jullien, R\. Deffayet, J\. Renders, P\. Groth, and M\. de Rijke \(2023\)Distributional reinforcement learning with dual expectile\-quantile regression\.arXiv preprint arXiv:2305\.16877\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3),[§1](https://arxiv.org/html/2605.08182#S1.p3.4)\.
- X\. Lin, J\. McConnell, and B\. Englot \(2023\)Robust unmanned surface vehicle navigation with distributional reinforcement learning\.In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 6185–6191\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.08182#S4.SS1.p1.1)\.
- M\. C\. Machado, M\. G\. Bellemare, E\. Talvitie, J\. Veness, M\. Hausknecht, and M\. Bowling \(2018\)Revisiting the arcade learning environment: evaluation protocols and open problems for general agents\.Journal of Artificial Intelligence Research61,pp\. 523–562\.Cited by:[§4\.2](https://arxiv.org/html/2605.08182#S4.SS2.p2.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. Riedmiller \(2013\)Playing atari with deep reinforcement learning\.arXiv preprint arXiv:1312\.5602\.Cited by:[§1](https://arxiv.org/html/2605.08182#S1.p2.3),[§2\.3](https://arxiv.org/html/2605.08182#S2.SS3.p1.10)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.\(2015\)Human\-level control through deep reinforcement learning\.nature518\(7540\),pp\. 529–533\.Cited by:[§4\.1](https://arxiv.org/html/2605.08182#S4.SS1.p1.1)\.
- W\. K\. Newey and J\. L\. Powell \(1987\)Asymmetric least squares estimation and testing\.Econometrica: Journal of the Econometric Society,pp\. 819–847\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3)\.
- T\. T\. Nguyen, S\. Gupta, and S\. Venkatesh \(2020\)Distributional reinforcement learning with maximum mean discrepancy\.Association for the Advancement of Artificial Intelligence \(AAAI\)\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3),[§1](https://arxiv.org/html/2605.08182#S1.p3.4)\.
- M\. Rowland, R\. Dadashi, S\. Kumar, R\. Munos, M\. G\. Bellemare, and W\. Dabney \(2019\)Statistics and samples in distributional reinforcement learning\.InInternational Conference on Machine Learning,pp\. 5528–5536\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3),[§1](https://arxiv.org/html/2605.08182#S1.p1.1),[§1](https://arxiv.org/html/2605.08182#S1.p3.4),[§2\.2](https://arxiv.org/html/2605.08182#S2.SS2.p1.1)\.
- J\. Shi, C\. Bai, H\. He, L\. Han, D\. Wang, B\. Zhao, M\. Zhao, X\. Li, and X\. Li \(2024\)Robust quadrupedal locomotion via risk\-averse policy learning\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 11459–11466\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1)\.
- N\. Tzavidis, S\. Marchetti, and R\. Chambers \(2010\)Robust estimation of small\-area means and quantiles\.Australian & New Zealand Journal of Statistics52\(2\),pp\. 167–186\.Cited by:[§2\.4](https://arxiv.org/html/2605.08182#S2.SS4.p2.1)\.
- D\. Yang, L\. Zhao, Z\. Lin, T\. Qin, J\. Bian, and T\. Liu \(2019\)Fully parameterized quantile function for distributional reinforcement learning\.Advances in neural information processing systems32\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1)\.
- C\. Zhang, T\. Mao, and R\. Wang \(2026\)Wasserstein distributionally robust quantile regression\.arXiv preprint arXiv:2603\.14991\.Cited by:[§2\.4](https://arxiv.org/html/2605.08182#S2.SS4.p2.1),[§3](https://arxiv.org/html/2605.08182#S3.p4.3)\.
- W\. Zhang, J\. Wang, and Y\. Yu \(2024\)Distributional reinforcement learning with sample\-set bellman update\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 2852–2858\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p2.3)\.
- Z\. Zhang, M\. Yang, S\. Xie, and H\. Xiong \(2025\)Perturbation\-mitigated usv navigation with distributionally robust reinforcement learning\.arXiv preprint arXiv:2512\.00030\.Cited by:[§1\.1](https://arxiv.org/html/2605.08182#S1.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.08182#S4.SS1.p1.1)\.