Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification

# Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification
Source: [https://arxiv.org/html/2609.03308](https://arxiv.org/html/2609.03308)
Ziqi Zhang††thanks:Z\. Zhang is with the College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 211100, China \(E\-mail: 1124562662@qq\.com \)\.

###### Abstract

Reliable operation of modern distribution networks requires timely identification of operational risks and anomalous events under pervasive uncertainty\. In practice, operators must identify risks that are inherent in stochastic yet in\-distribution conditions, and anomalies that correspond to out\-of\-distribution behaviors such as unusual load patterns, extreme weather or cyber\-physical attacks\. This paper addresses this joint risk and anomaly identification problem for optimal distribution network operation and proposes a deep reinforcement learning framework that is explicitly uncertainty aware\. We integrate distributional and Bayesian deep reinforcement learning to realize a second\-order uncertainty quantification scheme that decomposes total uncertainty into aleatoric and epistemic components, which are respectively used to characterize inherent risk and out\-of\-distribution anomalies\. The resulting epistemic estimates drive both exploration during training and out\-of\-distribution detection with fallback control during deployment, whereas aleatoric estimates are used to characterize intrinsic operational risk\. Simulation results demonstrate the performance of our DRL agent and the effectiveness of the uncertainty quantification\.

###### Index Terms:

Anomaly detection, deep reinforcement learning, distribution network operation, energy management systems, uncertainty quantification\.

## IIntroduction

Modern distribution network \(DN\) operation is increasingly shaped by renewable energy sources \(RES\) such as solar photovoltaics \(PV\), distributed energy storage systems \(ESS\) and distributed diesel generators \(DGs\)\. These factors make operation stochastic, high\-dimensional, and time\-coupled\. Model\-free deep reinforcement learning \(DRL\)\[[1](https://arxiv.org/html/2609.03308#bib.bib1)\]gives real\-time policy for nonlinear DNs\. However, despite DRL’s advantages, vanilla DRL techniques face trustworthiness issues when applied to real\-world DNs, namely lacking the ability of uncertainty quantification \(UQ\)\. In the realm of UQ, uncertainties are generally classified into two categories: \(1\) epistemic uncertainty \(EU\) arising from a lack of knowledge or information about the current situation; \(2\) aleatoric uncertainty \(AU\) inherent to the environment and cannot be reduced by gathering more information\. The combination of the two is termed as total uncertainty\.

For a DRL agent, high EU may be caused by out\-of\-distribution \(OOD\) scenarios, where the operational environment deviates significantly from training data, contrasting to in\-distribution \(InD\) scenarios where the operational environment remains close to training data\. Such distributional shifts may arise from various DN anomalies, encompassing: \(1\) atypical load profiles, such as variable charging patterns of electric vehicles, air conditioning demand surge due to heatwaves and cold snaps, or spikes because of unplanned public events, all of which introduce complex and non\-stationary patterns that are challenging to detect; \(2\) unexpected RES behaviors, such as storm\-induced damage, inverter malfunctions, or irregular cloud shading over PV systems, where spatio\-temporal dynamics of cloud movements create a combinatorial explosion of non\-uniform shading patterns on distributed PV arrays\[[2](https://arxiv.org/html/2609.03308#bib.bib2)\]; \(3\) the deterioration of infrastructures, such as slow degradation of transformer insulation, corrosion of conductors and connectors, and increased impedance in aging cables, which collectively result in small but cumulative shifts in the whole system’s physical parameters, often indistinguishable from random noises\.

In these OOD scenarios with conditions not seen during training, DRL agents based on neural networks \(NNs\) might fail to generalize robustly\[[3](https://arxiv.org/html/2609.03308#bib.bib3)\]\. This can result in confident yet suboptimal or unsafe actions, such as improper voltage regulation, excessive reactive power compensation or inefficient diesel generator dispatch\. These actions can lead to inefficient energy management, branch overloading, violation of nodal voltage limits, severe load imbalances, potentially causing increased operational costs, widespread power outages, or even physical damage to critical grid components\.

The situation can be exacerbated by limited availability of the training data, especially in DNs due to privacy concerns \(e\.g\., smart meters and electric vehicles’ privacy\) or infrastructure constraints \(e\.g\., sparse monitoring devices\)\. What is worse, the combination of the time\-series data and the large number of DN buses contributes to the high dimensionality of the scenario, bringing the curse\-of\-dimensionality phenomenon that most points tend to have similar distances among them\[[4](https://arxiv.org/html/2609.03308#bib.bib4)\]\. Such phenomenon, along with the complex interpolation and extrapolation of NNs, may render traditional OOD detection approaches such as those based on metrics or density estimation\[[5](https://arxiv.org/html/2609.03308#bib.bib5)\]less effective\. A related challenge to the OOD problem is the presence of adversarial examples\[[6](https://arxiv.org/html/2609.03308#bib.bib6)\]\. These examples are much closer to the training data distribution, with their differences from certain training samples often resembling small perturbations\. Despite these subtle differences, adversarial examples can still lead to incorrect classifications by neural networks\[[6](https://arxiv.org/html/2609.03308#bib.bib6),[7](https://arxiv.org/html/2609.03308#bib.bib7)\]\. In the context of power systems, such perturbations can be deliberately injected by malicious attackers into grid cyber\-physical system through false data injection attacks \(FDIA\)\[[8](https://arxiv.org/html/2609.03308#bib.bib8),[9](https://arxiv.org/html/2609.03308#bib.bib9)\], which can evade conventional anomaly detection in the SCADA system\. The difficulties mentioned necessitate the quest for advanced EU estimation methods\.

In the realm of DN, AU represents the irreducible stochasticity of the grid, including fluctuations in RES\[[10](https://arxiv.org/html/2609.03308#bib.bib10)\], electricity prices or load behaviors, that cannot be reduced through further data collection\. AU\-aware methods model a whole distribution of returns, not just the scalar expectation\[[11](https://arxiv.org/html/2609.03308#bib.bib11)\]\. Effectively capturing such inherent stochasticity is helpful because it enables the system to avoid high\-risk actions that might otherwise appear optimal based on expected returns but could lead to detrimental worst\-case outcomes\.

By juxtaposing the characteristics of AU and EU, it becomes evident that the two types of uncertainty require distinct handling strategies\. However, most previous methods concentrate on reducing the probability of constraint violations\[[12](https://arxiv.org/html/2609.03308#bib.bib12),[13](https://arxiv.org/html/2609.03308#bib.bib13)\], or mainly estimate the combined total uncertainty without differentiating between its sources\. Existing studies address only part of the uncertainty\-aware control problem\. Constrained and safe DRL methods improve feasibility in DN operation, but mainly under in\-distribution stochasticity and without identifying when the learned controller becomes epistemically unreliable\[[14](https://arxiv.org/html/2609.03308#bib.bib14),[15](https://arxiv.org/html/2609.03308#bib.bib15)\]\. Risk\-sensitive and distributional RL methods capture return variability and tail risk, yet they do not explicitly distinguish irreducible environmental randomness from model uncertainty\[[16](https://arxiv.org/html/2609.03308#bib.bib16),[17](https://arxiv.org/html/2609.03308#bib.bib17)\]\. Bayesian DRL and probabilistic OPF approaches introduce predictive uncertainty into decision making, but generally treat uncertainty as an aggregated quantity rather than decomposing it into AU and EU with different operational roles\[[18](https://arxiv.org/html/2609.03308#bib.bib18),[19](https://arxiv.org/html/2609.03308#bib.bib19)\]\. Bayesian UQ has also been used for power\-system forecasting and risk assessment, but mostly in offline prediction or assessment settings rather than in closed\-loop DRL control with OOD\-triggered fallback\[[20](https://arxiv.org/html/2609.03308#bib.bib20),[21](https://arxiv.org/html/2609.03308#bib.bib21)\]\.

The lack of UQ in naive DRL poses a barrier to its deployment in real\-world safety\-critical DN operations\. To close the gaps mentioned, we aim to develop an uncertainty\-aware framework to handle OOD cases and manage risks, subsuming previous distributional DRL and Bayesian DRL methods as submodules\. The main contributions are summarized as follows:

- \(1\)We formulate risk and anomaly identification in distribution\-network operation as a*return\-law uncertainty quantification*problem\. Previous methods consider only a single type of uncertainty, whereas our approach allows the total uncertainty to be further split into AU and EU\. Specifically, we adopt a second\-order UQ method to distinctly characterize risks and anomalies\.
- \(2\)We develop a critic\-side return\-space instantiation of distance\-based second\-order UQ, beyond the original finite\-label categorical instantiation\. By using one\-dimensional quantile representations and the Wasserstein barycenter, we convert the original distance\-based definitions into tractable forms for critic\-side AU and EU\. We establish the equivalence of these forms to the original definitions and prove the Monte Carlo consistency of their empirical estimators\.
- \(3\)We propose an epistemic\-reliability gated deployment scheme that turns critic\-side EU into an online fallback indicator\. When the actor’s action becomes epistemically unreliable, control is switched from fast DRL inference to a conservative MISOCP\-based OPF controller\. Thus, EU is not used as a passive anomaly score, but as an operational switching signal that couples data\-driven control in familiar regimes with optimization\-based recovery under distribution shift\.

The rest of this paper is organized as follows\. Section II rigorously formulates the DN operation problem\. Section III proposes the main framework\. Section IV conducts numerical simulations to demonstrate effectiveness of the proposed method\. Section V presents the conclusions, and the Appendix provides detailed proofs\.

## IIOptimal DN Operation Formulation

We first cast the DN operation problem into a constrained Markov decision process \(CMDP\), then specify the physical model and operational constraints of the DN\.

### II\-APreliminaries on CMDP

We consider a discounted constrained Markov decision process \(CMDP\)⟨𝒮,𝒜,P,R,C,γ⟩\\langle\\mathcal\{S\},\\mathcal\{A\},P,R,C,\\gamma\\rangle, where𝒮\\mathcal\{S\}is the state space,𝒜\\mathcal\{A\}is the action space,P:𝒮×𝒜×𝒮→\[0,1\]P:\\mathcal\{S\}\\times\\mathcal\{A\}\\times\\mathcal\{S\}\\to\[0,1\]is the transition kernel,R:𝒮×𝒜→ℝR:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}is the reward function,C:𝒮×𝒜→ℝmC:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}^\{m\}is a vector of constraint costs, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. A stationary policyπ:𝒮→Δ⁡\(𝒜\)\\pi:\\mathcal\{S\}\\to\\Delta\(\\mathcal\{A\}\)maps each state to a distributionπ\(⋅∣s\)\\pi\(\\cdot\\mid s\)over actions \(in practice we often use deterministic policies, written asat=π⁡\(st\)a\_\{t\}=\\pi\(s\_\{t\}\)\)\. For an initial states0s\_\{0\}, the discounted return and constraint returns of policyπ\\piare

JRπ​\(s0\)\\displaystyle J\_\{R\}^\{\\pi\}\(s\_\{0\}\)=𝔼π​\[∑t=0∞γt​R​\(st,at\)\],\\displaystyle=\\mathbb\{E\}^\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\(s\_\{t\},a\_\{t\}\)\\right\],\(1\)JCiπ​\(s0\)\\displaystyle J\_\{C\_\{i\}\}^\{\\pi\}\(s\_\{0\}\)=𝔼π\[∑t=0∞γtCi\(st,at\)\],i=1,…,m\.\\displaystyle=\\mathbb\{E\}^\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}C\_\{i\}\(s\_\{t\},a\_\{t\}\)\\right\],\\quad i=1,\\dots,m\.\(2\)The CMDP objective is

maxπ\\displaystyle\\max\_\{\\pi\}\\quadJRπ​\(s0\),\\displaystyle J\_\{R\}^\{\\pi\}\(s\_\{0\}\),\(3\)s\.t\.JCiπ\(s0\)≤di,i=1,…,m\.\\displaystyle J\_\{C\_\{i\}\}^\{\\pi\}\(s\_\{0\}\)\\leq d\_\{i\},\\quad i=1,\\dots,m\.\(4\)We employ Lagrangian relaxation to address the constrained optimization\. Instead of maintaining separate multipliers for each constraint, we aggregate all operational limits into a unified scalar costC⁡\(s,a\)C\(s,a\)associated with a single multiplierλ≥0\\lambda\\geq 0, and formulate the Lagrangian objective as:L⁡\(π,λ\)=JRπ​\(s0\)−λ⁡\(JCπ​\(s0\)−d\),L\(\\pi,\\lambda\)=J\_\{R\}^\{\\pi\}\(s\_\{0\}\)\-\\lambda\\bigl\(J\_\{C\}^\{\\pi\}\(s\_\{0\}\)\-d\\bigr\),whereJCπ​\(s0\)J\_\{C\}^\{\\pi\}\(s\_\{0\}\)denotes the expected discounted cumulative aggregated cost andddis the tolerance limit\. This yields the saddle point problem:maxπ⁡minλ≥0⁡L⁡\(π,λ\),\\max\_\{\\pi\}\\min\_\{\\lambda\\geq 0\}L\(\\pi,\\lambda\),which we solve by primal–dual updates\. At iterationkk, given policyπk\\pi\_\{k\}, we update the Lagrange multiplier \(LM\) by projected gradient ascent on the dual:λk\+1=Πℝ\+​\(λk\+ηλ​\(JCπk​\(s0\)−d\)\),\\lambda\_\{k\+1\}=\\Pi\_\{\\mathbb\{R\}\_\{\+\}\}\(\\lambda\_\{k\}\+\\eta\_\{\\lambda\}\\bigl\(J\_\{C\}^\{\\pi\_\{k\}\}\(s\_\{0\}\)\-d\\bigr\)\),whereηλ\>0\\eta\_\{\\lambda\}\>0is a stepsize andΠℝ\+​\(x\)=max⁡\{0,x\}\\Pi\_\{\\mathbb\{R\}\_\{\+\}\}\(x\)=\\max\\\{0,x\\\}enforcesλ≥0\\lambda\\geq 0\. For a fixedλ\\lambda, this is equivalent to solving an unconstrained MDP with the Lagrangian reward:

Rλ​\(s,a\)=R⁡\(s,a\)−λ​C​\(s,a\)\.R\_\{\\lambda\}\(s,a\)=R\(s,a\)\-\\lambda C\(s,a\)\.\(5\)The corresponding state–action value function is

Qλπ\(s,a\)=𝔼π\[∑t=0∞γtRλ\(st,at\)\|s0=s,a0=a\],Q\_\{\\lambda\}^\{\\pi\}\(s,a\)=\\mathbb\{E\}^\{\\pi\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{\\lambda\}\(s\_\{t\},a\_\{t\}\)\\,\\Big\|\\,s\_\{0\}=s,a\_\{0\}=a\\right\],\(6\)satisfying the Bellman equation:

Qλπ\(s,a\)=Rλ\(s,a\)\+γ𝔼s′∼P\(⋅∣s,a\)\[Qλπ\(s′,π\(s′\)\)\]\.Q\_\{\\lambda\}^\{\\pi\}\(s,a\)=R\_\{\\lambda\}\(s,a\)\+\\gamma\\,\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\bigl\[Q\_\{\\lambda\}^\{\\pi\}\(s^\{\\prime\},\\pi\(s^\{\\prime\}\)\)\\bigr\]\.\(7\)Instead of the scalar valueQλπ​\(s,a\)Q\_\{\\lambda\}^\{\\pi\}\(s,a\), distributional reinforcement learning models the full return distribution\. LetZπ​\(s,a\)Z^\{\\pi\}\(s,a\)denote a random variable with the same distribution as the discounted return underRλR\_\{\\lambda\}starting from\(s,a\)\(s,a\)\. It satisfies the distributional Bellman equation:

Zπ​\(s,a\)=dRλ​\(s,a\)\+γ​Zπ​\(s′,a′\),Z^\{\\pi\}\(s,a\)\\;\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\\;R\_\{\\lambda\}\(s,a\)\+\\gamma\\,Z^\{\\pi\}\(s^\{\\prime\},a^\{\\prime\}\),\(8\)where=d\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}denotes equality in distribution,s′∼P\(⋅∣s,a\)s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)anda′∼π\(⋅∣s′\)a^\{\\prime\}\\sim\\pi\(\\cdot\\mid s^\{\\prime\}\)\.

### II\-BPhysical model and operational constraints

We instantiate the CMDP tuple by a radial DN model that determines the state and action spaces, the transition kernel, the operating cost, and the operational constraints\. The physical model specifies how the state evolves under control actions, the economic objective defines the reward, and the engineering limits induce the CMDP constraints in \([1](https://arxiv.org/html/2609.03308#S2.E1)\)–\([4](https://arxiv.org/html/2609.03308#S2.E4)\)\.

#### II\-B1States and actions

Consider a radial DN with bus set𝒩\\mathcal\{N\}, line setℒ\\mathcal\{L\}, distributed generators \(DGs\)𝒢\\mathcal\{G\}, energy storage systems \(ESSs\)ℰESS\\mathcal\{E\}^\{\\mathrm\{ESS\}\}, and shunt capacitor banks \(SCBs\)𝒞\\mathcal\{C\}, operated in discrete timet=0,1,…t=0,1,\\dotswith step lengthΔ​t\\Delta t\. All power quantities are defined positive when injected into the DN, and reactive powerQ\>0Q\>0\(resp\.Q<0Q<0\) denotes inductive \(resp\. capacitive\) injection\. The statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}collects bus voltage magnitudesVi,tV\_\{i,t\}and anglesθi,t\\theta\_\{i,t\}fori∈𝒩i\\in\\mathcal\{N\}, DG outputs, ESS states of charge \(SOC\), tap positions of on\-load tap changers \(OLTCs\), SCB statuses, and relevant exogenous information such as load and renewable forecasts\. The actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}aggregates DG active and reactive set\-points\{Pi,tDG,Qi,tDG\}i∈𝒢\\\{P\_\{i,t\}^\{\\mathrm\{DG\}\},Q\_\{i,t\}^\{\\mathrm\{DG\}\}\\\}\_\{i\\in\\mathcal\{G\}\}, ESS charging and discharging powers\{Pk,tch,Pk,tdis\}k∈ℰESS\\\{P\_\{k,t\}^\{\\mathrm\{ch\}\},P\_\{k,t\}^\{\\mathrm\{dis\}\}\\\}\_\{k\\in\\mathcal\{E\}^\{\\mathrm\{ESS\}\}\}, SCB switching commands and OLTC tap positions\.

#### II\-B2Operating cost and CMDP reward

The economic objective is encoded directly in the CMDP stage reward\. For each state–action pair\(st,at\)\(s\_\{t\},a\_\{t\}\), the reward is defined as the negative total operating cost, including energy purchase from the upstream grid, local generation costs, and network losses,

R⁡\(st,at\)\\displaystyle R\(s\_\{t\},a\_\{t\}\)=−\(Ptgrid​𝒫t\+∑i∈𝒢\(ai​\(Pi,tDG\)2\+bi​Pi,tDG\+ci\)CLOSE\\displaystyle=\-\\Big\(P^\{\\mathrm\{grid\}\}\_\{t\}\\,\\mathcal\{P\}\_\{t\}\+\\sum\_\{i\\in\\mathcal\{G\}\}\\big\(a\_\{i\}\(P\_\{i,t\}^\{\\mathrm\{DG\}\}\)^\{2\}\+b\_\{i\}P\_\{i,t\}^\{\\mathrm\{DG\}\}\+c\_\{i\}\\big\)\+αloss∑\(i,j\)∈ℒPi​j,tloss\),\\displaystyle\\qquad\\quad\+\\alpha\_\{\\mathrm\{loss\}\}\\sum\_\{\(i,j\)\\in\\mathcal\{L\}\}P\_\{ij,t\}^\{\\mathrm\{loss\}\}\\Big\),\(9\)where𝒫t\\mathcal\{P\}\_\{t\}is the electricity price,ai,bi,cia\_\{i\},b\_\{i\},c\_\{i\}are DG cost coefficients,Pi​j,tlossP\_\{ij,t\}^\{\\mathrm\{loss\}\}is the active loss on line\(i,j\)\(i,j\), andαloss\>0\\alpha\_\{\\mathrm\{loss\}\}\>0converts losses into monetary units\.

#### II\-B3Operational constraints

Operational security limits and device ratings are modeled as inequality constraints that define the feasible set of\(st,at\)\(s\_\{t\},a\_\{t\}\)\. These inequalities are encoded in the CMDP through the constraint functionsCiC\_\{i\}and boundsdid\_\{i\}in \([2](https://arxiv.org/html/2609.03308#S2.E2)\)–\([4](https://arxiv.org/html/2609.03308#S2.E4)\)\. Voltage magnitude limits and line thermal limits are enforced asVimin≤Vi,t≤Vimax,i∈𝒩,V\_\{i\}^\{\\min\}\\leq V\_\{i,t\}\\leq V\_\{i\}^\{\\max\},i\\in\\mathcal\{N\},\|Ii​j,t\|≤Ii​jmax,\(i,j\)∈ℒ,\|I\_\{ij,t\}\|\\leq I\_\{ij\}^\{\\max\},\(i,j\)\\in\\mathcal\{L\},whereVimin,VimaxV\_\{i\}^\{\\min\},V\_\{i\}^\{\\max\}andIi​jmaxI\_\{ij\}^\{\\max\}are voltage and current limits\. Each ESSk∈ℰESSk\\in\\mathcal\{E\}^\{\\mathrm\{ESS\}\}with rated energyEkESSE\_\{k\}^\{\\mathrm\{ESS\}\}satisfies the SOC dynamicsSOCk,t\+1=SOCk,t\+Δ​tEkESS​\(ηkch​Pk,tch−1ηkdis​Pk,tdis\),\\mathrm\{SOC\}\_\{k,t\+1\}=\\mathrm\{SOC\}\_\{k,t\}\+\\frac\{\\Delta t\}\{E\_\{k\}^\{\\mathrm\{ESS\}\}\}\\left\(\\eta\_\{k\}^\{\\mathrm\{ch\}\}P\_\{k,t\}^\{\\mathrm\{ch\}\}\-\\frac\{1\}\{\\eta\_\{k\}^\{\\mathrm\{dis\}\}\}P\_\{k,t\}^\{\\mathrm\{dis\}\}\\right\),together with SOC and power boundsSOCkmin≤SOCk,t≤SOCkmax,\\mathrm\{SOC\}\_\{k\}^\{\\min\}\\leq\\mathrm\{SOC\}\_\{k,t\}\\leq\\mathrm\{SOC\}\_\{k\}^\{\\max\},0≤Pk,tch≤Pkch,max,0\\leq P\_\{k,t\}^\{\\mathrm\{ch\}\}\\leq P\_\{k\}^\{\\mathrm\{ch\},\\max\},0≤Pk,tdis≤Pkdis,max,0\\leq P\_\{k,t\}^\{\\mathrm\{dis\}\}\\leq P\_\{k\}^\{\\mathrm\{dis\},\\max\},whereηkch,ηkdis∈\(0,1\]\\eta\_\{k\}^\{\\mathrm\{ch\}\},\\eta\_\{k\}^\{\\mathrm\{dis\}\}\\in\(0,1\]are charging and discharging efficiencies\. Device\-level constraints for each DGi∈𝒢i\\in\\mathcal\{G\}are0≤Pi,tDG≤PiDG,max,0\\leq P\_\{i,t\}^\{\\mathrm\{DG\}\}\\leq P\_\{i\}^\{\\mathrm\{DG\},\\max\},\(Pi,tDG\)2\+\(Qi,tDG\)2≤\(SiDG,max\)2,\\left\(P\_\{i,t\}^\{\\mathrm\{DG\}\}\\right\)^\{2\}\+\\left\(Q\_\{i,t\}^\{\\mathrm\{DG\}\}\\right\)^\{2\}\\leq\\left\(S\_\{i\}^\{\\mathrm\{DG\},\\max\}\\right\)^\{2\},cos⁡ϕimin≤\\cos\\phi\_\{i\}^\{\\min\}\\leqPi,tDG\(Pi,tDG\)2\+\(Qi,tDG\)2≤\\frac\{P\_\{i,t\}^\{\\mathrm\{DG\}\}\}\{\\sqrt\{\\left\(P\_\{i,t\}^\{\\mathrm\{DG\}\}\\right\)^\{2\}\+\\left\(Q\_\{i,t\}^\{\\mathrm\{DG\}\}\\right\)^\{2\}\}\}\\leqcos⁡ϕimax,\\cos\\phi\_\{i\}^\{\\max\},wherePiDG,maxP\_\{i\}^\{\\mathrm\{DG\},\\max\},SiDG,maxS\_\{i\}^\{\\mathrm\{DG\},\\max\}andcos⁡ϕimin,cos⁡ϕimax\\cos\\phi\_\{i\}^\{\\min\},\\cos\\phi\_\{i\}^\{\\max\}denote the corresponding ratings\. Each SCBc∈𝒞c\\in\\mathcal\{C\}hasncmaxn\_\{c\}^\{\\max\}identical steps\. Letnc,t∈\{0,…,ncmax\}n\_\{c,t\}\\in\\\{0,\\dots,n\_\{c\}^\{\\max\}\\\}be the number of energized steps andQcstepQ\_\{c\}^\{\\mathrm\{step\}\}the reactive power per step\. ThenQc,tSCB=nc,t​Qcstep,Q\_\{c,t\}^\{\\mathrm\{SCB\}\}=n\_\{c,t\}\\,Q\_\{c\}^\{\\mathrm\{step\}\},0≤nc,t≤ncmax\.0\\leq n\_\{c,t\}\\leq n\_\{c\}^\{\\max\}\.OLTC tap positions are likewise restricted to discrete sets\.

#### II\-B4Network constraints and AC power flow

LetGi​j\+j​Bi​jG\_\{ij\}\+jB\_\{ij\}be the\(i,j\)\(i,j\)th admittance matrix entry and𝒩i\\mathcal\{N\}\_\{i\}the set of buses connected to busii\. The nodal injections satisfy the AC power flow equationsPi,tinj=P\_\{i,t\}^\{\\mathrm\{inj\}\}=Vi,t​∑j∈𝒩iVj,tV\_\{i,t\}\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}\}V\_\{j,t\}\[Gi​j​cos⁡\(θi,t−θj,t\)\+Bi​j​sin⁡\(θi,t−θj,t\)\],\\Big\[G\_\{ij\}\\cos\(\\theta\_\{i,t\}\-\\theta\_\{j,t\}\)\+B\_\{ij\}\\sin\(\\theta\_\{i,t\}\-\\theta\_\{j,t\}\)\\Big\],Qi,tinj=Vi,t​∑j∈𝒩iVj,t​\[Gi​j​sin⁡\(θi,t−θj,t\)−Bi​j​cos⁡\(θi,t−θj,t\)\]\.Q\_\{i,t\}^\{\\mathrm\{inj\}\}=V\_\{i,t\}\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}\}V\_\{j,t\}\\Big\[G\_\{ij\}\\sin\(\\theta\_\{i,t\}\-\\theta\_\{j,t\}\)\-B\_\{ij\}\\cos\(\\theta\_\{i,t\}\-\\theta\_\{j,t\}\)\\Big\]\.For each non\-slack busii,Pi,tinj=Pi,tDG\+Pi,tdis−Pi,tch−Pi,tload,P\_\{i,t\}^\{\\mathrm\{inj\}\}=P\_\{i,t\}^\{\\mathrm\{DG\}\}\+P\_\{i,t\}^\{\\mathrm\{dis\}\}\-P\_\{i,t\}^\{\\mathrm\{ch\}\}\-P\_\{i,t\}^\{\\mathrm\{load\}\},Qi,tinj=Qi,tDG\+Qi,tSCB−Qi,tload,Q\_\{i,t\}^\{\\mathrm\{inj\}\}=Q\_\{i,t\}^\{\\mathrm\{DG\}\}\+Q\_\{i,t\}^\{\\mathrm\{SCB\}\}\-Q\_\{i,t\}^\{\\mathrm\{load\}\},while the slack bus injection includes the imported powerPtgridP^\{\\mathrm\{grid\}\}\_\{t\}\. For notational compactness, bus\-indexed device variables such asPi,tDG,Qi,tDG,Pi,tch,Pi,tdis,Qi,tSCBP\_\{i,t\}^\{\\mathrm\{DG\}\},Q\_\{i,t\}^\{\\mathrm\{DG\}\},P\_\{i,t\}^\{\\mathrm\{ch\}\},P\_\{i,t\}^\{\\mathrm\{dis\}\},Q\_\{i,t\}^\{\\mathrm\{SCB\}\}are understood as the aggregated injections of all corresponding devices connected to busiiat timett, and are defined to be zero if busiihosts no such device\.

#### II\-B5Scalar constraint cost

The operational limits of DGs, ESSs, SCBs, and OLTCs are handled by feasible action parameterization, action clipping, and device operating bounds\. Accordingly, the scalar constraint costC⁡\(st,at\)C\(s\_\{t\},a\_\{t\}\)used in the primal–dual update is constructed from the residual network\-security terms evaluated after the AC power\-flow calculation, including voltage violations, line or transformer overloading, and power\-flow non\-convergence\. LetVi,tV\_\{i,t\}denote the voltage magnitude at busii, and letℓe,t\\ell\_\{e,t\}denote the loading percentage of line or transformeree\. The aggregate voltage violation is defined asνV​\(st,at\)=∑i∈𝒩\(\[Vimin−Vi,t\]\+\+\[Vi,t−Vimax\]\+\)\\nu\_\{V\}\(s\_\{t\},a\_\{t\}\)=\\sum\_\{i\\in\\mathcal\{N\}\}\\left\(\[V\_\{i\}^\{\\min\}\-V\_\{i,t\}\]\_\{\+\}\+\[V\_\{i,t\}\-V\_\{i\}^\{\\max\}\]\_\{\+\}\\right\), and the aggregate thermal\-loading violation is defined asνL​\(st,at\)=∑e∈ℒ∪𝒯\[ℓe,t−100\]\+\\nu\_\{L\}\(s\_\{t\},a\_\{t\}\)=\\sum\_\{e\\in\\mathcal\{L\}\\cup\\mathcal\{T\}\}\[\\ell\_\{e,t\}\-100\]\_\{\+\}\. Here,νV\\nu\_\{V\}is measured in p\.u\., whereasνL\\nu\_\{L\}is measured in percentage points\. To combine these residual terms into a scalar constraint cost, normalization is absorbed into the penalty coefficients\. The resulting constraint cost isC⁡\(st,at\)=wV​νV​\(st,at\)\+wL​νL​\(st,at\)\+wpf​Ipf,tC\(s\_\{t\},a\_\{t\}\)=w\_\{V\}\\nu\_\{V\}\(s\_\{t\},a\_\{t\}\)\+w\_\{L\}\\nu\_\{L\}\(s\_\{t\},a\_\{t\}\)\+w\_\{\\mathrm\{pf\}\}I\_\{\\mathrm\{pf\},t\}, wherewVw\_\{V\},wLw\_\{L\}, andwpfw\_\{\\mathrm\{pf\}\}are penalty weights, andIpf,t∈\{0,1\}I\_\{\\mathrm\{pf\},t\}\\in\\\{0,1\\\}indicates whether the AC power flow fails to converge\.

## IIIUncertainty\-Aware DRL Framework

### III\-AFramework Overview

Building on the CMDP formulation in Section[II](https://arxiv.org/html/2609.03308#S2), we construct an uncertainty\-aware DRL framework with three components: a distributional actor–critic backbone, a second\-order UQ module, and an uncertainty\-aware training and deployment scheme\. The backbone is a deterministic actor–critic architecture with an implicit quantile network \(IQN\)\-based critic\[[22](https://arxiv.org/html/2609.03308#bib.bib22)\]that approximates the return distribution under the Lagrangian rewardRλR\_\{\\lambda\}, so that both performance and CMDP constraints are handled in a unified way\.

To quantify uncertainty, the critic produces multiple stochastic approximations of the return distribution for each state–action pair, using Monte Carlo dropout\[[23](https://arxiv.org/html/2609.03308#bib.bib23)\]or deep ensembles\[[24](https://arxiv.org/html/2609.03308#bib.bib24)\]\. These predictive distributions are aggregated to obtain total predictive uncertainty and then processed by a distance\-based second\-order scheme to decompose it into aleatoric and epistemic components\. The resulting AU captures irreducible stochasticity from RES, loads, and prices, while EU reflects model uncertainty due to limited data or distributional shift, including OOD conditions and cyber\-physical attacks\.

During training, EU serves as a curiosity signal to encourage exploration of unseen scenarios, while AU quantifies the inherent environmental risk\. During deployment, EU functions as a reliability indicator: if the EU associated with the actor’s action exceeds a threshold, the actor is treated as epistemically unreliable and control is handed over to a conservative mathematical programming\-based fallback strategy\.

The overall procedure is shown in Algorithm[1](https://arxiv.org/html/2609.03308#alg1)\.

Algorithm 1Lagrangian Distributional Actor–Critic With Second\-Order UQ and Fallback1:Input:replay buffer

𝒟\\mathcal\{D\}, discount factor

γ\\gamma, LM stepsize

ηλ\\eta\_\{\\lambda\}, tolerance

dd, ensemble size

MM, dropout samples

KK, missed\-detection tolerance

ϵmiss\\epsilon\_\{\\mathrm\{miss\}\}\.

2:Initialize actor

πϕ\\pi\_\{\\phi\}, target actor

πϕ¯\\pi\_\{\\bar\{\\phi\}\}, Bayesian distributional critics

\{Zψm\}m=1M\\\{Z\_\{\\psi\_\{m\}\}\\\}\_\{m=1\}^\{M\}, target critics

\{Zψ¯m\}m=1M\\\{Z\_\{\\bar\{\\psi\}\_\{m\}\}\\\}\_\{m=1\}^\{M\}, and Lagrange multiplier

λ≥0\\lambda\\geq 0\.

3:foreach training episodedo

4:Observe initial state

s0s\_\{0\}\.

5:for

t=0,1,…,T−1t=0,1,\\ldots,T\-1do

6:Construct the candidate action set

𝒜cand​\(st\)\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\)\.

7:For each

a∈𝒜cand​\(st\)a\\in\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\), estimate the critic mean return and

EUcrit​\(st,a\)\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\)using \([29](https://arxiv.org/html/2609.03308#S3.E29)\)–\([31](https://arxiv.org/html/2609.03308#S3.E31)\)\.

8:Select the exploratory action

ata\_\{t\}according to \([33](https://arxiv.org/html/2609.03308#S3.E33)\)\.

9:Execute

ata\_\{t\}, observe

st\+1s\_\{t\+1\},

R⁡\(st,at\)R\(s\_\{t\},a\_\{t\}\), and

C⁡\(st,at\)C\(s\_\{t\},a\_\{t\}\)\.

10:Compute

rtλr\_\{t\}^\{\\lambda\}by \([5](https://arxiv.org/html/2609.03308#S2.E5)\) and store the transition in

𝒟\\mathcal\{D\}\.

11:Sample a mini\-batch

ℬ\\mathcal\{B\}from

𝒟\\mathcal\{D\}\.

12:Update the Bayesian distributional critics by minimizing \([19](https://arxiv.org/html/2609.03308#S3.E19)\)\.

13:Update the actor by maximizing \([14](https://arxiv.org/html/2609.03308#S3.E14)\)\.

14:Update the Lagrange multiplier\.

15:Update target networks\.

16:endfor

17:endfor

18:Offline threshold calibration:

19:Construct

𝒟cal⊂𝒟hold\\mathcal\{D\}\_\{\\mathrm\{cal\}\}\\subset\\mathcal\{D\}\_\{\\mathrm\{hold\}\}\.

20:Form

𝒟calcrit\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}by retaining states with

Δ​C​\(s\)\>ηC\\Delta C\(s\)\>\\eta\_\{C\}or

Δ​R​\(s\)\>ηR\\Delta R\(s\)\>\\eta\_\{R\}\.

21:Compute

EUcrit​\(s,πϕ​\(s\)\)\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s,\\pi\_\{\\phi\}\(s\)\)for all

s∈𝒟calcrits\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}\.

22:Select

τfb\\tau\_\{\\mathrm\{fb\}\}by the calibration rule in \([36](https://arxiv.org/html/2609.03308#S3.E36)\)\.

23:Online deployment:

24:foreach deployment state

sts\_\{t\}do

25:Compute

aactor=πϕ​\(st\)a\_\{\\mathrm\{actor\}\}=\\pi\_\{\\phi\}\(s\_\{t\}\)and

EUcrit​\(st,aactor\)\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\_\{\\mathrm\{actor\}\}\)\.

26:Select the final control action by the fallback rule in \([37](https://arxiv.org/html/2609.03308#S3.E37)\)\.

27:endfor

### III\-BDistributional Actor–Critic Backbone

We adopt a distributional actor–critic architecture as the decision\-making backbone\. It operates on the Lagrangian MDP induced by the CMDP in Section[II](https://arxiv.org/html/2609.03308#S2), with scalar constraint costC⁡\(s,a\)C\(s,a\)and Lagrangian rewardRλ​\(s,a\)R\_\{\\lambda\}\(s,a\)defined in \([5](https://arxiv.org/html/2609.03308#S2.E5)\)\. The policy is parameterized by a deterministic actorπϕ:𝒮→𝒜\\pi\_\{\\phi\}:\\mathcal\{S\}\\to\\mathcal\{A\}, and the value function is represented by a distributional critic that approximates the return distributionZπZ^\{\\pi\}in \([8](https://arxiv.org/html/2609.03308#S2.E8)\)\. The critic follows the IQN formulation\[[22](https://arxiv.org/html/2609.03308#bib.bib22)\]\. Given a state–action pair\(s,a\)\(s,a\)and a sampled quantile fractionτ∈\(0,1\)\\tau\\in\(0,1\), the criticZψ​\(s,a,τ\)Z\_\{\\psi\}\(s,a,\\tau\)outputs an approximation of the corresponding quantile of the discounted return underRλR\_\{\\lambda\}\. Let\(st,at,rtλ,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\}^\{\\lambda\},s\_\{t\+1\}\)be a transition sampled from the replay buffer, wherertλ=Rλ​\(st,at\)r\_\{t\}^\{\\lambda\}=R\_\{\\lambda\}\(s\_\{t\},a\_\{t\}\), and letτ,τ′∼𝒰⁡\(0,1\)\\tau,\\tau^\{\\prime\}\\sim\\mathcal\{U\}\(0,1\)be independent quantile samples\. The distributional Bellman target is

z^t=rtλ\+γ​Zψ¯​\(st\+1,πϕ¯​\(st\+1\),τ′\),\\hat\{z\}\_\{t\}=r\_\{t\}^\{\\lambda\}\+\\gamma\\,Z\_\{\\bar\{\\psi\}\}\\bigl\(s\_\{t\+1\},\\pi\_\{\\bar\{\\phi\}\}\(s\_\{t\+1\}\),\\tau^\{\\prime\}\\bigr\),\(10\)whereψ¯\\bar\{\\psi\}andϕ¯\\bar\{\\phi\}denote target\-network parameters\. The critic is trained by minimizing the expected quantile Huber loss

ℒcrit=𝔼⁡\[ρτκ​\(z^t−Zψ​\(st,at,τ\)\)\],\\mathcal\{L\}\_\{\\text\{crit\}\}=\\mathbb\{E\}\\Bigl\[\\rho\_\{\\tau\}^\{\\kappa\}\\\!\\bigl\(\\hat\{z\}\_\{t\}\-Z\_\{\\psi\}\(s\_\{t\},a\_\{t\},\\tau\)\\bigr\)\\Bigr\],\(11\)whereκ\>0\\kappa\>0is the Huber threshold and

ρτκ​\(u\)\\displaystyle\\rho\_\{\\tau\}^\{\\kappa\}\(u\)=\|τ−𝟏\{u<0\}\|Lκ\(u\),\\displaystyle=\\bigl\|\\tau\-\\mathbf\{1\}\_\{\\\{u<0\\\}\}\\bigr\|L\_\{\\kappa\}\(u\),\(12\)Lκ​\(u\)\\displaystyle L\_\{\\kappa\}\(u\)=\{12​u2,\|u\|≤κ,κ⁡\(\|u\|−12​κ\),\|u\|\>κ,\\displaystyle=\\begin\{cases\}\\frac\{1\}\{2\}u^\{2\},&\|u\|\\leq\\kappa,\\\\\[1\.99997pt\] \\kappa\\bigl\(\|u\|\-\\frac\{1\}\{2\}\\kappa\\bigr\),&\|u\|\>\\kappa,\\end\{cases\}\(13\)is the standard smoothed asymmetric quantile regression loss\. The actor is updated by maximizing the expected Lagrangian return estimated by the distributional critic,

Jact​\(ϕ\)=𝔼s∼𝒟​\[𝔼τ​\[Zψ​\(s,πϕ​\(s\),τ\)\]\],J\_\{\\text\{act\}\}\(\\phi\)=\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\}\}\\Bigl\[\\mathbb\{E\}\_\{\\tau\}\\bigl\[Z\_\{\\psi\}\\bigl\(s,\\pi\_\{\\phi\}\(s\),\\tau\\bigr\)\\bigr\]\\Bigr\],\(14\)where𝒟\\mathcal\{D\}is the state distribution induced by the replay buffer\. The policy parametersϕ\\phiare updated by gradient ascent onJact​\(ϕ\)J\_\{\\text\{act\}\}\(\\phi\), while the Lagrange multiplierλ\\lambdais updated online using stochastic gradient ascent on the dual objective based on sampled constraint costsC⁡\(st,at\)C\(s\_\{t\},a\_\{t\}\)\. Even without the UQ introduced below, this backbone already yields a standard constrained distributional RL scheme for DN operation under the CMDP model in Section[II](https://arxiv.org/html/2609.03308#S2)\.

### III\-CBayesian Distributional Critic

Starting from the deterministic distributional actor–critic backbone in Section[III](https://arxiv.org/html/2609.03308#S3), we now endow the distributional critic with Bayesian structure\. We treat the critic networks as approximate Bayesian neural networks and obtain stochastic predictions via a combination of Monte Carlo dropout\[[23](https://arxiv.org/html/2609.03308#bib.bib23)\]and deep ensembles\[[24](https://arxiv.org/html/2609.03308#bib.bib24)\]\. Monte Carlo dropout perturbs a single network and yields an essentially unimodal approximate posterior around one mode, whereas deep ensembles maintain multiple independently trained networks and thus approximate a multi\-modal posterior over value and policy functions\. More specifically, we maintain an ensemble ofMMIQN\-based distributional critics\{Zψm\}m=1M\\\{Z\_\{\\psi\_\{m\}\}\\\}\_\{m=1\}^\{M\}with parametersψm\\psi\_\{m\}\. Each critic approximates the quantile function of the Lagrangian return distribution under the current policy\. Given a state–action pair\(s,a\)\(s,a\), a quantile fractionτ∈\(0,1\)\\tau\\in\(0,1\), and a dropout maskξm,k\\xi\_\{m,k\}, the stochastic critic output is

zm,k​\(s,a,τ\)\\displaystyle z\_\{m,k\}\(s,a,\\tau\)=Zψm​\(s,a,τ,ξm,k\),\\displaystyle=Z\_\{\\psi\_\{m\}\}\\bigl\(s,a,\\tau;\\xi\_\{m,k\}\\bigr\),\(15\)m=1,…,M,k=1,…,K,\\displaystyle m=1,\\dots,M,\\;k=1,\\dots,K,where dropout masksξm,k\\xi\_\{m,k\}are sampled i\.i\.d\. as in\[[23](https://arxiv.org/html/2609.03308#bib.bib23)\]\. For fixed\(s,a\)\(s,a\)and critic membermm, the mappingτ↦zm,k​\(s,a,τ\)\\tau\\mapsto z\_\{m,k\}\(s,a,\\tau\)defines a stochastic approximation of the return quantile function under thekk\-th dropout realization\.

Each critic member is trained with the IQN loss \([11](https://arxiv.org/html/2609.03308#S3.E11)\), using the actor actiona⁡\(st\+1\)a\(s\_\{t\+1\}\)as input to the target network\. Specifically, for a transition\(st,at,rtλ,st\+1\)\(s\_\{t\},a\_\{t\},r\_\{t\}^\{\\lambda\},s\_\{t\+1\}\)in the replay buffer, the Bellman target for criticmmunder dropout maskξm,k\\xi\_\{m,k\}is

z^t,m,k=rtλ\+γ​Zψ¯m​\(st\+1,a¯​\(st\+1\),τ′,ξm,k\),\\hat\{z\}\_\{t,m,k\}=r\_\{t\}^\{\\lambda\}\+\\gamma\\,Z\_\{\\bar\{\\psi\}\_\{m\}\}\\bigl\(s\_\{t\+1\},\\bar\{a\}\(s\_\{t\+1\}\),\\tau^\{\\prime\};\\xi\_\{m,k\}\\bigr\),\(16\)whereψ¯m\\bar\{\\psi\}\_\{m\}denotes the target parameters of criticmmandτ′∼𝒰⁡\(0,1\)\\tau^\{\\prime\}\\sim\\mathcal\{U\}\(0,1\)\. The critic loss for membermmon a mini\-batchℬ\\mathcal\{B\}is

ℒcrit\(m\)\\displaystyle\\mathcal\{L\}\_\{\\text\{crit\}\}^\{\(m\)\}=𝔼\(st,at,rtλ,st\+1\)∈ℬ𝔼τ,τ′𝔼k\[ρτκ\(z^t,m,k−\\displaystyle=\\mathbb\{E\}\_\{\(s\_\{t\},a\_\{t\},r\_\{t\}^\{\\lambda\},s\_\{t\+1\}\)\\in\\mathcal\{B\}\}\\mathbb\{E\}\_\{\\tau,\\tau^\{\\prime\}\}\\mathbb\{E\}\_\{k\}\\Bigl\[\\rho\_\{\\tau\}^\{\\kappa\}\\\!\\bigl\(\\hat\{z\}\_\{t,m,k\}\-\(17\)Zψm\(st,at,τ;ξm,k\)\)\],\\displaystyle Z\_\{\\psi\_\{m\}\}\(s\_\{t\},a\_\{t\},\\tau;\\xi\_\{m,k\}\)\\bigr\)\\Bigr\],where the expectations overτ,τ′\\tau,\\tau^\{\\prime\}are with respect to the uniform distribution on\(0,1\)\(0,1\), and the expectation overkkaverages over dropout realizations\. To prevent ensemble collapse and encourage meaningful epistemic spread, we use a repulsive diversity regularizer based on their mean return estimatesμm​\(s,a\)=𝔼τ​\[Zψm​\(s,a,τ\)\]\\mu\_\{m\}\(s,a\)=\\mathbb\{E\}\_\{\\tau\}\[Z\_\{\\psi\_\{m\}\}\(s,a,\\tau\)\]:

ℒdiv=𝔼\(s,a\)∈ℬ​\[2M⁡\(M−1\)​∑m<m′\(μm​\(s,a\)−μm′​\(s,a\)\)2\],\\mathcal\{L\}\_\{\\text\{div\}\}=\\mathbb\{E\}\_\{\(s,a\)\\in\\mathcal\{B\}\}\\Biggl\[\\frac\{2\}\{M\(M\-1\)\}\\sum\_\{m<m^\{\\prime\}\}\\bigl\(\\mu\_\{m\}\(s,a\)\-\\mu\_\{m^\{\\prime\}\}\(s,a\)\\bigr\)^\{2\}\\Biggr\],\(18\)ℒBayes=1M​∑m=1Mℒcrit\(m\)−βdiv​ℒdiv\.\\mathcal\{L\}\_\{\\text\{Bayes\}\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\mathcal\{L\}\_\{\\text\{crit\}\}^\{\(m\)\}\-\\beta\_\{\\text\{div\}\}\\,\\mathcal\{L\}\_\{\\text\{div\}\}\.\(19\)
The Bayesian critic module therefore producesB=M​KB=MKstochastic return quantile functions, which are flattened into the return\-law samples\{Pb\}b=1B\\\{P\_\{b\}\\\}\_\{b=1\}^\{B\}used in Section[III\-D2](https://arxiv.org/html/2609.03308#S3.SS4.SSS2)\.

### III\-DDistance\-based Second\-order UQ

#### III\-D1General distance\-based second\-order UQ framework

We adopt the distance\-based second\-order UQ framework of\[[25](https://arxiv.org/html/2609.03308#bib.bib25)\]\. LetYYbe an outcome space equipped with its Borelσ\\sigma\-algebra\. A first\-order predictive distribution is a probability measurep∈𝒫⁡\(Y\)p\\in\\mathcal\{P\}\(Y\)\. Higher\-order uncertainty about the predictive distribution itself is modeled by a*second\-order*distributionQ∈𝒪⁡\(Y\):=𝒫⁡\(𝒫⁡\(Y\)\)Q\\in\\mathcal\{O\}\(Y\):=\\mathcal\{P\}\(\\mathcal\{P\}\(Y\)\), namely a distribution overp∈𝒫⁡\(Y\)p\\in\\mathcal\{P\}\(Y\)\. To compare second\-order distributions, we first equip the first\-order space𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)with a metricd1:𝒫⁡\(Y\)×𝒫⁡\(Y\)→\[0,∞\)d\_\{1\}:\\mathcal\{P\}\(Y\)\\times\\mathcal\{P\}\(Y\)\\to\[0,\\infty\)\. Givend1d\_\{1\}, we measure distances on the second\-order space𝒪⁡\(Y\)\\mathcal\{O\}\(Y\)by the Wasserstein–ppmetric induced byd1d\_\{1\}, withp≥1p\\geq 1: forQ,Q′∈𝒪⁡\(Y\)Q,Q^\{\\prime\}\\in\\mathcal\{O\}\(Y\),

Wp​\(Q,Q′\)=infγ∈Γ⁡\(Q,Q′\)\(∫𝒫⁡\(Y\)×𝒫⁡\(Y\)d1p​\(p,p~\)​𝑑γ​\(p,p~\)\)1/p,W\_\{p\}\(Q,Q^\{\\prime\}\)=\\inf\_\{\\gamma\\in\\Gamma\(Q,Q^\{\\prime\}\)\}\\Biggl\(\\int\_\{\\mathcal\{P\}\(Y\)\\times\\mathcal\{P\}\(Y\)\}d\_\{1\}^\{p\}\(p,\\tilde\{p\}\)\\,\\mathrm\{d\}\\gamma\(p,\\tilde\{p\}\)\\Biggr\)^\{1/p\},\(20\)whereΓ⁡\(Q,Q′\)\\Gamma\(Q,Q^\{\\prime\}\)is the set of all couplings betweenQQandQ′Q^\{\\prime\}\. Intuitively,Wp​\(Q,Q′\)W\_\{p\}\(Q,Q^\{\\prime\}\)is the minimal transport cost needed to move mass fromQQtoQ′Q^\{\\prime\}when transporting one unit of mass frompptop~\\tilde\{p\}costsd1​\(p,p~\)d\_\{1\}\(p,\\tilde\{p\}\)\. The distance\-based construction in\[[25](https://arxiv.org/html/2609.03308#bib.bib25)\]quantifies total, aleatoric, and epistemic uncertainty by measuring the distance of a givenQ∈𝒪⁡\(Y\)Q\\in\\mathcal\{O\}\(Y\)to three canonical*reference families*\. Crucially, these families are chosen to represent*least\-uncertain*states for the corresponding uncertainty type\.

##### Reference family for total uncertainty

Total certainty corresponds to knowing both the outcome and the predictive distribution exactly\. This is represented by second\-order Dirac masses at first\-order Dirac measures:

𝒮tot=\{δδy:y∈Y\}⊂𝒪⁡\(Y\),\\mathcal\{S\}\_\{\\mathrm\{tot\}\}=\\bigl\\\{\\delta\_\{\\delta\_\{y\}\}:y\\in Y\\bigr\\\}\\subset\\mathcal\{O\}\(Y\),\(21\)whereδy∈𝒫⁡\(Y\)\\delta\_\{y\}\\in\\mathcal\{P\}\(Y\)is the Dirac measure atyy, andδδy∈𝒪⁡\(Y\)\\delta\_\{\\delta\_\{y\}\}\\in\\mathcal\{O\}\(Y\)is the Dirac mass atδy\\delta\_\{y\}\. Elements of𝒮tot\\mathcal\{S\}\_\{\\mathrm\{tot\}\}carry neither first\-order nor second\-order uncertainty\.

##### Reference family for aleatoric uncertainty \(zero\-aleatoric states\)

To isolate aleatoric uncertainty,\[[25](https://arxiv.org/html/2609.03308#bib.bib25)\]uses as reference the set of second\-order distributions supported on*deterministic*predictors \(first\-order Dirac measures\)\. Concretely, for anym∈𝒫⁡\(Y\)m\\in\\mathcal\{P\}\(Y\), define

δm=∫Yδδy​𝑑m​\(y\),\\delta\_\{m\}=\\int\_\{Y\}\\delta\_\{\\delta\_\{y\}\}\\,\\mathrm\{d\}m\(y\),\(22\)and set

𝒮al=\{δm:m∈𝒫⁡\(Y\)\}⊂𝒪⁡\(Y\)\.\\mathcal\{S\}\_\{\\mathrm\{al\}\}=\\Bigl\\\{\\delta\_\{m\}:m\\in\\mathcal\{P\}\(Y\)\\Bigr\\\}\\subset\\mathcal\{O\}\(Y\)\.\(23\)EveryQ′∈𝒮alQ^\{\\prime\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}assigns probability one to first\-order Dirac measures, hence it has*no aleatoric \(first\-order\)*uncertainty\. Measuring the distance of a generalQQto𝒮al\\mathcal\{S\}\_\{\\mathrm\{al\}\}therefore quantifies how farQQis from having deterministic first\-order predictors\.

##### Reference family for epistemic uncertainty \(zero\-epistemic states\)

To remove epistemic uncertainty, the reference family is the set of second\-order Dirac measures on arbitrary first\-order distributions:

𝒮ep=\{δp:p∈𝒫⁡\(Y\)\}⊂𝒪⁡\(Y\)\.\\mathcal\{S\}\_\{\\mathrm\{ep\}\}=\\bigl\\\{\\delta\_\{p\}:p\\in\\mathcal\{P\}\(Y\)\\bigr\\\}\\subset\\mathcal\{O\}\(Y\)\.\(24\)An elementδp\\delta\_\{p\}fixes the predictive distributionppand thus eliminates second\-order randomness, meaning it has*no epistemic \(second\-order\)*uncertainty, whileppitself may remain diffuse\.

Given these reference families, the distance\-based indices are defined as

Utot​\(Q\)\\displaystyle U\_\{\\mathrm\{tot\}\}\(Q\)=infQ′∈𝒮totWp​\(Q,Q′\),\\displaystyle=\\inf\_\{Q^\{\\prime\}\\in\\mathcal\{S\}\_\{\\mathrm\{tot\}\}\}W\_\{p\}\\bigl\(Q,Q^\{\\prime\}\\bigr\),\(25\)Ual​\(Q\)\\displaystyle U\_\{\\mathrm\{al\}\}\(Q\)=infQ′∈𝒮alWp​\(Q,Q′\),\\displaystyle=\\inf\_\{Q^\{\\prime\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}\}W\_\{p\}\\bigl\(Q,Q^\{\\prime\}\\bigr\),\(26\)Uep​\(Q\)\\displaystyle U\_\{\\mathrm\{ep\}\}\(Q\)=infQ′∈𝒮epWp​\(Q,Q′\)\.\\displaystyle=\\inf\_\{Q^\{\\prime\}\\in\\mathcal\{S\}\_\{\\mathrm\{ep\}\}\}W\_\{p\}\\bigl\(Q,Q^\{\\prime\}\\bigr\)\.\(27\)Thus,Utot​\(Q\)U\_\{\\mathrm\{tot\}\}\(Q\)measures the distance to complete certainty,Ual​\(Q\)U\_\{\\mathrm\{al\}\}\(Q\)measures the distance to the family of zero\-aleatoric reference states, andUep​\(Q\)U\_\{\\mathrm\{ep\}\}\(Q\)measures the distance to the family of zero\-epistemic reference states\. For suitable choices ofd1d\_\{1\}, these indices satisfy the axioms proposed in\[[25](https://arxiv.org/html/2609.03308#bib.bib25)\], including invariance and a consistent separation of first\-order \(aleatoric\) and second\-order \(epistemic\) effects\.

#### III\-D2Return\-based second\-order UQ for the critic

We now instantiate the distance\-based framework of Section[III\-D1](https://arxiv.org/html/2609.03308#S3.SS4.SSS1)on the one\-dimensional return space and use it to quantify second\-order uncertainty for the critic\.

##### Outcome space and metrics on return laws

LetY⊂ℝY\\subset\\mathbb\{R\}denote the return space, and let𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)be the corresponding space of return laws\. We useW2W\_\{2\}for the 2\-Wasserstein distance between return laws, induced by the squared Euclidean cost onYY; the induced second\-order construction is detailed in Appendix[A](https://arxiv.org/html/2609.03308#A1)\. In the one\-dimensional setting, the squared first\-order 2\-Wasserstein distance on𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)admits the quantile representation

\(W2​\(p,p~\)\)2=∫01\[Qp​\(τ\)−Qp~​\(τ\)\]2​𝑑τ,\\bigl\(W\_\{2\}\(p,\\tilde\{p\}\)\\bigr\)^\{2\}=\\int\_\{0\}^\{1\}\\bigl\[Q\_\{p\}\(\\tau\)\-Q\_\{\\tilde\{p\}\}\(\\tau\)\\bigr\]^\{2\}\\,\\mathrm\{d\}\\tau,\(28\)whereQpQ\_\{p\}andQp~Q\_\{\\tilde\{p\}\}are the quantile functions ofp,p~∈𝒫⁡\(Y\)p,\\tilde\{p\}\\in\\mathcal\{P\}\(Y\), see Lemma[2](https://arxiv.org/html/2609.03308#Thmlemma2)in Appendix[A](https://arxiv.org/html/2609.03308#A1)\. The IQN critic provides direct access to approximate quantiles, which makes \([28](https://arxiv.org/html/2609.03308#S3.E28)\) convenient for numerical evaluation\. In what follows, we work with the squared first\-order Wasserstein costW22W\_\{2\}^\{2\}; since the mappingr↦r2r\\mapsto r^\{2\}is monotone on\[0,∞\)\[0,\\infty\), this corresponds to a monotone transformation of the indices in \([25](https://arxiv.org/html/2609.03308#S3.E25)\)–\([27](https://arxiv.org/html/2609.03308#S3.E27)\) without changing their ordering\. Detailed derivations are given in Appendix[A](https://arxiv.org/html/2609.03308#A1)\.

##### Second\-order UQ for the distributional critic

For each state–action pair\(s,a\)\(s,a\), the Bayesian distributional critic \(with ensembles and Monte Carlo dropout\) yieldsBBstochastic predictive return laws\. For compactness, we writePb:=pcrit\(b\)\(⋅\|s,a\)P\_\{b\}:=p\_\{\\mathrm\{crit\}\}^\{\(b\)\}\(\\cdot\|s,a\),b=1,…,Bb=1,\\dots,B, andP⋆:=pcrit⋆\(⋅\|s,a\)P\_\{\\star\}:=p\_\{\\mathrm\{crit\}\}^\{\\star\}\(\\cdot\|s,a\), omitting the dependence on\(s,a\)\(s,a\)when no ambiguity arises\. Here,BBis the total number of stochastic forward passes\. These draws induce the empirical second\-order measure

Q^B​\(s,a\)=1B​∑b=1BδPb,\\widehat\{Q\}\_\{B\}\(s,a\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\delta\_\{P\_\{b\}\},\(29\)which summarizes higher\-order uncertainty over return distributions at\(s,a\)\(s,a\)\. For interpretation, we may view\{Pb\}b=1B\\\{P\_\{b\}\\\}\_\{b=1\}^\{B\}as i\.i\.d\. samples from an underlying population lawQcrit​\(s,a\)Q\_\{\\mathrm\{crit\}\}\(s,a\)\. We choose the ground metric on𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)asd1=W2d\_\{1\}=W\_\{2\}\. Under this choice, critic epistemic uncertainty corresponds to the Fréchet dispersion ofQ^B​\(s,a\)\\widehat\{Q\}\_\{B\}\(s,a\)in\(𝒫⁡\(Y\),W2\)\(\\mathcal\{P\}\(Y\),W\_\{2\}\)\. Let

P⋆∈arg​minp∈𝒫⁡\(Y\)⁡1B​∑b=1B\(W2​\(Pb,p\)\)2\.P\_\{\\star\}\\in\\argmin\_\{p\\in\\mathcal\{P\}\(Y\)\}\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\bigl\(W\_\{2\}\(P\_\{b\},p\)\\bigr\)^\{2\}\.\(30\)be a Wasserstein barycenter of theBBreturn laws\. In one dimension it has a closed form via quantile averaging: ifFb−1​\(τ∣s,a\)F\_\{b\}^\{\-1\}\(\\tau\\mid s,a\)is the generalized quantile ofPbP\_\{b\}, thenF⋆−1​\(τ∣s,a\)=1B​∑b=1BFb−1​\(τ∣s,a\)F\_\{\\star\}^\{\-1\}\(\\tau\\mid s,a\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}F\_\{b\}^\{\-1\}\(\\tau\\mid s,a\)for allτ∈\(0,1\)\\tau\\in\(0,1\); see Lemma[3](https://arxiv.org/html/2609.03308#Thmlemma3)in Appendix[A](https://arxiv.org/html/2609.03308#A1)\. We then define the critic epistemic and aleatoric indices as

EUcrit​\(s,a\):=1B​∑b=1B\(W2​\(Pb,P⋆\)\)2,\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s,a\):=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\bigl\(W\_\{2\}\(P\_\{b\},P\_\{\\star\}\)\\bigr\)^\{2\},\(31\)AUcrit​\(s,a\):=1B​∑b=1BVarY∼Pb​\(Y\),\\mathrm\{AU\}\_\{\\mathrm\{crit\}\}\(s,a\):=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathrm\{Var\}\_\{Y\\sim P\_\{b\}\}\(Y\),\(32\)whereEUcrit\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}measures disagreement across stochastic return laws around their barycenter, andAUcrit\\mathrm\{AU\}\_\{\\mathrm\{crit\}\}averages the intrinsic variance within each return law\. Appendix[A](https://arxiv.org/html/2609.03308#A1)shows that \([30](https://arxiv.org/html/2609.03308#S3.E30)\)–\([32](https://arxiv.org/html/2609.03308#S3.E32)\) match the general distance\-based definitions of Section[III\-D1](https://arxiv.org/html/2609.03308#S3.SS4.SSS1)under the chosen metrics, and Appendix[B](https://arxiv.org/html/2609.03308#A2)establishes almost\-sure consistency of the Monte Carlo estimators in \([31](https://arxiv.org/html/2609.03308#S3.E31)\)–\([32](https://arxiv.org/html/2609.03308#S3.E32)\)\.

### III\-EUncertainty\-aware Exploration

We now describe how EU is used during training\. At each training statests\_\{t\}, we form a finite feasible candidate action set𝒜cand​\(st\)\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\)by assigning admissible discrete choices to individual discrete devices and sampling bounded local perturbations around the actor output for continuous controls\. The unperturbed actor action is also included in𝒜cand​\(st\)\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\)\. For each candidate actiona∈𝒜cand​\(st\)a\\in\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\), we evaluate both the critic mean return and the critic\-side epistemic uncertainty\. The exploratory behavior action is selected according to the upper confidence rule

atexpl∈arg​maxa∈𝒜cand​\(st\)⁡𝔼τ​\[Zψ​\(st,a,τ\)\]\+αexpl​EUcrit​\(st,a\),a\_\{t\}^\{\\mathrm\{expl\}\}\\in\\argmax\_\{a\\in\\mathcal\{A\}\_\{\\mathrm\{cand\}\}\(s\_\{t\}\)\}\\mathbb\{E\}\_\{\\tau\}\\bigl\[Z\_\{\\psi\}\(s\_\{t\},a,\\tau\)\\bigr\]\+\\alpha\_\{\\mathrm\{expl\}\}\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\),\(33\)whereαexpl\>0\\alpha\_\{\\mathrm\{expl\}\}\>0controls the relative strength of the epistemic bonus\. To balance exploitation and uncertainty\-driven exploration,αexpl\\alpha\_\{\\mathrm\{expl\}\}is adapted according to the running magnitudes of the two terms in \([33](https://arxiv.org/html/2609.03308#S3.E33)\)\. Specifically, during training we maintain exponential moving estimates of the absolute critic\-return term and the raw EU term over sampled candidate actions, and updateαexpl\\alpha\_\{\\mathrm\{expl\}\}so that the averaged bonusαexpl​EUcrit​\(st,a\)\\alpha\_\{\\mathrm\{expl\}\}\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\)occupies a prescribed fraction of the critic\-return magnitude\. This target fraction is gradually reduced as training proceeds, encouraging broader exploration in the early stage and more exploitation\-oriented behavior after the critic ensemble becomes better calibrated\. The epistemic term therefore acts as an intrinsic bonus\[[26](https://arxiv.org/html/2609.03308#bib.bib26)\]that directs exploration toward insufficiently learned state\-action regions\.

### III\-FUncertainty\-guided fallback control

To safeguard deployment under distributional shift, we equip the agent with an uncertainty\-guided fallback mechanism\. The fallback controller is a single\-step mixed\-integer second\-order cone programming \(MISOCP\)\-based optimal power flow, which solves a convex relaxation of the one\-period DN operation problem with the same network and device constraints as in Section[II](https://arxiv.org/html/2609.03308#S2), e\.g\. based on the branch\-flow model and its convexification\[[27](https://arxiv.org/html/2609.03308#bib.bib27)\]\. The DG constraints are handled by keeping the capability limit as a second\-order cone and rewriting the power\-factor bound into linear inequalities−κi​Pi,tDG≤Qi,tDG≤κi​Pi,tDG\-\\kappa\_\{i\}P\_\{i,t\}^\{\\mathrm\{DG\}\}\\leq Q\_\{i,t\}^\{\\mathrm\{DG\}\}\\leq\\kappa\_\{i\}P\_\{i,t\}^\{\\mathrm\{DG\}\}, whereκi=tan⁡\(ϕimax\)\\kappa\_\{i\}=\\tan\(\\phi\_\{i\}^\{\\max\}\)\(and similarly for the lower bound viaϕimin\\phi\_\{i\}^\{\\min\}when asymmetric limits are imposed\)\. Given the current statess, this MISOCP returns a feasible actionaMISOCP​\(s\)a^\{\\mathrm\{MISOCP\}\}\(s\)that minimizes the instantaneous operating cost while enforcing the modeled single\-step constraints\. It serves as a conservative yet reliable baseline when the RL policy is deemed epistemically unreliable\.

##### Offline calibration of epistemic thresholds

We calibrate the epistemic threshold on a separate hold\-out set𝒟hold\\mathcal\{D\}\_\{\\mathrm\{hold\}\}that is not used for training the deployed policy\. On this hold\-out environment, we fix a reference optimal policyπ⋆\\pi^\{\\star\}and use it to define state\-wise performance benchmarks\. For any statessand policyπ\\pi, letJRπ​\(s\)J\_\{R\}^\{\\pi\}\(s\)andJCπ​\(s\)J\_\{C\}^\{\\pi\}\(s\)denote the discounted return and aggregated constraint return starting fromss, as in \([1](https://arxiv.org/html/2609.03308#S2.E1)\)–\([2](https://arxiv.org/html/2609.03308#S2.E2)\), withs0=ss\_\{0\}=sandCCthe scalar constraint cost in Section[II](https://arxiv.org/html/2609.03308#S2)\. For the deployed policyπ\\piand the reference policyπ⋆\\pi^\{\\star\}, we define the long\-horizon constraint and reward gaps as

Δ​C​\(s\)\\displaystyle\\Delta C\(s\)=max⁡\(0,JCπ​\(s\)−JC⋆​\(s\)\),\\displaystyle=\\max\\\!\\left\(0,\\,J\_\{C\}^\{\\pi\}\(s\)\-J\_\{C\}^\{\\star\}\(s\)\\right\),\(34\)Δ​R​\(s\)\\displaystyle\\Delta R\(s\)=max⁡\(0,JR⋆​\(s\)−JRπ​\(s\)\)\.\\displaystyle=\\max\\\!\\left\(0,\\,J\_\{R\}^\{\\star\}\(s\)\-J\_\{R\}^\{\\pi\}\(s\)\\right\)\.\(35\)Thus,Δ​C​\(s\)\>0\\Delta C\(s\)\>0means that the deployed policy incurs higher cumulative constraint cost than the reference policy, whileΔ​R​\(s\)\>0\\Delta R\(s\)\>0means that it obtains lower cumulative reward\. GivenηC,ηR≥0\\eta\_\{C\},\\eta\_\{R\}\\geq 0, let𝒟calcrit\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}collect the hold\-out calibration states withΔ​C​\(s\)\>ηC\\Delta C\(s\)\>\\eta\_\{C\}orΔ​R​\(s\)\>ηR\\Delta R\(s\)\>\\eta\_\{R\}\. This set is separate from the training data and the final evaluation episodes\. For eachs∈𝒟calcrits\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}, we computees=EUcrit​\(s,πϕ​\(s\)\)e\_\{s\}=\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s,\\pi\_\{\\phi\}\(s\)\)\. The fallback threshold is chosen as the empiricalϵmiss\\epsilon\_\{\\mathrm\{miss\}\}\-lower quantile:

τfb=Q^ϵmiss​\(\{es:s∈𝒟calcrit\}\)\.\\tau\_\{\\mathrm\{fb\}\}=\\widehat\{Q\}\_\{\\epsilon\_\{\\mathrm\{miss\}\}\}\\left\(\\\{e\_\{s\}:s\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}\\\}\\right\)\.\(36\)This rule directly controls the fraction of critical calibration states that are covered by fallback\.

##### Online decision rule

At deployment time, given the current statests\_\{t\}, the agent first computes the actor actionaactor=πϕ​\(st\)a\_\{\\mathrm\{actor\}\}=\\pi\_\{\\phi\}\(s\_\{t\}\)and the corresponding epistemic scoreEUcrit​\(st,aactor\)\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\_\{\\mathrm\{actor\}\}\)\. The final control action is selected by

at=\{aMISOCP​\(st\),EUcrit​\(st,aactor\)≥τfb,aactor,EUcrit​\(st,aactor\)<τfb\.a\_\{t\}=\\begin\{cases\}a^\{\\mathrm\{MISOCP\}\}\(s\_\{t\}\),&\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\_\{\\mathrm\{actor\}\}\)\\geq\\tau\_\{\\mathrm\{fb\}\},\\\\\[3\.00003pt\] a\_\{\\mathrm\{actor\}\},&\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\_\{\\mathrm\{actor\}\}\)<\\tau\_\{\\mathrm\{fb\}\}\.\\end\{cases\}\(37\)States withEUcrit​\(st,aactor\)≥τfb\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s\_\{t\},a\_\{\\mathrm\{actor\}\}\)\\geq\\tau\_\{\\mathrm\{fb\}\}are treated as epistemically unreliable and handed over to the MISOCP\-based fallback controller\.

## IVSimulation Results

### IV\-AExperimental Protocol and OOD Scenario Construction

#### IV\-A1Test Cases and Network Adaptation

We organize the numerical study around two test cases\.Test Case Iis based on the MV Oberrhein network in pandapower\[[28](https://arxiv.org/html/2609.03308#bib.bib28)\], a realistic 20 kV medium\-voltage distribution network supplied by two substations\. The original static profiles are converted into a 24\-hour operation task by using UCI load curves\[[29](https://arxiv.org/html/2609.03308#bib.bib29)\]while preserving the original load power factors, and Kaggle PV generation curves\[[30](https://arxiv.org/html/2609.03308#bib.bib30)\]for renewable injections\. The electricity price during 8:00–21:00 is set to 1\.558 times the off\-peak price\. We add 10 SCBs, each with four 0\.12 MVAR steps, and 10 ESSs, each with 2 MWh capacity and 0\.5 MW maximum charging/discharging power\. Among the 153 static generators in the original network, one third are treated as PV units and the rest as diesel DGs with a minimum power factor of 0\.7\. Bus voltages are constrained within 0\.95–1\.05 p\.u\., and the two OLTC transformers are modeled with tap positions from−9\-9to99and a 1\.5% tap step\.Test Case IIis adapted from the IEEE European low\-voltage network in pandapower\[[28](https://arxiv.org/html/2609.03308#bib.bib28)\]\. The original benchmark is a 0\.416 kV radial feeder with 907 buses and 905 lines\. We convert this benchmark into a balanced three\-phase operation case by assigning equal per\-phase load and generation profiles at each active bus\. We extend the transformer to a controllable tap\-changing transformer\. We add 51 PV units\. The action space includes 102 dispatchable DGs, 13 ESSs, 10 SCBs, and 1 OLTC transformer\. These devices form a 286\-dimensional action space\.

To check whether the constraint\-compliance behavior of the proposed method is sensitive to the scalarization of network\-security violations, we conduct a penalty\-ratio ablation in Test Case II\. The default setting useswV:wL=1:1w\_\{V\}:w\_\{L\}=1:1, withwV=wL=103w\_\{V\}=w\_\{L\}=10^\{3\}, while power\-flow non\-convergence is treated as a hard failure and assigned a much larger penalty,wpf=106w\_\{\\mathrm\{pf\}\}=10^\{6\}\. For the ablation, the voltage/loading penalties are kept at the same overall scale and only their relative ratio is varied, namelywV:wL∈\{0\.75:1,1:1,2:1\}w\_\{V\}:w\_\{L\}\\in\\\{0\.75:1,1:1,2:1\\\}\.

TABLE I:Sensitivity to the relative voltage/loading penalty weights\.Table[I](https://arxiv.org/html/2609.03308#S4.T1)shows the expected tradeoff: largerwVw\_\{V\}reduces voltage violations, whereas relatively larger loading penalties reduce thermal violations and yield a slightly higher reward\. Across all ratios, both violation terms remain small, indicating that the safety behavior of the proposed method is not tied to a narrowly tuned penalty ratio\.

#### IV\-A2OOD Scenario Construction

The OOD scenarios include four complementary families\. First, observation\-level OOD cases perturb the agent input while keeping the physical exogenous trajectories unchanged\. This family includes additive Gaussian observation noise and FDIA\-style adversarial sensor manipulation\. For Gaussian observation noise, the deployed policy observeso~t=ot\+ϵt\\tilde\{o\}\_\{t\}=o\_\{t\}\+\\epsilon\_\{t\}, whereϵt∼𝒩⁡\(0,σ2​I\)\\epsilon\_\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\)andσ∈\{0\.5,1\.0\}\\sigma\\in\\\{0\.5,1\.0\\\}\. For FDIA\-style attacks, only measurement\-related state components are perturbed, including bus voltages, device telemetry, branch loading, and grid exchange\. The attacked observation is written aso~t=ot\+m⊙δt\\tilde\{o\}\_\{t\}=o\_\{t\}\+m\\odot\\delta\_\{t\}, wheremmis a binary mask selecting the measurement\-related components and‖δt‖∞≤ε\\\|\\delta\_\{t\}\\\|\_\{\\infty\}\\leq\\varepsilon\. We consider both white\-box and black\-box attack variants\. In the white\-box case, the attacker has access to the learned model and uses projected gradient steps against the critic objective, withδtk\+1=Π‖δ‖∞≤ε​\(δtk\+ηatt​sign⁡\(∇otJatt​\(ot\+m⊙δtk\)\)\)\\delta\_\{t\}^\{k\+1\}=\\Pi\_\{\\\|\\delta\\\|\_\{\\infty\}\\leq\\varepsilon\}\(\\delta\_\{t\}^\{k\}\+\\eta\_\{\\mathrm\{att\}\}\\operatorname\{sign\}\(\\nabla\_\{o\_\{t\}\}J\_\{\\mathrm\{att\}\}\(o\_\{t\}\+m\\odot\\delta\_\{t\}^\{k\}\)\)\), whereJattJ\_\{\\mathrm\{att\}\}is chosen to decrease the critic\-estimated return, increase the critic\-estimated cost, or combine the two objectives\. In the black\-box case, the attacker has no access to the model parameters and only queries the deployed agent\. The scalar query objective is defined asf⁡\(o~t\):=−𝔼τ​\[Zψ​\(o~t,π⁡\(o~t\),τ\)\]f\(\\tilde\{o\}\_\{t\}\):=\-\\mathbb\{E\}\_\{\\tau\}\[Z\_\{\\psi\}\(\\tilde\{o\}\_\{t\},\\pi\(\\tilde\{o\}\_\{t\}\),\\tau\)\], and the attack direction is estimated by the symmetric finite\-difference estimatorg^t=1Kq​∑k=1Kqf⁡\(o~t\+ζ​uk\)−f⁡\(o~t−ζ​uk\)2​ζ​uk\\hat\{g\}\_\{t\}=\\frac\{1\}\{K\_\{q\}\}\\sum\_\{k=1\}^\{K\_\{q\}\}\\frac\{f\(\\tilde\{o\}\_\{t\}\+\\zeta u\_\{k\}\)\-f\(\\tilde\{o\}\_\{t\}\-\\zeta u\_\{k\}\)\}\{2\\zeta\}u\_\{k\}, whereuk∼𝒩⁡\(0,I\)u\_\{k\}\\sim\\mathcal\{N\}\(0,I\),KqK\_\{q\}is the number of query probes, andζ\>0\\zeta\>0is the probing scale\. The black\-box perturbation is then updated byδtk\+1=Π‖δ‖∞≤ε​\(δtk\+ηatt​sign⁡\(m⊙g^t\)\)\\delta\_\{t\}^\{k\+1\}=\\Pi\_\{\\\|\\delta\\\|\_\{\\infty\}\\leq\\varepsilon\}\(\\delta\_\{t\}^\{k\}\+\\eta\_\{\\mathrm\{att\}\}\\operatorname\{sign\}\(m\\odot\\hat\{g\}\_\{t\}\)\)\. The default FDIA budget isε=0\.1\\varepsilon=0\.1, with projected steps of size0\.0250\.025\. Second, synthetic profile\-shift scenarios impose structured deviations on the daily exogenous trajectories, including load surges, PV dropouts, price spikes, low\-renewable high\-load periods, evening net\-load ramps, renewable overgeneration, and cloud\-ramp PV attenuation\. These cases emulate operationally meaningful stress events with randomized affected windows and stress magnitudes\. Third, frequency\-domain OOD cases perturb the spectral components of the load and PV trajectories and then reconstruct the time\-domain profiles\. This changes the temporal shape of the exogenous curves while keeping their daily scale comparable to the original profiles\. Fourth, data\-driven OOD cases are built from hold\-out or external real\-world datasets that are not used during training, including unseen UCI load profiles\[[29](https://arxiv.org/html/2609.03308#bib.bib29)\], London smart\-meter household consumption data\[[31](https://arxiv.org/html/2609.03308#bib.bib31)\], hold\-out Kaggle PV curves\[[30](https://arxiv.org/html/2609.03308#bib.bib30)\], and NSRDB irradiance and temperature records\[[32](https://arxiv.org/html/2609.03308#bib.bib32)\]\. For external load scenarios, the London smart\-meter readings are first aggregated to hourly resolution to match the 24\-step scheduling horizon\. The aggregated profile is converted into a unit\-mean temporal multiplierrtext=Ltext/\(1T​∑τ=1TLτext\)r\_\{t\}^\{\\mathrm\{ext\}\}=L\_\{t\}^\{\\mathrm\{ext\}\}/\(\\frac\{1\}\{T\}\\sum\_\{\\tau=1\}^\{T\}L\_\{\\tau\}^\{\\mathrm\{ext\}\}\), and then applied to the original bus\-level nominal active loads asPi,tload,ext=αd​Piload,0​rtextP\_\{i,t\}^\{\\mathrm\{load,ext\}\}=\\alpha\_\{d\}P\_\{i\}^\{\\mathrm\{load\},0\}r\_\{t\}^\{\\mathrm\{ext\}\}, wherePiload,0P\_\{i\}^\{\\mathrm\{load\},0\}is the nominal active load at busii, andαd\\alpha\_\{d\}rescales the daily energy to the same operating range as the in\-distribution cases\. The reactive load is reconstructed using the original bus power factor,Qi,tload,ext=Pi,tload,ext​tan⁡\(arccos⁡\(pfi0\)\)Q\_\{i,t\}^\{\\mathrm\{load,ext\}\}=P\_\{i,t\}^\{\\mathrm\{load,ext\}\}\\tan\(\\arccos\(\\mathrm\{pf\}\_\{i\}^\{0\}\)\)\. Therefore, the feeder\-specific spatial allocation and power\-factor structure are inherited from the target network, while the external data only alter the temporal demand shape\. For NSRDB\-based PV scenarios, the irradiance and temperature records are used to construct a bounded PV availability factor rather than a detailed inverter model\. Specifically,ρtNSRDB=min⁡\{1,max⁡\{0,\(GHIt/1000\)​\[1\+βpv​\(Tt−25\)\]\}\}\\rho\_\{t\}^\{\\mathrm\{NSRDB\}\}=\\min\\\{1,\\max\\\{0,\(\\mathrm\{GHI\}\_\{t\}/1000\)\[1\+\\beta\_\{\\mathrm\{pv\}\}\(T\_\{t\}\-25\)\]\\\}\\\}, whereGHIt\\mathrm\{GHI\}\_\{t\}is the hourly global horizontal irradiance,TtT\_\{t\}is the ambient temperature, andβpv=−0\.004∘C−1\\beta\_\{\\mathrm\{pv\}\}=\-0\.004~\{\}^\{\\circ\}\\mathrm\{C\}^\{\-1\}\. The PV active power at busbbis thenPb,tPV,ext=Pb,ratedPV​ρtNSRDBP\_\{b,t\}^\{\\mathrm\{PV,ext\}\}=P\_\{b,\\mathrm\{rated\}\}^\{\\mathrm\{PV\}\}\\rho\_\{t\}^\{\\mathrm\{NSRDB\}\}\. This construction preserves the installed PV capacities and the network\-side operational constraints of the target feeder, while introducing real meteorological temporal patterns\. Finally, we include VAE\-generated near\-OOD trajectories\[[33](https://arxiv.org/html/2609.03308#bib.bib33)\], which are decoded from a latent model trained on in\-distribution profiles and therefore represent milder distribution shifts\.

### IV\-BTest Case I: Qualitative Demonstration on the Original System

For Test Case I, the proposed agent usesM=5M=5Bayesian distributional critics,K=4K=4Monte Carlo dropout samples, and dropout rateρ=0\.05\\rho=0\.05\. Training usesγ=0\.95\\gamma=0\.95, learning rate10−410^\{\-4\}, buffer size20002000, warm\-up length10001000,αexpl=0\.2\\alpha\_\{\\mathrm\{expl\}\}=0\.2, and Lagrange multiplier stepsize10−310^\{\-3\}\.

#### IV\-B1AU Evaluation

Fig\.[1](https://arxiv.org/html/2609.03308#S4.F1)shows the AU estimates in Test Case I under different environmental noise scales\. The estimated AU generally increases as the noise scale grows\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/au.png)Fig\. 1:Estimated AU under different environmental noise scales in Test Case I\. Solid curves show the mean and shaded areas denote one standard deviation\. This confirms that the proposed AU estimate is sensitive to irreducible environmental stochasticity\.
#### IV\-B2EU Evaluation

Fig\.[2](https://arxiv.org/html/2609.03308#S4.F2)reports the estimated EU in Test Case I under different OOD settings\. Compared with the InD baseline, all OOD cases produce clearly higher EU values\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/eu.png)Fig\. 2:Estimated EU under different OOD scenarios in Test Case I\. The InD curve is shown as the reference baseline\. Solid curves show the mean and shaded areas denote one standard deviation\. The increase is mild for near\-OOD VAE samples and frequency perturbations, but becomes much stronger under larger observation noise, real\-world LCL load data, NSRDB solar data, and their combined shift\. This shows that the proposed EU estimate can distinguish unfamiliar operating conditions from normal InD operation\.Fig\.[3](https://arxiv.org/html/2609.03308#S4.F3)reports the EU response to FDIA\-style sensor manipulation in Test Case I\. As the attack steps increase, EU generally rises across the 24\-hour horizon, especially under the white\-box setting\. Black\-box attacks also produce increasing EU in several timesteps, although with a different temporal pattern\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/eu_grid.png)Fig\. 3:EU response to FDIA\-style sensor attacks in Test Case I\. This shows that critic\-side EU can serve as a sensitivity indicator for adversarial observation shifts\. Solid curves denote white\-box attacks, dashed curves denote black\-box attacks, and shaded areas denote one standard deviation\.

### IV\-CTest Case II: Quantitative Benchmark Study

#### IV\-C1AU Evaluation

To evaluate AU estimation in Test Case II, we use the same set of trained critic parameters and change only the AU estimation method\. Specifically, the evaluation uses five independently trained models, each with six ensemble members\. The proposed AU estimator is compared with an IQN\-style member estimator, where each ensemble member is treated as an independent IQN critic and its return variance is used as AU\. As shown in Fig\.[4](https://arxiv.org/html/2609.03308#S4.F4), both estimators increase monotonically with the environmental noise scale\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/au2.png)Fig\. 4:AU evaluation under different environmental noise scales\. Both methods successfully track the increase of intrinsic stochasticity\.This confirms that the proposed AU can track the increase of irreducible environmental stochasticity\. Fig\.[5](https://arxiv.org/html/2609.03308#S4.F5)further evaluates the estimator variance of AU, rather than the AU magnitude itself\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/au3.png)Fig\. 5:Estimator variance of AU under different environmental noise scales\. The variance is computed over episode\-level AU estimates under the same noise scale\. The proposed AU estimator shows lower estimator variance than the IQN\-style baseline\.For each environmental noise scaleσau\\sigma\_\{\\mathrm\{au\}\}, we first compute one episode\-level AU estimate for each rollout, denoted byAU^e​\(σau\)\\widehat\{\\mathrm\{AU\}\}\_\{e\}\(\\sigma\_\{\\mathrm\{au\}\}\)\. We then reportVare:σe=σau\[AU^e\(σau\)\]\\operatorname\{Var\}\_\{e:\\sigma\_\{e\}=\\sigma\_\{\\mathrm\{au\}\}\}\[\\widehat\{\\mathrm\{AU\}\}\_\{e\}\(\\sigma\_\{\\mathrm\{au\}\}\)\], where the variance is taken over all episode\-level AU estimates collected under the same environmental noise scale\. Therefore, a smaller value in Fig\.[5](https://arxiv.org/html/2609.03308#S4.F5)indicates lower fluctuation of the AU estimator across repeated rollouts, not smaller AU of the environment\. The proposed AU estimator consistently shows a lower estimator variance than the IQN\-style baseline\. The advantage comes from coupling ensemble prediction with the second\-order AU definition\. The ensemble members are used as Monte Carlo samples of the second\-order predictive distribution\. The proposed AU then suppresses member\-specific fluctuation while retaining sensitivity to environmental stochasticity\. In contrast, the IQN\-style baseline directly uses each member’s raw return variance as an AU estimate, making it more sensitive to calibration differences, quantile\-spread fluctuations, and training randomness\.

#### IV\-C2EU Evaluation via Training Performance

We first evaluate whether the proposed EU estimate is useful during training\. Fig\.[6](https://arxiv.org/html/2609.03308#S4.F6)reports the evaluation reward, raw constraint cost, and log\-scale constraint cost in Test Case II\. The algorithm comparison includes TD3\[[34](https://arxiv.org/html/2609.03308#bib.bib34)\], PPO\[[35](https://arxiv.org/html/2609.03308#bib.bib35)\], PPO\-GAE\[[35](https://arxiv.org/html/2609.03308#bib.bib35),[36](https://arxiv.org/html/2609.03308#bib.bib36)\], CPO\-BDQN\[[37](https://arxiv.org/html/2609.03308#bib.bib37),[38](https://arxiv.org/html/2609.03308#bib.bib38)\], Thompson\-sampling DQN\[[39](https://arxiv.org/html/2609.03308#bib.bib39)\], and IQN\[[22](https://arxiv.org/html/2609.03308#bib.bib22)\]\. Among these learning\-based baselines, the proposed method achieves the best final reward and the lowest long\-term constraint cost\. This indicates that the performance gain is not only from distributional value learning, but also from using critic\-side EU as an exploration signal\. For the hyperparameter ablations, we adopt a two\-stage protocol\. First, a full grid search is conducted in a reduced action space\. This stage is used to select the default settingαexpl=0\.3\\alpha\_\{\\mathrm\{expl\}\}=0\.3, ensemble sizeM=6M=6, and dropout rateρ=0\.08\\rho=0\.08\. After fixing this setting, we return to the full action space and conduct single\-factor sensitivity tests by varying one hyperparameter at a time\. The exploration\-bonus ablation shows that too smallαexpl\\alpha\_\{\\mathrm\{expl\}\}gives insufficient epistemic exploration, whereas too largeαexpl\\alpha\_\{\\mathrm\{expl\}\}overemphasizes uncertain actions and slows exploitation\. The intermediate valueαexpl=0\.3\\alpha\_\{\\mathrm\{expl\}\}=0\.3gives the best balance between reward improvement and cost reduction\. The ensemble\-size and dropout\-rate ablations show similar robustness\. Moderate changes aroundM=6M=6andρ=0\.08\\rho=0\.08lead to comparable convergence trends, while the selected setting gives a favorable tradeoff between final reward, learning stability, and constraint\-cost suppression\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/EU2_1.png)Fig\. 6:EU evaluation via training performance in Test Case II\. Each row reports the evaluation reward, raw constraint cost, and log\-scale constraint cost\. Row \(a\) compares the proposed method with representative baselines\. Rows \(b\)–\(d\) report full action\-space sensitivity tests on the exploration bonus coefficient, ensemble size, and dropout rate, respectively\. The offline MPC reference in Test Case II remains fully feasible, with zero constraint cost across the evaluated episodes, and yields an average reward of−16\.53\-16\.53with low variance\(3\.03×10−2\)\(3\.03\\times 10^\{\-2\}\)\.To examine EU exploration itself, we map each visited state during training to a coarse bin\. The number of unique bins measures the coverage of distinct reachable operating regions\. A bin is counted as useful only when the associated transition is feasible, with converged AC power flow and without voltage or loading violation\. The useful ratio is then the fraction of useful unique bins among all unique bins\. As shown in Fig\.[7](https://arxiv.org/html/2609.03308#S4.F7), the proposed UQ exploration achieves both the largest number of unique bins and the highest useful ratio\. Compared with fixed Gaussian exploration noise, the proposed strategy produces broader coverage with a higher useful\-bin ratio\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/EU2_2.png)Fig\. 7:State\-space coverage and useful\-region ratio under different exploration strategies in Test Case II\. Panel \(a\) reports the cumulative number of unique bins visited during training\. Panel \(b\) reports the ratio of useful unique bins to all unique bins\. Solid curves show the mean over repeated runs, and shaded areas denote one standard deviation\. The proposed UQ exploration expands the explored operating region while maintaining a higher proportion of feasible and useful states\.
#### IV\-C3EU Evaluation via OOD Detection and Degradation Correlation

We next evaluate whether the proposed EU can detect OOD states and whether its magnitude is correlated with the resulting operational degradation\. Table[II](https://arxiv.org/html/2609.03308#S4.T2)compares the proposed EU score with classical OOD detection baselines\. For the proposed method, the episode\-level EU is directly used as the detection score\. For the baselines, we train conventional feature\-space detectors using actor features extracted from the raw observation\. The compared baselines include KDE, KNN \(k=5k=5\), GMM \(44components\), and Mahalanobis\-distance detectors\[[40](https://arxiv.org/html/2609.03308#bib.bib40)\]\. We report both AUROC and FPR95, where larger AUROC and smaller FPR95 indicate better OOD separation\. As the ensemble size decreases, the detection performance degrades mildly but remains competitive with the feature\-space baselines\.

TABLE II:OOD detection performance in Test Case II\.We further examine whether the EU score is aligned with the severity of operational degradation\. We compare EU with the episode\-level reward gapΔ​R\\Delta Rand constraint\-cost gapΔ​C\\Delta Cdefined in Section[III\-F](https://arxiv.org/html/2609.03308#S3.SS6)\. As shown in Fig\.[8](https://arxiv.org/html/2609.03308#S4.F8), each point corresponds to one held\-out episode\. Larger EU is generally associated with largerΔ​R\\Delta RandΔ​C\\Delta C\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/EU2_3.png)Fig\. 8:EU evaluation via degradation correlation\. Each point corresponds to one held\-out episode\. The horizontal axis is episode\-level EU\. The vertical axes report the episode\-level reward gapΔ​R\\Delta Rand constraint\-cost gapΔ​C\\Delta C\. Larger EU is associated with larger operational degradation, which supports its use as an epistemic reliability score for quantifying harmful distribution shifts\.Fig\.[9](https://arxiv.org/html/2609.03308#S4.F9)further examines how the EU score changes as degradation severity increases\. Since raw EU magnitudes are model\-dependent, we convert EU values into within\-model relative percentiles before pooling the selected models\. The resulting distributions show a clear upward shift with both reward degradation and cost degradation, indicating that larger OOD performance deterioration is associated with higher EU\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/EU2_5.png)Fig\. 9:Relative EU distributions across degradation severity bins\. Raw EU values are normalized to within\-model percentiles before pooling selected models\. Reward and cost degradation are binned into five data\-driven severity levels\. Violin plots show point\-level distributions; black markers show medians and interquartile ranges; red markers show model means with standard errors\.Table[III](https://arxiv.org/html/2609.03308#S4.T3)evaluates whether each score ranks harmful OOD episodes\. The actor\-feature baselines are unsupervised InD\-fitted novelty scores: GMM/KDE use feature\-density NLL, Mahal\. uses Mahalanobis distance, and kNN uses nearest\-neighbor distance\. Like our EU score, these baselines are computed without degradation labels\. For fairness, all scores are computed per trained model and converted to within\-model quantile ranks before reporting Spearman correlations withΔ​R,Δ​C\\Delta R,\\Delta CandP​@​20\\mathrm\{P@20\}for the top\-20% degraded episodes\.

TABLE III:Degradation\-ranking comparison\.
#### IV\-C4Closed\-Loop Evaluation With EU\-Triggered Fallback

Finally, we evaluate the deployment rule in \([37](https://arxiv.org/html/2609.03308#S3.E37)\)\. We instantiate𝒟calcrit\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\\mathrm\{crit\}\}using critical calibration states selected from the synthetic profile\-shift scenarios described above, including their combinations, which perturb daily load, PV, price, and net\-load trajectories and are disjoint from the held\-out deployment episodes\. A state is included in this calibration set when itsΔ​R\\Delta RandΔ​C\\Delta Cgap exceeds the acceptable\-state tolerance by a factor of three\. For each deployed model, the fallback thresholdτfb\\tau\_\{\\mathrm\{fb\}\}is calibrated from the lowerϵmiss\\epsilon\_\{\\mathrm\{miss\}\}\-quantile of the critical\-state EU scores in this calibration set, following \([36](https://arxiv.org/html/2609.03308#S3.E36)\)\. The calibration is performed per model because raw EU scales are not comparable across independently trained ensembles\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/EU2_4.png)Fig\. 10:Closed\-loop fallback ablation overϵmiss\\epsilon\_\{\\mathrm\{miss\}\}\. Panel \(a\) reports fallback invocation; panels \(b\) and \(c\) report the critical\-stateΔ​C\\Delta CandΔ​R\\Delta Rgaps removed by fallback\. Colors indicate ensemble size; black denotes the mean\.Fig\.[10](https://arxiv.org/html/2609.03308#S4.F10)summarizes the fallback\-rate/gap\-removal tradeoff asϵmiss\\epsilon\_\{\\mathrm\{miss\}\}varies\. Atϵmiss=0\.10\\epsilon\_\{\\mathrm\{miss\}\}=0\.10, fallback is invoked for75\.8%75\.8\\%of held\-out episodes and removes94\.0%94\.0\\%of the critical constraint\-cost gap and90\.7%90\.7\\%of the reward gap\. Atϵmiss=0\.20\\epsilon\_\{\\mathrm\{miss\}\}=0\.20, fallback use drops to61\.5%61\.5\\%while constraint\-cost gap removal remains88\.0%88\.0\\%, showing that EU\-triggered fallback can reduce closed\-loop cost with targeted MISOCP handoffs\. To avoid relying solely on a fixed severe\-OOD calibration threshold, we also evaluate a threshold\-free safety–intervention tradeoff in Fig\.[11](https://arxiv.org/html/2609.03308#S4.F11)\. OOD episodes are ranked by EU, and the fallback rate is swept by assigning the highest\-EU fraction to the MISOCP controller\. Increasing fallback coverage consistently reduces both cost exceedance probability and the 95th\-percentile constraint cost, indicating that EU prioritizes episodes with larger empirical safety risk\.

![Refer to caption](https://arxiv.org/html/2609.03308v1/fallback_rate_safety_tradeoff.png)Fig\. 11:Empirical safety–intervention tradeoff of EU\-triggered fallback\. Panels \(a\) and \(b\) use a threshold\-free evaluation: OOD episodes are ranked by EU, and the fallback rate is swept by assigning the highest\-EU fraction of episodes to the MISOCP controller while leaving the remaining episodes controlled by the RL policy\. Panel \(a\) reports the probability that the resulting episode constraint cost exceeds reference quantiles of the RL\-only cost distribution\. The numbers in parentheses are the corresponding RL\-only constraint\-cost quantile values; for example,\>Q​50​\(1887\)\>Q50\(1887\)means that the deployed controller has episode constraint cost larger than the RL\-only median cost of 1887\. Thus, lower curves indicate that EU\-guided fallback preferentially removes episodes from the high\-cost tail\. Panel \(b\) reports the retained 95th\-percentile constraint cost, normalized by the RL\-only 95th\-percentile cost\. Panel \(c\) reports theϵmiss\\epsilon\_\{\\mathrm\{miss\}\}\-based calibration ablation\. Colored thin curves denote individual selected models, and black curves denote the model mean\.

## VConclusion

We proposed an uncertainty\-aware DRL framework for safe distribution\-network operation by coupling distributional RL with UQ\. The resulting second\-order UQ decomposes predictive uncertainty into aleatoric and epistemic OOD uncertainty, enabling EU\-triggered fallback control with improved constraint compliance in diverse operating conditions\.

## References

- \[1\]X\. Chen, G\. Qu, Y\. Tang, S\. Low, and N\. Li, “Reinforcement learning for selective key applications in power systems: Recent advances and future challenges,”IEEE Trans\. Smart Grid, vol\. 13, no\. 4, pp\. 2935–2958, July 2022\.
- \[2\]K\. Lappalainen and S\. Valkealahti, “Analysis of shading periods caused by moving clouds,”Solar Energy, vol\. 135, pp\. 188–196, 2016\.
- \[3\]K\. Zhou, Z\. Liu, Y\. Qiao, T\. Xiang, and C\. C\. Loy, “Domain generalization: A survey,”IEEE Trans\. Pattern Anal\. Mach\. Intell\., vol\. 45, no\. 4, pp\. 4396–4415, Apr\. 2023\.
- \[4\]D\. L\. Donoho, “High\-dimensional data analysis: The curses and blessings of dimensionality,”AMS Math\. Challenges Lecture, 2000\.
- \[5\]V\. Chandola, A\. Banerjee, and V\. Kumar, “Anomaly detection: A survey,”ACM Comput\. Surv\., vol\. 41, no\. 3, pp\. 1–58, 2009\.
- \[6\]J\. Zhang and C\. Li, “Adversarial examples: Opportunities and challenges,”IEEE Trans\. Neural Netw\. Learn\. Syst\., vol\. 31, no\. 7, pp\. 2578–2593, July 2020\.
- \[7\]J\. Tian, B\. Wang, Z\. Wang, K\. Cao, J\. Li, and M\. Ozay, “Joint adversarial example and false data injection attacks for state estimation in power systems,”IEEE Trans\. Cybern\., vol\. 52, no\. 12, pp\. 13699–13713, Dec\. 2022\.
- \[8\]Z\. Zhou, G\. Liu, W\. Guo, and M\. Zhou, “Adversarial attacks on multiagent deep reinforcement learning models in continuous action space,”IEEE Trans\. Syst\. Man Cybern\. Syst\., vol\. 54, no\. 12, pp\. 7633–7646, Dec\. 2024\.
- \[9\]H\. Zhang, Z\. Chen, C\. Yu, D\. Yue, X\. Xie, and G\. P\. Hancke, “Event\-trigger\-based resilient distributed energy management against FDI and DoS attack of cyber–physical system of smart grid,”IEEE Trans\. Syst\. Man Cybern\. Syst\., vol\. 54, no\. 5, pp\. 3220–3230, May 2024\.
- \[10\]X\. Yang, H\. He, J\. Li, and Y\. Zhang, “Toward optimal risk\-averse configuration for HESS with CGANs\-based PV scenario generation,”IEEE Trans\. Syst\. Man Cybern\. Syst\., vol\. 51, no\. 3, pp\. 1779–1793, Mar\. 2021\.
- \[11\]A\. Mavor\-Parker, K\. Young, C\. Barry, and L\. Griffin, “How to stay curious while avoiding noisy TVs using aleatoric uncertainty estimation,” in*Proc\. 39th Int\. Conf\. Mach\. Learn\. \(ICML\)*,*Proc\. Mach\. Learn\. Res\.*, vol\. 162, pp\. 15220–15240, 2022\.
- \[12\]Y\. Ding, T\. Morstyn, and M\. D\. McCulloch, “Distributionally robust joint chance\-constrained optimization for networked microgrids considering contingencies and renewable uncertainty,”IEEE Trans\. Smart Grid, vol\. 13, no\. 3, pp\. 2467–2478, May 2022\.
- \[13\]M\. Rayati, M\. Bozorg, R\. Cherkaoui, and M\. Carpita, “Distributionally robust chance constrained optimization for providing flexibility in an active distribution network,”IEEE Trans\. Smart Grid, vol\. 13, no\. 4, pp\. 2920–2934, July 2022\.
- \[14\]W\. Wang, N\. Yu, Y\. Gao, and J\. Shi, “Safe off\-policy deep reinforcement learning algorithm for Volt\-VAR control in power distribution systems,”*IEEE Trans\. Smart Grid*, vol\. 11, no\. 4, pp\. 3008–3018, July 2020\.
- \[15\]P\. Wu, C\. Chen, D\. Lai, J\. Zhong, and Z\. Bie, “Real\-time optimal power flow method via safe deep reinforcement learning based on primal\-dual and prior knowledge guidance,”IEEE Trans\. Power Syst\., vol\. 40, no\. 1, pp\. 597–611, Jan\. 2025\.
- \[16\]J\. Xie and W\. Sun, “Distributional deep reinforcement learning\-based emergency frequency control,”IEEE Trans\. Power Syst\., vol\. 37, no\. 4, pp\. 2720–2730, July 2022\.
- \[17\]Z\. Wang, F\. Teng, Y\. Zhou, Q\. Guo, and H\. Sun, “Uncertainty\-aware transient stability\-constrained preventive redispatch: A distributional reinforcement learning approach,”IEEE Trans\. Power Syst\., vol\. 40, no\. 2, pp\. 1295–1308, Mar\. 2025\.
- \[18\]T\. Zhang, M\. Sun, D\. Qiu, X\. Zhang, G\. Strbac, and C\. Kang, “A Bayesian deep reinforcement learning\-based resilient control for multi\-energy micro\-grid,”IEEE Trans\. Power Syst\., vol\. 38, no\. 6, pp\. 5057–5072, Nov\. 2023\.
- \[19\]P\. Pareek and H\. D\. Nguyen, “Gaussian process learning\-based probabilistic optimal power flow,”IEEE Trans\. Power Syst\., vol\. 36, no\. 1, pp\. 541–544, Jan\. 2021\.
- \[20\]X\. Liu, J\. Liu, Y\. Zhao, T\. Ding, X\. Liu, and J\. Liu, “A Bayesian deep learning\-based probabilistic risk assessment and early\-warning model for power systems considering meteorological conditions,”*IEEE Trans\. Ind\. Informat\.*, vol\. 20, no\. 2, pp\. 1516–1527, Feb\. 2024\.
- \[21\]Y\. Yang, W\. Li, T\. A\. Gulliver, and S\. Li, “Bayesian deep learning\-based probabilistic load forecasting in smart grids,”*IEEE Trans\. Ind\. Informat\.*, vol\. 16, no\. 7, pp\. 4703–4713, July 2020\.
- \[22\]W\. Dabney, G\. Ostrovski, D\. Silver, and R\. Munos, “Implicit quantile networks for distributional reinforcement learning,” in*Proc\. 35th Int\. Conf\. Mach\. Learn\. \(ICML\)*,*Proc\. Mach\. Learn\. Res\.*, vol\. 80, pp\. 1096–1105, 2018\.
- \[23\]Y\. Gal and Z\. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in*Proc\. 33rd Int\. Conf\. Mach\. Learn\. \(ICML\)*,*Proc\. Mach\. Learn\. Res\.*, vol\. 48, pp\. 1050–1059, 2016\.
- \[24\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” inAdv\. Neural Inf\. Process\. Syst\., Long Beach, CA, USA, 2017\.
- \[25\]Y\. Sale, V\. Bengs, M\. Caprio, and E\. Hüllermeier, “Second\-order uncertainty quantification: A distance\-based approach,” in*Proc\. 41st Int\. Conf\. Mach\. Learn\. \(ICML\)*,*Proc\. Mach\. Learn\. Res\.*, vol\. 235, pp\. 43060–43076, 2024\.
- \[26\]H\. Hu, S\. Song, and G\. Huang, “Self\-attention\-based temporary curiosity in reinforcement learning exploration,”IEEE Trans\. Syst\. Man Cybern\. Syst\., vol\. 51, no\. 9, pp\. 5773–5784, Sept\. 2021\.
- \[27\]M\. Farivar and S\. H\. Low, “Branch flow model: Relaxations and convexification—Part I,”IEEE Trans\. Power Syst\., vol\. 28, no\. 3, pp\. 2554–2564, Aug\. 2013\.
- \[28\]L\. Thurner, A\. Scheidler, F\. Schäfer, J\. Menke, J\. Dollichon, F\. Meier, S\. Meinecke, and M\. Braun, “pandapower—An Open\-Source Python Tool for Convenient Modeling, Analysis, and Optimization of Electric Power Systems,”IEEE Trans\. Power Syst\., vol\. 33, no\. 6, pp\. 6510–6521, Nov\. 2018\.
- \[29\]A\. Trindade, “Electricity Load Diagrams 2011\-2014”, UCI Machine Learning Repository, 2015\. \[Online\]\. Available:https://doi\.org/10\.24432/C58C86
- \[30\]A\. Kannal, “Solar Power Generation Data,” Kaggle, data set\. \[Online\]\. Available:https://www\.kaggle\.com/datasets/anikannal/solar\-power\-generation\-data
- \[31\]UK Power Networks, “Smart meter energy consumption data in London households,” London Datastore\. \[Online\]\. Available:https://data\.london\.gov\.uk/dataset/smartmeter\-energy\-consumption\-data\-in\-london\-households\-vqm0d
- \[32\]National Renewable Energy Laboratory \(NREL\), “National Solar Radiation Database \(NSRDB\),” data set\. \[Online\]\. Available:https://nsrdb\.nrel\.gov
- \[33\]D\. P\. Kingma and M\. Welling, “Auto\-Encoding Variational Bayes,” inProc\. Int\. Conf\. Learn\. Representations \(ICLR\), Banff, AB, Canada, Apr\. 2014\.
- \[34\]S\. Fujimoto, H\. van Hoof, and D\. Meger, “Addressing function approximation error in actor\-critic methods,” inProc\. 35th Int\. Conf\. Mach\. Learn\. \(ICML\), vol\. 80, Jul\. 2018, pp\. 1587–1596\.
- \[35\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov, “Proximal policy optimization algorithms,”arXivpreprint arXiv:1707\.06347, 2017\.
- \[36\]J\. Schulman, P\. Moritz, S\. Levine, M\. I\. Jordan, and P\. Abbeel, “High\-dimensional continuous control using generalized advantage estimation,” in*Proc\. 4th Int\. Conf\. Learn\. Representations \(ICLR\)*, 2016\.
- \[37\]J\. Achiam, D\. Held, A\. Tamar, and P\. Abbeel, “Constrained policy optimization,” in*Proc\. 34th Int\. Conf\. Mach\. Learn\. \(ICML\)*, ser\.*Proc\. Mach\. Learn\. Res\.*, vol\. 70, 2017, pp\. 22–31\.
- \[38\]I\. Osband, C\. Blundell, A\. Pritzel, and B\. Van Roy, “Deep exploration via bootstrapped DQN,” in*Adv\. Neural Inf\. Process\. Syst\.*, vol\. 29, 2016, pp\. 4026–4034\.
- \[39\]I\. Osband, B\. Van Roy, D\. J\. Russo, and Z\. Wen, “Deep exploration via randomized value functions,”*J\. Mach\. Learn\. Res\.*, vol\. 20, no\. 124, pp\. 1–62, 2019\.
- \[40\]L\. Ruff, J\. R\. Kauffmann, R\. A\. Vandermeulen, G\. Montavon, W\. Samek, M\. Kloft, T\. G\. Dietterich, and K\.\-R\. Müller, “A unifying review of deep and shallow anomaly detection,”*Proc\. IEEE*, vol\. 109, no\. 5, pp\. 756–795, May 2021\.
- \[41\]G\. Peyré and M\. Cuturi, “Computational optimal transport: With applications to data science,”Found\. Trends Mach\. Learn\., vol\. 11, nos\. 5–6, pp\. 355–607, 2019\.
- \[42\]V\. M\. Panaretos and Y\. Zemel,*An Invitation to Statistics in Wasserstein Space*\. Cham, Switzerland: Springer, 2020\.
- \[43\]M\. Agueh and G\. Carlier, “Barycenters in the Wasserstein space,”*SIAM J\. Math\. Anal\.*, vol\. 43, no\. 2, pp\. 904–924, 2011\.
- \[44\]T\. Le Gouic and J\.\-M\. Loubes, “Existence and consistency of Wasserstein barycenters,”*Probab\. Theory Relat\. Fields*, vol\. 168, no\. 3–4, pp\. 901–917, Aug\. 2017\.

## Appendix AProofs for Second\-order UQ Constructions

We adopt the same setting and notation as in Section[III\-D2](https://arxiv.org/html/2609.03308#S3.SS4.SSS2)\. In particular, the return space is one\-dimensional\. On𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)we use the first\-order 2\-Wasserstein distanceW2\(1\)W\_\{2\}^\{\(1\)\}\(induced by the squared Euclidean cost onYY\), and on𝒪⁡\(Y\)=𝒫⁡\(𝒫⁡\(Y\)\)\\mathcal\{O\}\(Y\)=\\mathcal\{P\}\(\\mathcal\{P\}\(Y\)\)we use the induced second\-order 2\-Wasserstein distanceW2\(2\)W\_\{2\}^\{\(2\)\}with ground metricd1=W2\(1\)d\_\{1\}=W\_\{2\}^\{\(1\)\}\.

### A\-AAuxiliary Results on Wasserstein Distance in One Dimension

We first record several standard lemmas on Wasserstein distances and barycenters in one dimension\[[41](https://arxiv.org/html/2609.03308#bib.bib41),[42](https://arxiv.org/html/2609.03308#bib.bib42)\]\.

###### Lemma 1\(Dirac second\-order measures and couplings\)

For anyQ∈𝒪⁡\(Y\)Q\\in\\mathcal\{O\}\(Y\)andp∈𝒫⁡\(Y\)p\\in\\mathcal\{P\}\(Y\)such that𝔼P∼Q​\[\(W2\(1\)​\(P,p\)\)2\]<∞\\mathbb\{E\}\_\{P\\sim Q\}\\bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,p\)\\bigr\)^\{2\}\\bigr\]<\\infty,\(W2\(2\)​\(Q,δp\)\)2=𝔼P∼Q​\[\(W2\(1\)​\(P,p\)\)2\]\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{p\}\)\\bigr\)^\{2\}=\\mathbb\{E\}\_\{P\\sim Q\}\\bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,p\)\\bigr\)^\{2\}\\bigr\], whereδp\\delta\_\{p\}denotes the Dirac mass atppin𝒪⁡\(Y\)\\mathcal\{O\}\(Y\)\. This follows from\[[25](https://arxiv.org/html/2609.03308#bib.bib25)\]\.

###### Lemma 2\(Quantile representation ofW2\(1\)W\_\{2\}^\{\(1\)\}in one dimension\[[41](https://arxiv.org/html/2609.03308#bib.bib41)\]\)

LetY⊂ℝY\\subset\\mathbb\{R\}, and letp,p~∈𝒫⁡\(Y\)p,\\tilde\{p\}\\in\\mathcal\{P\}\(Y\)with quantile functionsQp,Qp~:\(0,1\)→ℝQ\_\{p\},Q\_\{\\tilde\{p\}\}:\(0,1\)\\to\\mathbb\{R\}\. Then\(W2\(1\)​\(p,p~\)\)2=∫01\[Qp​\(τ\)−Qp~​\(τ\)\]2​𝑑τ\\bigl\(W\_\{2\}^\{\(1\)\}\(p,\\tilde\{p\}\)\\bigr\)^\{2\}=\\int\_\{0\}^\{1\}\\bigl\[Q\_\{p\}\(\\tau\)\-Q\_\{\\tilde\{p\}\}\(\\tau\)\\bigr\]^\{2\}\\,\\mathrm\{d\}\\tau\.

###### Lemma 3\(One\-dimensional Wasserstein barycenter\[[43](https://arxiv.org/html/2609.03308#bib.bib43),[41](https://arxiv.org/html/2609.03308#bib.bib41)\]\)

Letp1,…,pR∈𝒫⁡\(Y\)p\_\{1\},\\dots,p\_\{R\}\\in\\mathcal\{P\}\(Y\)with quantile functionsQprQ\_\{p\_\{r\}\}\. Considerp⋆∈arg​minp∈𝒫⁡\(Y\)⁡1R​∑r=1R\(W2\(1\)​\(pr,p\)\)2p^\{\\star\}\\in\\argmin\_\{p\\in\\mathcal\{P\}\(Y\)\}\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\bigl\(W\_\{2\}^\{\(1\)\}\(p\_\{r\},p\)\\bigr\)^\{2\}\. Then a minimiser exists and any minimiser has quantile functionQp⋆​\(τ\)=1R​∑r=1RQpr​\(τ\)Q\_\{p^\{\\star\}\}\(\\tau\)=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}Q\_\{p\_\{r\}\}\(\\tau\)forτ∈\(0,1\)\\tau\\in\(0,1\)\.

###### Lemma 4\(Projection onto Dirac measures and variance\[[42](https://arxiv.org/html/2609.03308#bib.bib42)\]\)

Letp∈𝒫⁡\(Y\)p\\in\\mathcal\{P\}\(Y\)have finite second moment and meanμ⁡\(p\)\\mu\(p\)\. Theninfy∈Y\(W2\(1\)​\(p,δy\)\)2=\(W2\(1\)​\(p,δμ⁡\(p\)\)\)2=VarY∼p​\(Y\)\\inf\_\{y\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(p,\\delta\_\{y\}\)\\bigr\)^\{2\}=\\bigl\(W\_\{2\}^\{\(1\)\}\(p,\\delta\_\{\\mu\(p\)\}\)\\bigr\)^\{2\}=\\mathrm\{Var\}\_\{Y\\sim p\}\(Y\), whereδy\\delta\_\{y\}is the Dirac mass atyy\.

### A\-BReturn\-based Critic

We now derive the critic indices in Section[III\-D2](https://arxiv.org/html/2609.03308#S3.SS4.SSS2)from the general distance\-based definitions \([25](https://arxiv.org/html/2609.03308#S3.E25)\)–\([27](https://arxiv.org/html/2609.03308#S3.E27)\), using the one\-dimensional results in Section[A](https://arxiv.org/html/2609.03308#A1)\.

#### A\-B1Epistemic component as Wasserstein Fréchet variance

We start from the epistemic indexUep​\(Q\)U\_\{\\mathrm\{ep\}\}\(Q\)in \([27](https://arxiv.org/html/2609.03308#S3.E27)\) with the choicesd1=W2\(1\)d\_\{1\}=W\_\{2\}^\{\(1\)\}on𝒫⁡\(Y\)\\mathcal\{P\}\(Y\)and the induced second\-order metricW2\(2\)W\_\{2\}^\{\(2\)\}on𝒪⁡\(Y\)\\mathcal\{O\}\(Y\)\. For anyQ∈𝒪⁡\(Y\)Q\\in\\mathcal\{O\}\(Y\),Uep​\(Q\)=infδp∈𝒮epW2\(2\)​\(Q,δp\)=infp∈𝒫⁡\(Y\)W2\(2\)​\(Q,δp\)U\_\{\\mathrm\{ep\}\}\(Q\)=\\inf\_\{\\delta\_\{p\}\\in\\mathcal\{S\}\_\{\\mathrm\{ep\}\}\}W\_\{2\}^\{\(2\)\}\\bigl\(Q,\\delta\_\{p\}\\bigr\)=\\inf\_\{p\\in\\mathcal\{P\}\(Y\)\}W\_\{2\}^\{\(2\)\}\\bigl\(Q,\\delta\_\{p\}\\bigr\)\. By Lemma[1](https://arxiv.org/html/2609.03308#Thmlemma1),\(W2\(2\)​\(Q,δp\)\)2=𝔼P∼Q​\[\(W2\(1\)​\(P,p\)\)2\]\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{p\}\)\\bigr\)^\{2\}=\\mathbb\{E\}\_\{P\\sim Q\}\\bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,p\)\\bigr\)^\{2\}\\bigr\]\. Thus, up to a square root, the epistemic index is given by the Fréchet functional on\(𝒫⁡\(Y\),W2\(1\)\)\(\\mathcal\{P\}\(Y\),W\_\{2\}^\{\(1\)\}\):Uep2​\(Q\):=infp∈𝒫⁡\(Y\)𝔼P∼Q​\[\(W2\(1\)​\(P,p\)\)2\]U\_\{\\mathrm\{ep\}\}^\{2\}\(Q\):=\\inf\_\{p\\in\\mathcal\{P\}\(Y\)\}\\mathbb\{E\}\_\{P\\sim Q\}\\bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,p\)\\bigr\)^\{2\}\\bigr\]\. We useUep2​\(Q\)U\_\{\\mathrm\{ep\}\}^\{2\}\(Q\)as the basic epistemic quantity; it is a monotone function ofUep​\(Q\)U\_\{\\mathrm\{ep\}\}\(Q\)and preserves the ordering of epistemic uncertainty levels\. SpecializingUep2​\(Q\)U\_\{\\mathrm\{ep\}\}^\{2\}\(Q\)to the empirical lawQ^B​\(s,a\)\\widehat\{Q\}\_\{B\}\(s,a\)in \([29](https://arxiv.org/html/2609.03308#S3.E29)\) implies that any minimizer is aW2\(1\)W\_\{2\}^\{\(1\)\}\-barycenter of\{Pb\}b=1B\\\{P\_\{b\}\\\}\_\{b=1\}^\{B\}, which in one dimension admits the closed\-form quantile averaging in Lemma[3](https://arxiv.org/html/2609.03308#Thmlemma3); hence the resulting empirical Fréchet variance coincides withEUcrit​\(s,a\)\\mathrm\{EU\}\_\{\\mathrm\{crit\}\}\(s,a\)in \([31](https://arxiv.org/html/2609.03308#S3.E31)\)\.

#### A\-B2Aleatoric component as expected intrinsic variance

Starting from the aleatoric definition \([26](https://arxiv.org/html/2609.03308#S3.E26)\) with the reference family𝒮al\\mathcal\{S\}\_\{\\mathrm\{al\}\}, we quantify the distance ofQQto mixtures of Dirac predictors\. For the derivations below, we work with the squared 2\-Wasserstein distance on𝒪⁡\(Y\)\\mathcal\{O\}\(Y\)\. In the one\-dimensional setting of Section[III\-D2](https://arxiv.org/html/2609.03308#S3.SS4.SSS2), this projection admits an exact characterization through pointwise projection of each first\-order drawP∼QP\\sim Qonto Dirac measures\. The resulting expression reduces the aleatoric index to the expected intrinsic variance of the first\-order distributions, which leads directly to the closed\-form estimator used in \([32](https://arxiv.org/html/2609.03308#S3.E32)\)\.

###### Lemma 5\(Exact expression via projection to Dirac predictors\)

LetQ∈𝒪⁡\(Y\)Q\\in\\mathcal\{O\}\(Y\)be any second\-order distribution\. DefineU~al2​\(Q\)=𝔼P∼Q​\[infy∈Y\(W2\(1\)​\(P,δy\)\)2\]\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)=\\mathbb\{E\}\_\{P\\sim Q\}\\\!\\Bigl\[\\inf\_\{y\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y\}\)\\bigr\)^\{2\}\\Bigr\]\. ThenUal2​\(Q\):=infδm∈𝒮al\(W2\(2\)​\(Q,δm\)\)2=U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\):=\\inf\_\{\\delta\_\{m\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}\}\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{m\}\)\\bigr\)^\{2\}=\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\.

###### Proof:

We prove the two inequalities\.

##### Step 1

We show thatUal2​\(Q\)≤U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\\leq\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. For eachP∈𝒫⁡\(Y\)P\\in\\mathcal\{P\}\(Y\)with finite second moment, takey⋆​\(P\):=μ​\(P\)y^\{\\star\}\(P\):=\\mu\(P\), the mean ofPP\. By Lemma[4](https://arxiv.org/html/2609.03308#Thmlemma4),y⋆​\(P\)y^\{\\star\}\(P\)is an optimizer ofinfy∈Y\(W2\(1\)​\(P,δy\)\)2\\inf\_\{y\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y\}\)\\bigr\)^\{2\}, andinfy∈Y\(W2\(1\)​\(P,δy\)\)2=\(W2\(1\)​\(P,δμ⁡\(P\)\)\)2\\inf\_\{y\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y\}\)\\bigr\)^\{2\}=\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{\\mu\(P\)\}\)\\bigr\)^\{2\}\. Define the pushforward measurem:=\(y⋆\)\#​Q∈𝒫⁡\(Y\)m:=\(y^\{\\star\}\)\_\{\\\#\}Q\\in\\mathcal\{P\}\(Y\)and considerδm∈𝒮al\\delta\_\{m\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}\. DefineT:𝒫⁡\(Y\)→𝒫⁡\(Y\)T:\\mathcal\{P\}\(Y\)\\to\\mathcal\{P\}\(Y\)byT⁡\(P\):=δy⋆​\(P\)=δμ⁡\(P\)T\(P\):=\\delta\_\{y^\{\\star\}\(P\)\}=\\delta\_\{\\mu\(P\)\}, and construct the couplingγ:=\(Id,T\)\#​Q∈Γ⁡\(Q,δm\)\\gamma:=\(\\mathrm\{Id\},T\)\_\{\\\#\}Q\\in\\Gamma\(Q,\\delta\_\{m\}\)\. Then, by the definition of the second\-order distanceW2\(2\)W\_\{2\}^\{\(2\)\}on𝒪⁡\(Y\)\\mathcal\{O\}\(Y\)with ground metricW2\(1\)W\_\{2\}^\{\(1\)\}on𝒫⁡\(Y\)\\mathcal\{P\}\(Y\),\(W2\(2\)​\(Q,δm\)\)2≤∫𝒫⁡\(Y\)×𝒫⁡\(Y\)\(W2\(1\)​\(P,P~\)\)2​𝑑γ​\(P,P~\)=𝔼P∼Q​\[\(W2\(1\)​\(P,T⁡\(P\)\)\)2\]\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{m\}\)\\bigr\)^\{2\}\\leq\\int\_\{\\mathcal\{P\}\(Y\)\\times\\mathcal\{P\}\(Y\)\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\tilde\{P\}\)\\bigr\)^\{2\}\\,\\mathrm\{d\}\\gamma\(P,\\tilde\{P\}\)=\\mathbb\{E\}\_\{P\\sim Q\}\\Bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,T\(P\)\)\\bigr\)^\{2\}\\Bigr\]\. Thus,\(W2\(2\)​\(Q,δm\)\)2≤𝔼P∼Q​\[\(W2\(1\)​\(P,δμ⁡\(P\)\)\)2\]=U~al2​\(Q\)\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{m\}\)\\bigr\)^\{2\}\\leq\\mathbb\{E\}\_\{P\\sim Q\}\\Bigl\[\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{\\mu\(P\)\}\)\\bigr\)^\{2\}\\Bigr\]=\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. Taking the infimum overδm∈𝒮al\\delta\_\{m\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}yieldsUal2​\(Q\)≤U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\\leq\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\.

##### Step 2

We show thatUal2​\(Q\)≥U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\\geq\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. Fix anym∈𝒫⁡\(Y\)m\\in\\mathcal\{P\}\(Y\)and any couplingγ∈Γ⁡\(Q,δm\)\\gamma\\in\\Gamma\(Q,\\delta\_\{m\}\)\. Sinceδm\\delta\_\{m\}is supported on Dirac predictorsδy\\delta\_\{y\}, we haveP~=δy\\tilde\{P\}=\\delta\_\{y\}γ\\gamma\-almost surely for somey∈Yy\\in Y\. Hence, forγ\\gamma\-almost every\(P,P~\)\(P,\\tilde\{P\}\),\(W2\(1\)​\(P,P~\)\)2=\(W2\(1\)​\(P,δy\)\)2≥infy′∈Y\(W2\(1\)​\(P,δy′\)\)2\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\tilde\{P\}\)\\bigr\)^\{2\}=\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y\}\)\\bigr\)^\{2\}\\geq\\inf\_\{y^\{\\prime\}\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y^\{\\prime\}\}\)\\bigr\)^\{2\}\. Integrating both sides with respect toγ\\gammaand using that the first marginal ofγ\\gammaisQQ, we obtain∫𝒫⁡\(Y\)×𝒫⁡\(Y\)\(W2\(1\)​\(P,P~\)\)2​𝑑γ​\(P,P~\)≥𝔼P∼Q​\[infy′∈Y\(W2\(1\)​\(P,δy′\)\)2\]=U~al2​\(Q\)\\int\_\{\\mathcal\{P\}\(Y\)\\times\\mathcal\{P\}\(Y\)\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\tilde\{P\}\)\\bigr\)^\{2\}\\,\\mathrm\{d\}\\gamma\(P,\\tilde\{P\}\)\\geq\\mathbb\{E\}\_\{P\\sim Q\}\\Bigl\[\\inf\_\{y^\{\\prime\}\\in Y\}\\bigl\(W\_\{2\}^\{\(1\)\}\(P,\\delta\_\{y^\{\\prime\}\}\)\\bigr\)^\{2\}\\Bigr\]=\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. Taking the infimum over all couplingsγ∈Γ⁡\(Q,δm\)\\gamma\\in\\Gamma\(Q,\\delta\_\{m\}\)yields\(W2\(2\)​\(Q,δm\)\)2≥U~al2​\(Q\)\\bigl\(W\_\{2\}^\{\(2\)\}\(Q,\\delta\_\{m\}\)\\bigr\)^\{2\}\\geq\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. Finally, taking the infimum overδm∈𝒮al\\delta\_\{m\}\\in\\mathcal\{S\}\_\{\\mathrm\{al\}\}givesUal2​\(Q\)≥U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\\geq\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\.

Combining Step 1 and Step 2 concludes thatUal2​\(Q\)=U~al2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)=\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)\. ∎

Combining Lemma[5](https://arxiv.org/html/2609.03308#Thmlemma5)with Lemma[4](https://arxiv.org/html/2609.03308#Thmlemma4)yields an explicit expression forU~al2​\(Q\)\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)in terms of intrinsic variances\.

###### Corollary 1\(Expected intrinsic variance\)

LetQ∈𝒪⁡\(Y\)Q\\in\\mathcal\{O\}\(Y\)and letP∼QP\\sim Qhave finite second moments almost surely\. ThenU~al2​\(Q\)=𝔼P∼Q​\[VarY∼P​\(Y\)\]\.\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)=\\mathbb\{E\}\_\{P\\sim Q\}\\Bigl\[\\mathrm\{Var\}\_\{Y\\sim P\}\(Y\)\\Bigr\]\.

We takeU~al2​\(Q\)\\widetilde\{U\}\_\{\\mathrm\{al\}\}^\{2\}\(Q\)as the basic aleatoric quantity; by Lemma[5](https://arxiv.org/html/2609.03308#Thmlemma5)it equalsUal2​\(Q\)U\_\{\\mathrm\{al\}\}^\{2\}\(Q\)and thus is consistent with the distance\-based interpretation\. Applying Corollary[1](https://arxiv.org/html/2609.03308#Thmcorollary1)toQ^B​\(s,a\)\\widehat\{Q\}\_\{B\}\(s,a\)in \([29](https://arxiv.org/html/2609.03308#S3.E29)\) givesUal2​\(Q^B​\(s,a\)\)=1B​∑b=1BVarY∼Pb​\(Y\),U\_\{\\mathrm\{al\}\}^\{2\}\\\!\\bigl\(\\widehat\{Q\}\_\{B\}\(s,a\)\\bigr\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathrm\{Var\}\_\{Y\\sim P\_\{b\}\}\(Y\),which is exactlyAUcrit​\(s,a\)\\mathrm\{AU\}\_\{\\mathrm\{crit\}\}\(s,a\)in \([32](https://arxiv.org/html/2609.03308#S3.E32)\)\.

## Appendix BConsistency of Second\-order UQ Estimators

In implementation, each drawPbP\_\{b\}is obtained by independently sampling an ensemble member and a dropout mask \(and the IQN quantile fractions\), which yields i\.i\.d\. draws fromQcrit​\(s,a\)Q\_\{\\mathrm\{crit\}\}\(s,a\)\. The corresponding empirical second\-order measure isQ^B​\(s,a\)\\widehat\{Q\}\_\{B\}\(s,a\)in \([29](https://arxiv.org/html/2609.03308#S3.E29)\)\.

###### Theorem 1\(Consistency of Monte\-Carlo EU/AU estimators\)

Define the population indices byEUcrit⋆​\(s,a\):=𝔼⁡\[\(W2\(1\)​\(P,p⋆\)\)2\],\\mathrm\{EU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\):=\\mathbb\{E\}\\\!\\left\[\(W\_\{2\}^\{\(1\)\}\(P,p^\{\\star\}\)\)^\{2\}\\right\],AUcrit⋆​\(s,a\):=𝔼⁡\[VarY∼P​\(Y\)\]\.\\mathrm\{AU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\):=\\mathbb\{E\}\\\!\\left\[\\mathrm\{Var\}\_\{Y\\sim P\}\(Y\)\\right\]\.Then, asB→∞B\\to\\infty,EU^crit,B​\(s,a\)\\widehat\{\\mathrm\{EU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)→almost​surely\\xrightarrow\{\\mathrm\{almost\\,surely\}\}EUcrit⋆​\(s,a\),\\mathrm\{EU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\),AU^crit,B​\(s,a\)\\widehat\{\\mathrm\{AU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)→almost​surelyAUcrit⋆​\(s,a\)\.\\xrightarrow\{\\mathrm\{almost\\,surely\}\}\\mathrm\{AU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\)\.

###### Proof:

Let\{Pb\}b=1B\\\{P\_\{b\}\\\}\_\{b=1\}^\{B\}be i\.i\.d\. fromQcrit​\(s,a\)Q\_\{\\mathrm\{crit\}\}\(s,a\)and letp^B\\widehat\{p\}\_\{B\}be an empirical barycenter as in \([30](https://arxiv.org/html/2609.03308#S3.E30)\)\. By the existence and strong consistency of empirical Wasserstein barycenters on𝒫2​\(Y\)\\mathcal\{P\}\_\{2\}\(Y\)\[[44](https://arxiv.org/html/2609.03308#bib.bib44)\], we haveW2\(1\)​\(p^B,p⋆\)→0W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\\to 0almost surely\. For the AU part, from \([32](https://arxiv.org/html/2609.03308#S3.E32)\),AU^crit,B​\(s,a\)=1B​∑b=1BVarY∼Pb​\(Y\)\\widehat\{\\mathrm\{AU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\mathrm\{Var\}\_\{Y\\sim P\_\{b\}\}\(Y\)\. SinceP∈𝒫2​\(Y\)P\\in\\mathcal\{P\}\_\{2\}\(Y\)almost surely,VarY∼P​\(Y\)<∞\\mathrm\{Var\}\_\{Y\\sim P\}\(Y\)<\\inftyalmost surely and is integrable; thus by the strong law of large numbers,AU^crit,B​\(s,a\)→𝔼⁡\[VarY∼P​\(Y\)\]=AUcrit⋆​\(s,a\)\\widehat\{\\mathrm\{AU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)\\to\\mathbb\{E\}\[\\mathrm\{Var\}\_\{Y\\sim P\}\(Y\)\]=\\mathrm\{AU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\)almost surely\. For the EU part, by \([31](https://arxiv.org/html/2609.03308#S3.E31)\),EU^crit,B​\(s,a\)=1B​∑b=1B\(W2\(1\)​\(Pb,p^B\)\)2\\widehat\{\\mathrm\{EU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)=\\frac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},\\widehat\{p\}\_\{B\}\)\)^\{2\}\. Add and subtract the population barycenter:

EU^crit,B​\(s,a\)−EUcrit⋆​\(s,a\)\\displaystyle\\widehat\{\\mathrm\{EU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)\-\\mathrm\{EU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\)=\(1B​∑b=1B\(W2\(1\)​\(Pb,p⋆\)\)2−𝔼⁡\[\(W2\(1\)​\(P,p⋆\)\)2\]\)⏟\(I\)\\displaystyle\\quad=\\underbrace\{\\Bigl\(\\tfrac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\)^\{2\}\-\\mathbb\{E\}\[\(W\_\{2\}^\{\(1\)\}\(P,p^\{\\star\}\)\)^\{2\}\]\\Bigr\)\}\_\{\(I\)\}\+1B​∑b=1B\(\(W2\(1\)​\(Pb,p^B\)\)2−\(W2\(1\)​\(Pb,p⋆\)\)2\)⏟\(I​I\)\.\\displaystyle\\quad\\quad\+\\underbrace\{\\tfrac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}\\Bigl\(\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},\\widehat\{p\}\_\{B\}\)\)^\{2\}\-\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\)^\{2\}\\Bigr\)\}\_\{\(II\)\}\.\(38\)Term\(I\)→0\(I\)\\to 0almost surely by the strong law since\(W2\(1\)​\(P,p⋆\)\)2\(W\_\{2\}^\{\(1\)\}\(P,p^\{\\star\}\)\)^\{2\}is integrable underP∈𝒫2​\(Y\)P\\in\\mathcal\{P\}\_\{2\}\(Y\)\. For\(I​I\)\(II\), using\|x2−y2\|≤\(x\+y\)​\|x−y\|\|x^\{2\}\-y^\{2\}\|\\leq\(x\+y\)\|x\-y\|withx=W2\(1\)​\(Pb,p^B\)x=W\_\{2\}^\{\(1\)\}\(P\_\{b\},\\widehat\{p\}\_\{B\}\)andy=W2\(1\)​\(Pb,p⋆\)y=W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\), then the triangle inequality gives

\|\(W2\(1\)​\(Pb,p^B\)\)2−\(W2\(1\)​\(Pb,p⋆\)\)2\|\\displaystyle\\bigl\|\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},\\widehat\{p\}\_\{B\}\)\)^\{2\}\-\(W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\)^\{2\}\\bigr\|≤\(2​W2\(1\)​\(Pb,p⋆\)\+W2\(1\)​\(p^B,p⋆\)\)​W2\(1\)​\(p^B,p⋆\)\.\\displaystyle\\quad\\leq\\bigl\(2W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\+W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\\bigr\)\\,W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\.\(39\)Averaging \([39](https://arxiv.org/html/2609.03308#A2.E39)\) overbbyields

\|\(II\)\|≤\(2⋅1B∑b=1BW2\(1\)\(Pb,p⋆\)\+W2\(1\)\(p^B,p⋆\)\)W2\(1\)\(p^B,p⋆\)\.\|\(II\)\|\\leq\\Bigl\(2\\cdot\\tfrac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\+W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\\Bigr\)\\,W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\.\(40\)Moreover,P∈𝒫2​\(Y\)P\\in\\mathcal\{P\}\_\{2\}\(Y\)implies𝔼⁡\[W2\(1\)​\(P,p⋆\)\]<∞\\mathbb\{E\}\[W\_\{2\}^\{\(1\)\}\(P,p^\{\\star\}\)\]<\\infty, hence by the strong law1B​∑b=1BW2\(1\)​\(Pb,p⋆\)→𝔼⁡\[W2\(1\)​\(P,p⋆\)\]\\tfrac\{1\}\{B\}\\sum\_\{b=1\}^\{B\}W\_\{2\}^\{\(1\)\}\(P\_\{b\},p^\{\\star\}\)\\to\\mathbb\{E\}\[W\_\{2\}^\{\(1\)\}\(P,p^\{\\star\}\)\]almost surely, whileW2\(1\)​\(p^B,p⋆\)→0W\_\{2\}^\{\(1\)\}\(\\widehat\{p\}\_\{B\},p^\{\\star\}\)\\to 0almost surely\. Therefore the right\-hand side of \([40](https://arxiv.org/html/2609.03308#A2.E40)\) converges to00almost surely, so\(I​I\)→0\(II\)\\to 0almost surely\. Combining\(I\)\(I\)and\(I​I\)\(II\)in \([38](https://arxiv.org/html/2609.03308#A2.E38)\) provesEU^crit,B​\(s,a\)→EUcrit⋆​\(s,a\)\\widehat\{\\mathrm\{EU\}\}\_\{\\mathrm\{crit\},B\}\(s,a\)\\to\\mathrm\{EU\}^\{\\star\}\_\{\\mathrm\{crit\}\}\(s,a\)almost surely\. ∎