用于配电网社区电池协调的物理信息驱动图安全强化学习与约束专用评价网络

# Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network
Source: [https://arxiv.org/html/2610.33654](https://arxiv.org/html/2610.33654)
Yue Yangand Hao Wang††thanks:This work was supported in part by the Australian Research Council \(ARC\) Discovery Early Career Researcher Award \(DECRA\) under Grant DE230100046\. \(Corresponding author: Hao Wang\)††thanks:C\. Huang, Y\. Yang, and H\. Wang are with the Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University, Melbourne, VIC 3800, Australia \(e\-mails: \{chenghao\.huang, yue\.yang1, hao\.wang2\}@monash\.edu\)\.

###### Abstract

High penetrations of distributed energy resources \(DERs\) and fast\-varying electric vehicle \(EV\) charging intensify voltage and line\-limit violations in distribution networks \(DNs\), making safe, low\-cost operation increasingly challenging\. This paper proposes a physics\-informed safe deep reinforcement learning \(DRL\) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost\. To represent nonlocal spatiotemporal interactions, we develop a dual\-level physics\-informed graph representation in which line impedance and capacity shape spatial information propagation, while power\-balance, voltage\-profile, and flow\-reconstruction losses regularize the latent state toward control\-relevant physical consistency\. Building on this representation, we develop Lagrangian Dedicated\-Critic PPO \(LDCPPO\), which implements constraint\-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage\- and line\-loading\-related learning signals and combine their advantages only during the Lagrangian policy update\. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage\-violation degree by up to 80\.5% compared with the tested baselines, while also achieving the lowest voltage\-violation count and line\-overload degree with competitive operating cost\.

###### Index Terms:

Battery coordination, distribution network, voltage safety, deep reinforcement learning, graph neural network, physics\-informed neural network\.

## IIntroduction

### I\-ABackground and Motivation

Global efforts to reduce carbon emissions and achieve net\-zero energy systems are driving the decentralization of electricity grids and the rapid growth of Distributed Energy Resources \(DERs\), especially the behind\-the\-meter ones such as rooftop PV and electric vehicles \(EVs\)\[[1](https://arxiv.org/html/2610.33654#bib.bib18)\]\. In Australia, for example, around 40% of households have rooftop solar, over 185,000 have battery storage\[[2](https://arxiv.org/html/2610.33654#bib.bib14)\], and EV ownership exceeded 200,000 by the end of 2024\[[3](https://arxiv.org/html/2610.33654#bib.bib16)\]\. However, the increasing penetration of behind\-the\-meter DERs poses new operational challenges for distribution networks \(DNs\), such as voltage violations and line load capacity violations, which threaten grid safety\[[4](https://arxiv.org/html/2610.33654#bib.bib36),[5](https://arxiv.org/html/2610.33654#bib.bib17)\], demanding better DER coordination\[[6](https://arxiv.org/html/2610.33654#bib.bib3)\]\. Traditional regulation devices struggle under high\-DER conditions due to limited responsiveness, typically like on\-load tap controllers \(OLTCs\) lacking flexibility, and static voltage compensators requiring escalating deployment cost\. On the other hand, behind\-the\-meter solutions such as inverter control\[[7](https://arxiv.org/html/2610.33654#bib.bib35)\], household batteries\[[8](https://arxiv.org/html/2610.33654#bib.bib38)\], or coupling PV with EVs\[[9](https://arxiv.org/html/2610.33654#bib.bib37)\]offer limited and fragmented regulation capacity due to small sizes but large quantities, and intermittency, making network\-wide safety difficult to comply\.

To overcome these limitations, Community\-scale Battery Energy Storage Systems \(CBESSs\)\[[10](https://arxiv.org/html/2610.33654#bib.bib12)\]have emerged as shared assets and managed by Distribution System Operators \(DSOs\), designed to deliver grid support and community\-wide benefits\. CBESSs are often installed at the neighborhood or substation level with the capacity of 200kWh\-2MWh commonly\[[11](https://arxiv.org/html/2610.33654#bib.bib13)\], serving consumer communities with DERs\. Compared with individual household batteries, CBESSs offer a versatile solution that can simultaneously provide voltage and power flow regulation, meet collective consumer needs, and participate in energy markets for arbitrage\. Furthermore, unlike distributed batteries, CBESSs can be owned and scheduled by DSOs, making CBESS coordination a viable solution to safe DN operation\[[12](https://arxiv.org/html/2610.33654#bib.bib24),[13](https://arxiv.org/html/2610.33654#bib.bib23),[14](https://arxiv.org/html/2610.33654#bib.bib4)\]\. Australia’s DSOs, known as Distribution Network Service Providers, have already piloted CBESSs for voltage support and solar hosting, such as Energy Queensland’s deployment of 69 neighborhood batteries \(with a total capacity of 4\.41 MW/7\.47 MWh\) across Brisbane and southeast Queensland\[[15](https://arxiv.org/html/2610.33654#bib.bib10)\], demonstrating CBESSs’ potential of supporting a safe, reliable, and economical energy transition under behind\-the\-meter DER growth\.

### I\-BLiterature Review and Challenges

TABLE I:Comparison of grid\-level voltage regulation methods for medium\-size BESS, including business BESS and CBESS\.Ref\.Problem setting \(mainly about DERs\)MethodologyPV deployment \(%\)PV penetration \(%\)Residential nighttime EVCSolutionConstraintsFeature correlation modelingPhysics\-informed\[[16](https://arxiv.org/html/2610.33654#bib.bib27)\]27%50%NoControl theorySeparateNoNo\[[17](https://arxiv.org/html/2610.33654#bib.bib8),[18](https://arxiv.org/html/2610.33654#bib.bib7),[19](https://arxiv.org/html/2610.33654#bib.bib6),[20](https://arxiv.org/html/2610.33654#bib.bib19),[21](https://arxiv.org/html/2610.33654#bib.bib20)\]\-, \-, \-, 40%,18%40%,40%,70%, \-, \-NoOptimizationSeparateNoNo\[[22](https://arxiv.org/html/2610.33654#bib.bib34)\]44%∼\\sim50%NoDRLMixedNoNo\[[23](https://arxiv.org/html/2610.33654#bib.bib28),[24](https://arxiv.org/html/2610.33654#bib.bib31)\]\-, 24%90%, 30%NoDRLMixedNoNo\[[25](https://arxiv.org/html/2610.33654#bib.bib30)\]18%70%NoDRLMixedNoNo\[[26](https://arxiv.org/html/2610.33654#bib.bib33)\]8%∼\\sim40%NoDRLMixedNoNo\[[27](https://arxiv.org/html/2610.33654#bib.bib29)\]9%∼\\sim40%NoDRLMixedSpatialYes\[[28](https://arxiv.org/html/2610.33654#bib.bib44)\]6%15%NoDRLPartially separateNoNoProposed80%90%YesDRLSeparateSpatiotemporalYes
Note:“\-”: Not applicable, since the related works have not considered the system\-wide cases, or explicitly shown the exact values\. “PV deployment ratio”: The proportion of buses in the network that host rooftop PV installation\. “PV penetration ratio”: The maximum ratio of installed PV peak capacity to the corresponding peak load at any bus in the network\.

Before the emergence of DSO\-owned CBESSs for network support, researchers focused on controlling BESS in power systems for economic benefits, such as arbitrage or service sale\. Early efforts primarily relied on optimization\-based frameworks\[[29](https://arxiv.org/html/2610.33654#bib.bib25),[17](https://arxiv.org/html/2610.33654#bib.bib8),[19](https://arxiv.org/html/2610.33654#bib.bib6),[18](https://arxiv.org/html/2610.33654#bib.bib7)\]or heuristic methods\[[30](https://arxiv.org/html/2610.33654#bib.bib22)\], aimed at maximizing economic efficiency to improve energy sharing, investment returns, and flexible service participation\. But these works often do not consider physical network constraints, so more recent optimization\-based studies have begun to incorporate typical network constraints, especially voltage violation, into the BESS scheduling problem\[[20](https://arxiv.org/html/2610.33654#bib.bib19),[21](https://arxiv.org/html/2610.33654#bib.bib20)\]as uncoordinated operation of DERs can harm grid safety\. Moreover,\[[16](https://arxiv.org/html/2610.33654#bib.bib27)\]proposes a two\-stage coordinated control strategy that centrally manages OLTC adjustments while locally controlling multiple BESS units for fast active power response, aiming to minimize voltage deviations\. Nevertheless, optimization\-based approaches depend on accurate system models, making them less suitable for highly dynamic and uncertain environments, which however are increasingly common in distribution networks with volatile DERs\.

As energy systems become more decentralized and uncertain, DRL offers a learning\-based approach that can learn causal control policies from historical interactions and respond rapidly to changing operating conditions during execution\. In the context of BESS scheduling, DRL has demonstrated strong potential in handling stochasticity in renewable output, user demand, and electricity prices\[[31](https://arxiv.org/html/2610.33654#bib.bib21),[32](https://arxiv.org/html/2610.33654#bib.bib32),[33](https://arxiv.org/html/2610.33654#bib.bib2),[34](https://arxiv.org/html/2610.33654#bib.bib1),[35](https://arxiv.org/html/2610.33654#bib.bib26)\]via exploring flexible strategies and optimizing long\-term objectives under uncertain conditions\. Furthermore, some DRL\-based BESS controlling methods account for the physical constraints of DNs\. Based on DRL’s reward\-driven nature, some works integrated physical constraints into the reward function for long\-term optimization\[[23](https://arxiv.org/html/2610.33654#bib.bib28),[26](https://arxiv.org/html/2610.33654#bib.bib33),[22](https://arxiv.org/html/2610.33654#bib.bib34),[27](https://arxiv.org/html/2610.33654#bib.bib29)\], in which the work of\[[27](https://arxiv.org/html/2610.33654#bib.bib29)\]integrates structured graph information from the DN physics into the DRL training process to enhance decision robustness towards voltage violation\. However, merely embedding physical constraints into the reward signal remains insufficient, as it can lead to voltage violations under highly dynamic or extreme conditions\.

Recent RL studies have also explored multi\-critic designs\. However, only a limited number of studies have adopted such multi\-critic safe RL structures for power system operation\. The work of\[[28](https://arxiv.org/html/2610.33654#bib.bib44)\]developed a hybrid day\-ahead and real\-time framework combining optimization\-based prediction and the constrained DRL paradigm\. The study in\[[36](https://arxiv.org/html/2610.33654#bib.bib39)\]used multiple critics for heterogeneous operational objectives together with iterative safe exploration\. However, for DN operation with heterogeneous voltage\- and line\-loading constraints, these methods do not explicitly isolate the long\-term learning signals of individual physical constraint costs within a unified Lagrangian policy\-optimization framework\.

A brief overview of existing CBESS scheduling methods is summarized in Table[I](https://arxiv.org/html/2610.33654#S1.T1), illustrating that CBESS coordination for network constraint maintenance under rich DERs remains an important gap in current research\. Compared with prior works, our study considers a high\-PV deployment scenario \(80% bus coverage\) with elevated PV penetration \(up to 90%\) and explicitly incorporates residential nighttime EV charging demand\. Such settings reflect emerging feeders in regions with aggressive rooftop PV adoption and rapid EV uptake, where distributed PV systems are densely installed across residential buses\. High PV deployment significantly increases spatial voltage coupling, making local voltage deviations propagate feeder\-wide\. High PV penetration intensifies reverse power flow and overvoltage risk, shrinking the voltage\-feasible action region\. Introducing residential nighttime EV charging creates conflicting operating regimes: daytime overvoltage driven by PV surplus and nighttime undervoltage caused by concentrated charging demand\.

Based on the literature review, we identify three fundamental challenges:

- •Mixed constraint feedback under multi\-CBESS constraint\-aware learning:When multiple CBESS units operate, their charging and discharging actions interact through network voltage and line loading dynamics, causing constraint violations at different buses and lines to occur concurrently and interact\. In many DRL studies, different constraint signals are aggregated via reward shaping or weighted penalties into a single feedback signal\. This obscures which constraint is active and how severe individual violations are, thus weakening constraint awareness and destabilizing training\. A key challenge is therefore to design a constraint\-aware learning framework that separates reward from constraints and further decomposes constraints for independent estimation\.
- •Insufficient physical awareness in purely data\-driven policies:Although CBESS can regulate power injections to support grid operation, their large capacity means inappropriate dispatch may induce voltage fluctuations or line stress\. Purely data\-driven policies may not internalize power\-flow physics and electrical couplings, leading to unstable or oscillatory training behavior\. Incorporating physics\-informed structural priors into representation learning is therefore important for improving training robustness and convergence consistency\.
- •Nonlocal spatiotemporal interactions under large\-scale CBESS dispatch:Due to electrical coupling in distribution networks, CBESS actions at one location can affect voltages and line loadings across multiple buses\. These effects also accumulate over time through network dynamics and battery state evolution\. Effective coordination thus requires models that capture both spatial dependencies and temporal dynamics\. Policies that ignore such spatiotemporal structure may lead to suboptimal or unstable multi\-CBESS coordination\.

The resulting methodological question is how to construct a control state that retains nonlocal power\-flow relationships while separately assigning long\-term credit to heterogeneous operational constraints\. These two difficulties motivate the dual\-level PISTG representation and LDCPPO value decomposition developed in this work\.

### I\-CMain Work and Contributions

We study the coordinated operation of CBESSs in a DN with rich rooftop PV and residential nighttime EV charging, targeting reduced network\-wide physical constraint violations and lower DN\-level operating cost under high adoption of behind\-the\-meter DERs\. To meet these requirements, we propose a DRL framework that \(1\) embeds power\-flow priors into grid\-state representation through a physics\-informed spatiotemporal graph encoder, to anticipate spatially coupled dynamics for supporting smarter decision\-making; \(2\) casts CBESS control as a Constrained Markov Decision Process \(CMDP\) with multiple operational\-constraint costs and optimizes economic return subject to expected constraint\-cost budgets; \(3\) implements constraint\-specific value decomposition within a Lagrangian policy\-optimization framework\. The main contributions are summarized as follows\.

- •Problem\-level novelty in learning\-based safe multi\-CBESS coordination\.We investigate a DSO\-side learning problem for coordinating multiple CBESSs under jointly intensified rooftop\-PV and residential nighttime\-EV risks\. The problem requires learning a causal coordination policy when daytime PV surplus and nighttime EV charging create alternating overvoltage, undervoltage, and line\-loading risks, while multiple CBESS actions are nonlocally coupled through power flows and temporally coupled through PV generation, EV demand, electricity prices, and battery SOC dynamics\. This setting raises the distinct challenge of achieving rapid policy execution while retaining awareness of heterogeneous voltage\- and line\-loading\-related constraints\.
- •Dual\-level physics\-informed spatiotemporal representation\.We develop a Physics\-Informed Spatiotemporal Graph \(PISTG\) encoder that incorporates distribution\-network physics at two complementary levels\. At the graph\-structure level, line resistance, reactance, and capacity shape how information propagates across electrically coupled buses; at the latent\-representation level, consistency in the Kirchhoff’s Current Law \(KCL\), voltage\-drop consistency, and voltage/branch\-flow reconstruction regularize what control\-relevant physical relationships are preserved\. This design enables the policy to represent both the propagation of nonlocal network effects and their physical consequences for coordinated CBESS control\.
- •Constraint\-specific value decomposition for safe policy optimization\.We develop LDCPPO to preserve the distinct long\-term learning signals of the economic objective and individual operational constraints\. Separate critics estimate the economic return, voltage\-violation degree, voltage\-violation count, and line\-overload degree using independent return targets, and their advantages are combined only during the Lagrangian actor update\. This design avoids heuristic reward aggregation and reduces interference and credit\-assignment ambiguity among heterogeneous constraint signals\.
- •Effective constraint–economy performance with real\-time policy execution\.Extensive simulations on the IEEE 33\- and 69\-bus systems demonstrate the effectiveness and scalability of the proposed framework\. Compared with all tested baselines, PISTG\-LDCPPO reduces the voltage violation by approximately50%50\\%, and achieves the lowest voltage\-violation count and line\-overload degree, while maintaining competitive operating cost and real\-time responsiveness\. These results demonstrate that the physics\-informed representation and constraint\-specific value learning provide complementary benefits, supporting their problem\-driven coupling\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2610.33654#S2)models the CBESS coordination problem\. Section[III](https://arxiv.org/html/2610.33654#S3)introduces the developed method\. Section[IV](https://arxiv.org/html/2610.33654#S4)provides experimental results and analysis to validate the effectiveness of our method\. Section[V](https://arxiv.org/html/2610.33654#S5)draws the conclusion\.

## IISystem Modeling

We consider a radial DN with highly integrated rooftop PV and residential nighttime EV charging across multiple buses, where several CBESS units are strategically deployed at nodes within DER\-dense regions\. In such settings, voltage\-limit compliance and energy efficiency can be challenged by the temporal and spatial mismatch between daytime PV generation and nighttime EV charging demand, while CBESS operation may also affect line loading across the network\. An overview of the modeled system is illustrated in Fig\.[1](https://arxiv.org/html/2610.33654#S2.F1)\.

![Refer to caption](https://arxiv.org/html/2610.33654v1/system_model.png)Fig\. 1:System model of CBESS coordination via a DSO’s centralized training in a rich\-DER DN\.### II\-ASystem Description

We consider a radial DN\(𝒩,ℒ\)\(\\mathcal\{N\},\\mathcal\{L\}\)managed by a DSO over intra\-day periodst∈𝒯=\{1,…,T\}t\\in\\mathcal\{T\}=\\\{1,\\dots,T\\\}\. Then, we successively define CBESS attributes, high DER\-penetration buses, bus\-level inverters, and DN\-level principles\.

#### II\-A1CBESS

To coordinate the rooftop PV shift for meeting both system safety and energy efficiency, we consider a set of CBESSs owned by the DSO, denoted byℳ\\mathcal\{M\}\. Each CBESSm∈ℳm\\in\\mathcal\{M\}is connected at busξ⁡\(m\)\\xi\(m\)and characterized by charge/discharge efficiencies\(ηmch,ηmdis\)\(\\eta^\{\\text\{ch\}\}\_\{m\},\\eta^\{\\text\{dis\}\}\_\{m\}\), charge/discharge limits\(P¯mch,P¯mdis\)\(\\overline\{P\}^\{\\text\{ch\}\}\_\{m\},\\overline\{P\}^\{\\text\{dis\}\}\_\{m\}\), and state\-of\-charge \(SOC\) bounds\[S​O​C¯m,S​O​C¯m\]\[\\underline\{SOC\}\_\{m\},\\overline\{SOC\}\_\{m\}\]\. The charging/discharging power variables arepm,tchp^\{\\text\{ch\}\}\_\{m,t\}andpm,tdisp^\{\\text\{dis\}\}\_\{m,t\}, the active power of CBESS ispm,tCB=pm,tdis−pm,tchp^\{\\text\{CB\}\}\_\{m,t\}=p^\{\\text\{dis\}\}\_\{m,t\}\-p^\{\\text\{ch\}\}\_\{m,t\}, the reactive support isqm,tCBq^\{\\text\{CB\}\}\_\{m,t\}, and the stored energy isEm,tE\_\{m,t\}with capacityEmcapE\_\{m\}^\{\\text\{cap\}\}\. Their dynamics and constraints are given by:

Em,t\+1=Em,t\+ηmch​pm,tch​Δ​t−1ηmdis​pm,tdis​Δ​t,\\displaystyle E\_\{m,t\+1\}=E\_\{m,t\}\+\\eta^\{\\text\{ch\}\}\_\{m\}p^\{\\text\{ch\}\}\_\{m,t\}\\Delta t\-\\frac\{1\}\{\\eta^\{\\text\{dis\}\}\_\{m\}\}p^\{\\text\{dis\}\}\_\{m,t\}\\Delta t,\(1a\)S​O​Cm,t=Em,tEmcap,S​O​C¯m≤S​O​Cm,t≤S​O​C¯m,\\displaystyle SOC\_\{m,t\}=\\frac\{E\_\{m,t\}\}\{E\_\{m\}^\{\\text\{cap\}\}\},\\quad\\underline\{SOC\}\_\{m\}\\leq SOC\_\{m,t\}\\leq\\overline\{SOC\}\_\{m\},\(1b\)0≤pm,tch≤P¯mch,0≤pm,tdis≤P¯mdis,\\displaystyle 0\\leq p^\{\\text\{ch\}\}\_\{m,t\}\\leq\\overline\{P\}^\{\\text\{ch\}\}\_\{m\},\\quad 0\\leq p^\{\\text\{dis\}\}\_\{m,t\}\\leq\\overline\{P\}^\{\\text\{dis\}\}\_\{m\},\(1c\)pm,tch⋅pm,tdis=0,\\displaystyle p^\{\\text\{ch\}\}\_\{m,t\}\\cdot p^\{\\text\{dis\}\}\_\{m,t\}=0,\(1d\)\(pm,tdis−pm,tch\)2\+\(qm,tCB\)2≤\(SmCB\)2,\\displaystyle\(p^\{\\text\{dis\}\}\_\{m,t\}\-p^\{\\text\{ch\}\}\_\{m,t\}\)^\{2\}\+\(q^\{\\text\{CB\}\}\_\{m,t\}\)^\{2\}\\leq\(S^\{\\text\{CB\}\}\_\{m\}\)^\{2\},\(1e\)Em,0=Eminit,\\displaystyle E\_\{m,0\}=E^\{\\text\{init\}\}\_\{m\},\(1f\)in which Eq\. \([1a](https://arxiv.org/html/2610.33654#S2.E1.1)\) defines CBESS dynamics\. Eq\. \([1b](https://arxiv.org/html/2610.33654#S2.E1.2)\) defines SOC boundaries\. Eq\. \([1c](https://arxiv.org/html/2610.33654#S2.E1.3)\) defines charging/discharging rate boundaries\. Eq\. \([1d](https://arxiv.org/html/2610.33654#S2.E1.4)\) defines charging/discharging exclusivity\. Eq\. \([1e](https://arxiv.org/html/2610.33654#S2.E1.5)\) defines the CBESS loading limit, whereSmCBS^\{\\text\{CB\}\}\_\{m\}is inverter apparent power rating\. Eq\. \([1f](https://arxiv.org/html/2610.33654#S2.E1.6)\) defines the CBESS storage initialization at each day’s beginning, whereEminitE^\{\\text\{init\}\}\_\{m\}is the initial volume of themm\-th CBESS and generally set as 50% ofS​O​C¯m\\overline\{SOC\}\_\{m\}\.

#### II\-A2High DER\-Penetration Buses

Let𝒩DER⊆𝒩\\mathcal\{N\}^\{\\text\{DER\}\}\\subseteq\\mathcal\{N\}denote the set of high\-DER\-penetration buses, characterized by both significant rooftop PV installation and residential EV charging demand\. This reflects regions with high adoption of DERs\. To avoid overly dispersed DER deployment that weakens spatial interaction effects, high\-DER buses are primarily located around CBESS\-connected buses, forming locally intensive DER regions\.

For eachi∈𝒩DERi\\in\\mathcal\{N\}^\{\\text\{DER\}\}, we define time\-varying active and reactive load profilespi,tloadp^\{\\text\{load\}\}\_\{i,t\},qi,tloadq^\{\\text\{load\}\}\_\{i,t\}, and available PV generation profilep~i,tPV\\tilde\{p\}^\{\\text\{PV\}\}\_\{i,t\}\. For buses outside𝒩DER\\mathcal\{N\}^\{\\text\{DER\}\}, standard load profiles are assumed without PV generation and EV charging\.

#### II\-A3PV Inverters

The PV system at each busi∈𝒩DERi\\in\\mathcal\{N\}^\{\\text\{DER\}\}is assumed to have an inverter that regulates rooftop PV output and provides reactive support\. The PV curtailment ratio is denoted byκi,t∈\[0,1\]\\kappa\_\{i,t\}\\in\[0,1\], whereκi,t=0\\kappa\_\{i,t\}=0means no curtailment andκi,t=1\\kappa\_\{i,t\}=1means full curtailment\. The curtailed PV output is:

pi,tPV=\(1−κi,t\)​p~i,tPV\.\\displaystyle p^\{\\text\{PV\}\}\_\{i,t\}=\(1\-\\kappa\_\{i,t\}\)\\,\\tilde\{p\}^\{\\text\{PV\}\}\_\{i,t\}\.\(2\)The inverter can also inject or absorb reactive powerqi,tinvq^\{\\text\{inv\}\}\_\{i,t\}, bounded by its apparent power ratingSimaxS^\{\\max\}\_\{i\}:

\(pi,tPV\)2\+\(qi,tinv\)2≤\(Simax\)2\.\\displaystyle\(p^\{\\text\{PV\}\}\_\{i,t\}\)^\{2\}\+\(q^\{\\text\{inv\}\}\_\{i,t\}\)^\{2\}\\leq\(S^\{\\max\}\_\{i\}\)^\{2\}\.\(3\)wherep~i,tPV≥0\\tilde\{p\}^\{\\text\{PV\}\}\_\{i,t\}\\geq 0denote the original PV output at busii\.

#### II\-A4DN

We model the radial DN\(𝒩,ℒ\)\(\\mathcal\{N\},\\mathcal\{L\}\)using Baran\-Wu DistFlow formulation\[[37](https://arxiv.org/html/2610.33654#bib.bib9)\]\. The DN\(𝒩,ℒ\)\(\\mathcal\{N\},\\mathcal\{L\}\)can be equivalently described by an adjacency matrix𝐀∈\{0,1\}\|𝒩\|×\|𝒩\|\\mathbf\{A\}\\in\\\{0,1\\\}^\{\|\\mathcal\{N\}\|\\times\|\\mathcal\{N\}\|\}, where each entry is defined as:

Ai​j=\{1,if​\(i,j\)∈ℒ,0,otherwise\.\\displaystyle A\_\{ij\}=\\begin\{cases\}1,\\quad\\text\{if \}\(i,j\)\\in\\mathcal\{L\},\\\\ 0,\\quad\\text\{otherwise\}\.\\end\{cases\}\(4\)Since the network is radial,𝐀\\mathbf\{A\}is sparse and symmetric, with exactly one unique path between any two buses\. Each line\(i,j\)∈ℒ\(i,j\)\\in\\mathcal\{L\}denotes a branch from parent busiito child busjj\. The corresponding branch flows are denoted by\(pi​j,t,qi​j,t\)\(p\_\{ij,t\},q\_\{ij,t\}\)\. Bus00denotes the substation with fixed voltage magnitudeV0,t=V0V\_\{0,t\}=V\_\{0\}\.

For buses without CBESS, the balance reduces to the standard form between demand, PV, and power flow\. For buses hosting CBESS, additional charge/discharge terms are included:

∑j:\(j,i\)∈ℒpj​i,t\+pPVi,t\+∑m:ξ⁡\(m\)=ipCBm,t=ploadi,t\+∑j:\(i,j\)∈ℒpi​j,t,\\displaystyle\\sum\_\{j:\(j,i\)\\in\\mathcal\{L\}\}p\_\{ji,t\}\+p^\{\\text\{PV\}\}\_\{i,t\}\+\\sum\_\{m:\\xi\(m\)=i\}p^\{\\text\{CB\}\}\_\{m,t\}=p^\{\\text\{load\}\}\_\{i,t\}\+\\sum\_\{j:\(i,j\)\\in\\mathcal\{L\}\}p\_\{ij,t\},\(5a\)∑j:\(j,i\)∈ℒqj​i,t\+qinvi,t\+∑m:ξ⁡\(m\)=iqCBm,t=qloadi,t\+∑j:\(i,j\)∈ℒqi​j,t,\\displaystyle\\sum\_\{j:\(j,i\)\\in\\mathcal\{L\}\}q\_\{ji,t\}\+q^\{\\text\{inv\}\}\_\{i,t\}\+\\sum\_\{m:\\xi\(m\)=i\}q^\{\\text\{CB\}\}\_\{m,t\}=q^\{\\text\{load\}\}\_\{i,t\}\+\\sum\_\{j:\(i,j\)\\in\\mathcal\{L\}\}q\_\{ij,t\},\(5b\)where\(j,i\)\(j,i\)denotes all downstream branches of thejj\-th bus, andξ⁡\(m\)\\xi\(m\)denotes the bus at which themm\-th CBESS is connected\.

Based on the classical Baran\-Wu DistFlow model, the branch power balance and voltage magnitude relations along line\(i,j\)\(i,j\)are described as follows:

pi​j,t=∑k:\(j,k\)pj​k,t\+pj,t\+ri​jℓi​j,t,\\displaystyle p\_\{ij,t\}=\\sum\_\{k:\(j,k\)\}p\_\{jk,t\}\+p\_\{j,t\}\+r\_\{ij\}\\ell\_\{ij,t\},\(6a\)qi​j,t=∑k:\(j,k\)qj​k,t\+qj,t\+xi​jℓi​j,t,\\displaystyle q\_\{ij,t\}=\\sum\_\{k:\(j,k\)\}q\_\{jk,t\}\+q\_\{j,t\}\+x\_\{ij\}\\ell\_\{ij,t\},\(6b\)Vj,t2=Vi,t2−2​\(ri​j​pi​j,t\+xi​j​qi​j,t\)\+\(ri​j2\+xi​j2\)​ℓi​j,t,\\displaystyle V^\{2\}\_\{j,t\}=V^\{2\}\_\{i,t\}\-2\(r\_\{ij\}p\_\{ij,t\}\+x\_\{ij\}q\_\{ij,t\}\)\+\(r^\{2\}\_\{ij\}\+x^\{2\}\_\{ij\}\)\\ell\_\{ij,t\},\(6c\)ℓi​j,t​Vi,t2=pi​j,t2\+qi​j,t2,\\displaystyle\\ell\_\{ij,t\}V^\{2\}\_\{i,t\}=p^\{2\}\_\{ij,t\}\+q^\{2\}\_\{ij,t\},\(6d\)V¯≤Vi,t≤V¯,∀i∈𝒩\.\\displaystyle\\underline\{V\}\\leq V\_\{i,t\}\\leq\\overline\{V\},\\quad\\forall i\\in\\mathcal\{N\}\.\(6e\)pi​j,t2\+qi​j,t2≤S¯i​j2,∀\(i,j\)∈ℒ,\\displaystyle p\_\{ij,t\}^\{2\}\+q\_\{ij,t\}^\{2\}\\leq\\overline\{S\}\_\{ij\}^\{2\},\\quad\\forall\(i,j\)\\in\\mathcal\{L\},\(6f\)whereri​jr\_\{ij\}andxi​jx\_\{ij\}are the resistance and reactance of the line\. In principle, bus voltages are required to remain within admissible bounds as shown in Eq\. \([6e](https://arxiv.org/html/2610.33654#S2.E6.5)\), and loads on branches are required to remain within line load limits as shown in Eq\. \([6f](https://arxiv.org/html/2610.33654#S2.E6.6)\), whereS¯i​j\\overline\{S\}\_\{ij\}is the apparent capacity limit\.

### II\-BObjective

This problem aims to learn a coordination strategy for multiple CBESS units operated by the DSO under high penetration of behind\-the\-meter DERs in a radial DN\. To clarify the target problem before introducing the CMDP formulation, the task can be interpreted as a finite\-horizon stochastic coordination problem\. Letω\\omegadenote the uncertain operating scenario, including rooftop PV availability, active/reactive load demand, residential EV charging demand, wholesale electricity price, and the train–test distribution mismatch caused by sampled daily profiles\. The DSO aims to learn a causal policyπ\\pithat minimizes the expected daily economic cost while limiting network constraint violations:

minπ\\displaystyle\\min\_\{\\pi\}\\quad𝔼ω​\[∑t∈𝒯\(fBD​\(t\)\+fEC​\(t\)\)\],\\displaystyle\\mathbb\{E\}\_\{\\omega\}\\left\[\\sum\_\{t\\in\\mathcal\{T\}\}\\left\(f^\{\\mathrm\{BD\}\}\(t\)\+f^\{\\mathrm\{EC\}\}\(t\)\\right\)\\right\],\(7a\)s\.t\.\\displaystyle\\mathrm\{s\.t\.\}\\quad\([1](https://arxiv.org/html/2610.33654#S2.E1)\),\([2](https://arxiv.org/html/2610.33654#S2.E2)\),\([3](https://arxiv.org/html/2610.33654#S2.E3)\),\([5](https://arxiv.org/html/2610.33654#S2.E5)\),\([6](https://arxiv.org/html/2610.33654#S2.E6)\),\\displaystyle\\eqref\{eq: battery\},~\\eqref\{eq:pv curt\},~\\eqref\{eq:inverter\},~\\eqref\{eq: bus power\},~\\eqref\{eq: distflow const\},\(7b\)𝔼ω​\[∑t∈𝒯fVD​\(t\)\]≤dVD,\\displaystyle\\mathbb\{E\}\_\{\\omega\}\\left\[\\sum\_\{t\\in\\mathcal\{T\}\}f^\{\\mathrm\{VD\}\}\(t\)\\right\]\\leq d\_\{\\mathrm\{VD\}\},\(7c\)𝔼ω​\[∑t∈𝒯fVN​\(t\)\]≤dVN,\\displaystyle\\mathbb\{E\}\_\{\\omega\}\\left\[\\sum\_\{t\\in\\mathcal\{T\}\}f^\{\\mathrm\{VN\}\}\(t\)\\right\]\\leq d\_\{\\mathrm\{VN\}\},\(7d\)𝔼ω​\[∑t∈𝒯fLO​\(t\)\]≤dLO\.\\displaystyle\\mathbb\{E\}\_\{\\omega\}\\left\[\\sum\_\{t\\in\\mathcal\{T\}\}f^\{\\mathrm\{LO\}\}\(t\)\\right\]\\leq d\_\{\\mathrm\{LO\}\}\.\(7e\)
Here,fEC​\(t\)f^\{\\mathrm\{EC\}\}\(t\)andfBD​\(t\)f^\{\\mathrm\{BD\}\}\(t\)denote the energy cost of the whole DN and the CBESS degradation cost, respectively, whilefVD​\(t\)f^\{\\mathrm\{VD\}\}\(t\),fVN​\(t\)f^\{\\mathrm\{VN\}\}\(t\), andfLO​\(t\)f^\{\\mathrm\{LO\}\}\(t\)denote the voltage\-violation degree, voltage\-violation count, and line\-overload degree, whose expected cumulative tolerances are specified bydVDd\_\{\\mathrm\{VD\}\},dVNd\_\{\\mathrm\{VN\}\}, anddLOd\_\{\\mathrm\{LO\}\}, respectively\. These terms correspond to the reward and operational\-constraint\-cost definitions in the subsequent CMDP formulation\. These constraints are evaluated at the network level through power\-flow relationships, and therefore are coupled across spatially distributed buses\. The additional challenge arises from the complex spatiotemporal interactions under large\-scale DER integration and multi\-CBESS dispatch, which makes constraint evaluation and economic optimization tightly interdependent\. Therefore, the above optimization\-oriented form is used to clarify the physical and economic target, while the actual sequential decision\-making problem is formulated as a constrained Markov decision process \(CMDP\) in Section[III\-B](https://arxiv.org/html/2610.33654#S3.SS2)\.

## IIIMethodology

Building on Section[II](https://arxiv.org/html/2610.33654#S2), this section presents the proposed framework\. First, the dual\-level Physics\-Informed Spatiotemporal Graph \(PISTG\) representation integrates line\-parameter\-weighted graph propagation with physics\-regularized latent representation learning\. Second, the CBESS coordination task is formulated as a CMDP, where PISTG embeddings define the states, CBESS/inverter set\-points define the actions, and voltage and line\-loading violations are modeled as separate operational\-constraint costs\. Third, Lagrangian Dedicated\-Critic PPO \(LDCPPO\) implements constraint\-specific value decomposition using independent reward and cost critics with adaptive Lagrangian multipliers to optimize economic return subject to expected constraint\-cost budgets\. Together, these components support physically informed state representation and structured multi\-constraint policy optimization\.

Fig\. 2:Problem\-driven coupling of the dual\-level PISTG representation \(Stage 1\) and constraint\-specific LDCPPO policy optimization \(Stage 2\) for DSO\-coordinated multi\-CBESS control\.Figure[2](https://arxiv.org/html/2610.33654#S3.F2)first provides an abstracted overview of the overall framework\. The DSO receives multi\-type DN observations, encodes them through the PISTG state encoder, generates coordinated CBESS/inverter/PV\-curtailment actions through the LDCPPO actor, and applies these actions to the distribution\-network environment\. The resulting next observations close the online execution loop, while the reward and operational\-constraint costs are fed to the LDCPPO learning process for training\-time policy update only\.

The overall workflow is illustrated in Fig\.[3](https://arxiv.org/html/2610.33654#S3.F3)\. The main body of the figure further separates the framework into Stage 1, i\.e\., physics\-regularized PISTG representation learning, and Stage 2, i\.e\., LDCPPO learning with rollout, learning\-signal computation, and actor/critic/multiplier updates\. The PISTG backbone is also highlighted to show how bus features, the physics\-informed adjacency matrix, and price information are processed by GCN, attention, and TCN modules\.

![Refer to caption](https://arxiv.org/html/2610.33654v1/method.png)Fig\. 3:Problem\-driven coupling of the dual\-level PISTG representation \(Stage 1\) and constraint\-specific LDCPPO policy optimization \(Stage 2\) for DSO\-coordinated multi\-CBESS control\.### III\-ADual\-Level Physics\-Informed Spatiotemporal Representation

To capture DN spatiotemporal dependencies, we design a dual\-level physics\-informed spatiotemporal graph \(PISTG\) representation that incorporates electrical coupling into graph propagation and physical consistency into latent representation learning\. Briefly, the encoder builds a physics\-informed adjacency matrix \(PIAM\) reflecting line coupling, extracts spatial features from power\-flow patterns and temporal features via temporal convolutions, and is trained under an auto\-encoder paradigm with physics\-informed losses\. PIAM and the physics\-informed losses play complementary roles: PIAM shapes the electrical message\-passing structure, while the KCL\- and voltage\-drop\-related losses regularize the physical consistency of the learned latent representation\.

#### III\-A1Data Structure Construction

First, a spatiotemporal graph snapshot is constructed by windowed bus features and prices to characterize the DN dynamics\. For the temporal dimension, we construct a window of lengthWW\(e\.g\.,WWconsecutive 5\-minute intervals\), which stacks recent observations\. Further on the spatial side, we organize the feature windows of all the buses, each of which contains active/reactive load, PV generation, voltage, and CBESS energy \(set to zero if no CBESS is connected to theii\-th bus\), as a three\-dimensional tensor\. The bus features𝐗bust−W\+1:t\\mathbf\{X\}^\{\\text\{bus\}\}\_\{t\-W\+1:t\}and temporal price window𝐗pricet−W\+1:t\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1:t\}are formulated as follows:

𝐗t−W\+1:tbus=\[\[𝐱i,t′bus\]t′=t−W\+1t\]i∈𝒩∈ℝW×\|𝒩\|×dbus,\\displaystyle\\mathbf\{X\}^\{\\text\{bus\}\}\_\{t\-W\+1:t\}=\\Big\[\\big\[\\mathbf\{x\}^\{\\text\{bus\}\}\_\{i,t^\{\\prime\}\}\\big\]\_\{t^\{\\prime\}=t\-W\+1\}^\{t\}\\Big\]\_\{i\\in\\mathcal\{N\}\}\\in\\mathbb\{R\}^\{W\\times\|\\mathcal\{N\}\|\\times d\_\{\\text\{bus\}\}\},\(8a\)𝐱i,t′bus=\[pi,t′load,qi,t′load,pi,t′PV,Vi,t′,Ei,t′⏟0,if​∄​m:ξ⁡\(m\)=i\]⊤∈ℝdbus,\\displaystyle\\mathbf\{x\}^\{\\text\{bus\}\}\_\{i,t^\{\\prime\}\}=\\Big\[p^\{\\text\{load\}\}\_\{i,t^\{\\prime\}\},q^\{\\text\{load\}\}\_\{i,t^\{\\prime\}\},p^\{\\text\{PV\}\}\_\{i,t^\{\\prime\}\},V\_\{i,t^\{\\prime\}\},\\underbrace\{E\_\{i,t^\{\\prime\}\}\}\_\{0,\\text\{if \}\\nexists m:\\xi\(m\)=i\}\\Big\]^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{bus\}\}\},\(8b\)𝐗t−W\+1:tprice=\[φt′wh\]t′=t−W\+1t∈ℝW×1,\\displaystyle\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1:t\}=\[\\varphi^\{\\text\{wh\}\}\_\{t^\{\\prime\}\}\]\_\{t^\{\\prime\}=t\-W\+1\}^\{t\}\\in\\mathbb\{R\}^\{W\\times 1\},\(8c\)wheredbusd\_\{\\text\{bus\}\}is the number of the bus\-level features at one moment\. The bus\-level variables, including load, PV generation, voltage magnitude, and CBESS energy state, are assumed to be obtained from pseudo\-measurements, while the feeder topology and line parameters are assumed to be available from the DSO’s network model\.

Additionally, we augment edges with line features:

𝐁=\[𝐛i​j\]\(i,j\)∈ℒ∈ℝ\|ℒ\|×dline,\\displaystyle\\mathbf\{B\}=\\big\[\\mathbf\{b\}\_\{ij\}\\big\]\_\{\(i,j\)\\in\\mathcal\{L\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{L\}\|\\times d\_\{\\text\{line\}\}\},\(9a\)𝐛i​j=\[ri​j,xi​j,S¯i​j\]⊤∈ℝdline,\\displaystyle\\mathbf\{b\}\_\{ij\}=\[r\_\{ij\},x\_\{ij\},\\overline\{S\}\_\{ij\}\]^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{line\}\}\},\(9b\)wheredlined\_\{\\text\{line\}\}is the number of the line\-level features\.

#### III\-A2PI Adjacency Matrix

We encode the line attributes in𝐁\\mathbf\{B\}into the edge weights of𝐀\\mathbf\{A\}, thereby constructing a physics\-informed \(PI\) adjacency matrix𝐀~\\tilde\{\\mathbf\{A\}\}\. Compared with a binary graph,𝐀~\\tilde\{\\mathbf\{A\}\}preserves topology while capturing the relative electrical coupling strength across connected buses, thus providing a more informative structural prior for physically consistent CBESS coordination\. The PI adjacency matrix𝐀~\\tilde\{\\mathbf\{A\}\}is devised based on Eq\. \([4](https://arxiv.org/html/2610.33654#S2.E4)\) as:

𝐀~=\[A~i​j\]i,j∈𝒩∈ℝ\|𝒩\|×\|𝒩\|,\\displaystyle\\tilde\{\\mathbf\{A\}\}=\[\\tilde\{A\}\_\{ij\}\]\_\{i,j\\in\\mathcal\{N\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{N\}\|\\times\|\\mathcal\{N\}\|\},\(10a\)A~i​j=\{σ⁡\[ψ⁡\(𝐛i​j\)\],if​\(i,j\)∈ℒ,0,otherwise,\\displaystyle\\tilde\{A\}\_\{ij\}=\\begin\{cases\}\\sigma\[\\psi\(\\mathbf\{b\}\_\{ij\}\)\],\\quad\\text\{if \}\(i,j\)\\in\\mathcal\{L\},\\\\ 0,\\quad\\text\{otherwise\},\\end\{cases\}\(10b\)ψ⁡\(𝐛i​j\)=−log⁡\(ε\+ri​j2\+xi​j2\)⏟Impedance↓,ECS↑−νS¯i​j⏟Capacity↑,ECS↑\.\\displaystyle\\psi\(\\mathbf\{b\}\_\{ij\}\)=\\underbrace\{\-\\log\(\\varepsilon\+\\sqrt\{r^\{2\}\_\{ij\}\+x^\{2\}\_\{ij\}\}\)\}\_\{\\text\{Impedance\}\\downarrow\\text\{,ECS\}\\uparrow\}\-\\underbrace\{\\frac\{\\nu\}\{\\overline\{S\}\_\{ij\}\}\}\_\{\\text\{Capacity\}\\uparrow\\text\{,ECS\}\\uparrow\}\.\(10c\)Here,ψ⁡\(𝐛i​j\)\\psi\(\\mathbf\{b\}\_\{ij\}\)is a physically derived coupling score that measures the relative electrical strength of line\(i,j\)\(i,j\)based on its impedance and capacity\. The sigmoid functionσ⁡\(⋅\)\\sigma\(\\cdot\)then maps this score smoothly into a bounded edge weight, preserving the relative coupling strength while maintaining numerical stability\. In \([10a](https://arxiv.org/html/2610.33654#S3.E10.1)\),ε\>0\\varepsilon\>0is a small constant used to avoid numerical instability when the electrical distanceri​j2\+xi​j2\\sqrt\{r\_\{ij\}^\{2\}\+x\_\{ij\}^\{2\}\}approaches zero\. The coefficientν\>0\\nu\>0controls the contribution of the capacity term1S¯i​j\\frac\{1\}\{\\overline\{S\}\_\{ij\}\}in the coupling score\. As a result,ψ⁡\(𝐛i​j\)\\psi\(\\mathbf\{b\}\_\{ij\}\)increases when a line has lower impedance and/or larger capacity, leading to a larger PI edge weight in𝐀~\\tilde\{\\mathbf\{A\}\}\. The encoder therefore takes\(𝐗t−W\+1bus,𝐗t−W\+1price,𝐀~\)\(\\mathbf\{X\}^\{\\text\{bus\}\}\_\{t\-W\+1\},\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1\},\\tilde\{\\mathbf\{A\}\}\)as inputs in a single forward pass\.

#### III\-A3Spatial Processing

We use GCN since it naturally propagates bus\-level signals along electrical couplings encoded in𝐀~\\tilde\{\\mathbf\{A\}\}\. Given bus\-level features𝐗t′∈ℝ\|𝒩\|×dbus\\mathbf\{X\}\_\{t^\{\\prime\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{N\}\|\\times d\_\{\\text\{bus\}\}\}at timet′t^\{\\prime\}, the spatial embedding𝐇t′\(l\)\\mathbf\{H\}^\{\(l\)\}\_\{t^\{\\prime\}\}is computed by GCN layersωGCN\\omega\_\{\\text\{GCN\}\}:

𝐇t′\(l\+1\)=σ⁡\(𝐃~−12​𝐀~​𝐃~−12​𝐇t′\(l\)​ωGCN\(l\)\),𝐇t′\(0\)=𝐗t′,\\displaystyle\\mathbf\{H\}^\{\(l\+1\)\}\_\{t^\{\\prime\}\}=\\sigma\\Big\(\\tilde\{\\mathbf\{D\}\}^\{\-\\frac\{1\}\{2\}\}\\tilde\{\\mathbf\{A\}\}\\tilde\{\\mathbf\{D\}\}^\{\-\\frac\{1\}\{2\}\}\\mathbf\{H\}^\{\(l\)\}\_\{t^\{\\prime\}\}\\omega\_\{\\text\{GCN\}\}^\{\(l\)\}\\Big\),\\,\\mathbf\{H\}^\{\(0\)\}\_\{t^\{\\prime\}\}=\\mathbf\{X\}\_\{t^\{\\prime\}\},\(11a\)𝐃~=\[D~i​i\]i∈𝒩,D~i​i=∑j∈𝒩A~i​j,\\displaystyle\\tilde\{\\mathbf\{D\}\}=\[\\tilde\{D\}\_\{ii\}\]\_\{i\\in\\mathcal\{N\}\},\\quad\\tilde\{D\}\_\{ii\}=\\sum\_\{j\\in\\mathcal\{N\}\}\\tilde\{A\}\_\{ij\},\(11b\)where𝐀~\\tilde\{\\mathbf\{A\}\}is the PI adjacency matrix, and𝐃~\\tilde\{\\mathbf\{D\}\}is the corresponding diagonal degree matrix, whose entriesD~i​i\\tilde\{D\}\_\{ii\}are computed by Eq\. \([11b](https://arxiv.org/html/2610.33654#S3.E11.2)\) as the sum of PI edge weights connected to busii\. The indexlldenotes the layer number ofωGCN\\omega\_\{\\text\{GCN\}\},LLis the total depth, andωGCN\(l\)\\omega^\{\(l\)\}\_\{\\text\{GCN\}\}is the learnable weight matrix at layerll\. This formulation propagates information through physically weighted neighbors and applies nonlinear activationσ⁡\(⋅\)\\sigma\(\\cdot\)\. Its effectiveness stems from smoothing and mixing bus\-level signals along real electrical pathways, thus embedding DN structure directly into bus\-level embedding\.

AfterLLlayers, the bus\-wise spatial embedding𝐇\(L\)\\mathbf\{H\}^\{\(L\)\}is obtained\. To produce a graph\-level summary where more critical buses will be more influential, we apply attention\-based pooling on bus\-wise embeddings:

𝐡t′S=∑i∈𝒩αi,t′​𝐡i,t′\(L\)∈ℝdS,\\displaystyle\\mathbf\{h\}^\{\\text\{S\}\}\_\{t^\{\\prime\}\}=\\sum\_\{i\\in\\mathcal\{N\}\}\\alpha\_\{i,t^\{\\prime\}\}\\mathbf\{h\}^\{\(L\)\}\_\{i,t^\{\\prime\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{S\}\}\},\(12a\)αi,t′=exp⁡\(ωatt​𝐡i,t′\(L\)\)∑jexp⁡\(ωatt​𝐡j,t′\(L\)\),\\displaystyle\\alpha\_\{i,t^\{\\prime\}\}=\\frac\{\\exp\(\\omega\_\{\\text\{att\}\}\\mathbf\{h\}^\{\(L\)\}\_\{i,t^\{\\prime\}\}\)\}\{\\sum\_\{j\}\\exp\(\\omega\_\{\\text\{att\}\}\\mathbf\{h\}^\{\(L\)\}\_\{j,t^\{\\prime\}\}\)\},\(12b\)whereωatt\\omega\_\{\\text\{att\}\}is a trainable vector for attention\-based weighting, deciding which buses are more critical for the final summary \(e\.g\., those near violations\)\. As a result, the encoder focuses on the most influential buses when forming𝐡t′S∈ℝdS\\mathbf\{h\}^\{\\text\{S\}\}\_\{t^\{\\prime\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{S\}\}\}\. Finally, we get the graph\-level summarized spatial embedding𝐇tS\\mathbf\{H\}^\{\\text\{S\}\}\_\{t\}via collecting over the window:

𝐇tS=\[𝐡t−W\+1S;…;𝐡tS\]∈ℝW×dS,\\displaystyle\\mathbf\{H\}^\{\\text\{S\}\}\_\{t\}=\\big\[\\mathbf\{h\}^\{\\text\{S\}\}\_\{t\-W\+1\};\.\.\.;\\mathbf\{h\}^\{\\text\{S\}\}\_\{t\}\\big\]\\in\\mathbb\{R\}^\{W\\times d\_\{\\text\{S\}\}\},\(13\)which compresses the system into a sequence of lengthWWwith embedding dimensiondSd\_\{\\text\{S\}\}\.

#### III\-A4Temporal Processing

To capture temporal dependencies such as fast PV/load fluctuations, TCN, whose causal and dilated convolutions can extract both short\- and long\-term correlations in parallel while avoiding recurrence, is incorporated to process the sequence of graph\-level vectors into spatiotemporal embedding:

𝐡tDN=ωTCNDN​\(𝐇tS\)∈ℝdSTDN\.\\displaystyle\\mathbf\{h\}^\{\\text\{DN\}\}\_\{t\}=\\omega^\{\\text\{DN\}\}\_\{\\text\{TCN\}\}\(\\mathbf\{H\}^\{\\text\{S\}\}\_\{t\}\)\\in\\mathbb\{R\}^\{d^\{\\text\{DN\}\}\_\{\\text\{ST\}\}\}\.\(14\)In addition, the price sequence𝐗pricet−W\+1:t\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1:t\}is processed through a separate TCN encoder to capture its temporal regularities:

𝐡tprice=ωTCNprice\(𝐗t−W\+1:tprice\)∈ℝdTprice\.\\displaystyle\\mathbf\{h\}^\{\\text\{price\}\}\_\{t\}=\\omega^\{\\text\{price\}\}\_\{\\text\{TCN\}\}\\Big\(\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1:t\}\\Big\)\\in\\mathbb\{R\}^\{d^\{\\text\{price\}\}\_\{\\text\{T\}\}\}\.\(15\)The final output is a compressed vector combined by𝐡tDN\\mathbf\{h\}^\{\\text\{DN\}\}\_\{t\}and𝐡tprice\\mathbf\{h\}^\{\\text\{price\}\}\_\{t\}, summarizing both network state and market signals:

𝐡tSTG=\[𝐡tDN,𝐡tprice\]∈ℝdSTDN\+dTprice\.\\displaystyle\\mathbf\{h\}\_\{t\}^\{\\text\{STG\}\}=\[\\mathbf\{h\}^\{\\text\{DN\}\}\_\{t\},\\mathbf\{h\}^\{\\text\{price\}\}\_\{t\}\]\\in\\mathbb\{R\}^\{d^\{\\text\{DN\}\}\_\{\\text\{ST\}\}\+d^\{\\text\{price\}\}\_\{\\text\{T\}\}\}\.\(16\)The resulting joint embedding provides the DRL agent with physical dynamics and external economic incentives\. The entire data flow is parameterized as:

ωPISTG\(𝐗t−W\+1:tbus;𝐗t−W\+1:tprice\)=𝐡tSTG,\\displaystyle\\omega\_\{\\text\{PISTG\}\}\(\\mathbf\{X\}^\{\\text\{bus\}\}\_\{t\-W\+1:t\};\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\-W\+1:t\}\)=\\mathbf\{h\}^\{\\text\{STG\}\}\_\{t\},\(17a\)ωPISTG=\[ωGCN​‖ωatt‖​\[ωTCNDN⊕ωTCNprice\]\],\\displaystyle\\omega\_\{\\text\{PISTG\}\}=\\Big\[\\omega\_\{\\text\{GCN\}\}\|\|\\omega\_\{\\text\{att\}\}\|\|\[\\omega^\{\\text\{DN\}\}\_\{\\text\{TCN\}\}\\oplus\\omega^\{\\text\{price\}\}\_\{\\text\{TCN\}\}\]\\Big\],\(17b\)whereωPISTG\\omega\_\{\\text\{PISTG\}\}is the entire network,\|⁣\|\|\|indicates the stacking between two modules, and⊕\\oplusindicates parallel concatenation\.

#### III\-A5Physics\-Regularized Latent Representation

To enhance the physical consistency of PISTG embeddings, we design a physics\-informed auto\-encoder \(PIAE\) by appending a lightweight reconstruction headωrecon\\omega\_\{\\text\{recon\}\}to the encoderωPISTG\\omega\_\{\\text\{PISTG\}\}\. The PIAE reconstructs key physical variables, i\.e\., branch active/reactive flows and bus voltages\(p^i​j,t,q^i​j,t,V^i,t\)\(\\hat\{p\}\_\{ij,t\},\\hat\{q\}\_\{ij,t\},\\hat\{V\}\_\{i,t\}\):

ωPIAE​\(𝐗tbus,𝐗tprice,𝐀~\)=\(p^i​j,t,q^i​j,t,V^i,t\),\\displaystyle\\omega\_\{\\text\{PIAE\}\}\(\\mathbf\{X\}^\{\\text\{bus\}\}\_\{t\};\\mathbf\{X\}^\{\\text\{price\}\}\_\{t\};\\tilde\{\\mathbf\{A\}\}\)=\(\\hat\{p\}\_\{ij,t\},\\hat\{q\}\_\{ij,t\},\\hat\{V\}\_\{i,t\}\),\(18a\)ωPIAE=\[ωPISTG\|\|ωrecon\],\\displaystyle\\omega\_\{\\text\{PIAE\}\}=\[\\omega\_\{\\text\{PISTG\}\}\|\|\\omega\_\{\\text\{recon\}\}\],\(18b\)so that the learned embeddings align with power\-system laws, improving convergence consistency and downstream decision reliability\.

Since the Kirchhoff’s Current Law and Ohm’s law capture the most fundamental electrical constraints, we incorporate them as physics\-consistency loss terms during training\. The bus\-level active power balance loss guides the PISTG encoder to follow KCL by aligning reconstructed flows with nodal injections and consumptions:

ℒKCL=\\displaystyle\\mathcal\{L\}\_\{\\text\{KCL\}\}=∑i∈𝒩∥∑j:\(j,i\)p^i​j,t\+pPVi,t\+∑m:ξ⁡\(m\)=ipdism,t\\displaystyle\\sum\_\{i\\in\\mathcal\{N\}\}\\Big\\\|\\sum\_\{j:\(j,i\)\}\\hat\{p\}\_\{ij,t\}\+p^\{\\text\{PV\}\}\_\{i,t\}\+\\sum\_\{m:\\xi\(m\)=i\}p^\{\\text\{dis\}\}\_\{m,t\}−\(pi,tload\+∑j:\(i,j\)p^i​j,t\+∑m:ξ⁡\(m\)=ipm,tch\)∥22\.\\displaystyle\-\\Big\(p^\{\\text\{load\}\}\_\{i,t\}\+\\sum\_\{j:\(i,j\)\}\\hat\{p\}\_\{ij,t\}\+\\sum\_\{m:\\xi\(m\)=i\}p^\{\\text\{ch\}\}\_\{m,t\}\\Big\)\\Big\\\|\_\{2\}^\{2\}\.\(19\)The line\-level voltage consistency term guides PISTG encoder to follow Ohm’s law by constraining voltage differences across connected buses:

ℒVDrop=\\displaystyle\\mathcal\{L\}\_\{\\text\{VDrop\}\}=∑\(i,j\)∈ℒ\|V^j,t2−V^i,t2\+2​\(ri​j​p^i​j,t\+xi​j​q^i​j,t\)\\displaystyle\\sum\_\{\(i,j\)\\in\\mathcal\{L\}\}\\Big\\\|\\hat\{V\}^\{2\}\_\{j,t\}\-\\hat\{V\}^\{2\}\_\{i,t\}\+2\\left\(r\_\{ij\}\\hat\{p\}\_\{ij,t\}\+x\_\{ij\}\\hat\{q\}\_\{ij,t\}\\right\)−\(ri​j2\+xi​j2\)ℓ^i​j,t∥22,\\displaystyle\-\\left\(r\_\{ij\}^\{2\}\+x\_\{ij\}^\{2\}\\right\)\\hat\{\\ell\}\_\{ij,t\}\\Big\\\|\_\{2\}^\{2\},\(20a\)ℓ^i​j,t=p^i​j,t2\+q^i​j,t2V^i,t2\.\\displaystyle\\hat\{\\ell\}\_\{ij,t\}=\\frac\{\\hat\{p\}\_\{ij,t\}^\{2\}\+\\hat\{q\}\_\{ij,t\}^\{2\}\}\{\\hat\{V\}^\{2\}\_\{i,t\}\}\.\(20b\)
Moreover, for regularization, a reconstruction loss term validates the accuracy of the PISTG embedding\. Specifically, we guide the reconstructed voltages and branch flows to match their true values by applying an MSE penalty on the predicted bus voltage, branch active power, and branch reactive power:

ℒrecon=∑i∈𝒩\|V^i,t−Vi,t\|22\+\\displaystyle\\mathcal\{L\}\_\{\\text\{recon\}\}=\\sum\_\{i\\in\\mathcal\{N\}\}\\big\|\\hat\{V\}\_\{i,t\}\-V\_\{i,t\}\\big\|\_\{2\}^\{2\}\+∑\(i,j\)∈ℒ\(\|p^i​j,t−pi​j,t\|22\+\|​q^i​j,t−qi​j,t\|22\)\.\\displaystyle\\sum\_\{\(i,j\)\\in\\mathcal\{L\}\}\\Big\(\\big\|\\hat\{p\}\_\{ij,t\}\-p\_\{ij,t\}\\big\|\_\{2\}^\{2\}\+\\big\|\\hat\{q\}\_\{ij,t\}\-q\_\{ij,t\}\\big\|\_\{2\}^\{2\}\\Big\)\.\(21\)This reconstruction loss promotes that latent representations produced by the PISTG encoder preserve sufficient physical information to faithfully recover voltage magnitudes and power\-flow patterns, thereby improving embedding fidelity and stabilizing downstream decision\-making\.

The overall objective jointly minimizes these three terms:

minωPIAE⁡\[β1​ℒKCL​\(ωPIAE\)\+β2​ℒVDrop​\(ωPIAE\)\+β3​ℒrecon​\(ωPIAE\)\],\\displaystyle\\min\_\{\\omega\_\{\\text\{PIAE\}\}\}\\big\[\\beta\_\{1\}\\mathcal\{L\}\_\{\\text\{KCL\}\}\(\\omega\_\{\\text\{PIAE\}\}\)\+\\beta\_\{2\}\\mathcal\{L\}\_\{\\text\{VDrop\}\}\(\\omega\_\{\\text\{PIAE\}\}\)\+\\beta\_\{3\}\\mathcal\{L\}\_\{\\text\{recon\}\}\(\\omega\_\{\\text\{PIAE\}\}\)\\big\],\(22\)whereβ1,β2,\\beta\_\{1\},\\beta\_\{2\},andβ3\\beta\_\{3\}are weights\. Through this, the PISTG encoder learns both compact and physically consistent embeddings\.

Overall, the PI adjacencyA~\\tilde\{A\}and the losses in Eq\. \([22](https://arxiv.org/html/2610.33654#S3.E22)\) provide structural priors and soft training regularization for representation learning, rather than a hard power\-flow feasibility layer during policy execution\.

### III\-BCMDP Modeling for Problem

The coordination of CBESS units in a DN with high DER penetration can be formulated as a CMDP, where the DSO seeks to minimize operating cost while explicitly accounting for network constraints\. Formally, the CMDP is defined as:

𝕄=⟨𝒮,𝒜,ℙ,ℛ,𝒞,γ⟩,\\displaystyle\\mathbb\{M\}=\\langle\\mathcal\{S\},\\mathcal\{A\},\\mathbb\{P\},\\mathcal\{R\},\\mathcal\{C\},\\gamma\\rangle,\(23\)where𝒮\\mathcal\{S\}denotes system states embedding DN and CBESS information,𝒜\\mathcal\{A\}denotes the joint actions of CBESSs and inverters,ℙ\\mathbb\{P\}denotes the transition governed by power\-flow dynamics,ℛ\\mathcal\{R\}denotes the economic\-oriented reward,𝒞\\mathcal\{C\}denotes the operational\-constraint costs, andγ∈\(0,1\)\\gamma\\in\(0,1\)denotes discount factor\.

#### III\-B1CMDP Elements

We formulate the CBESS coordination problem as a CMDP\. At each time step, the agent observes a compact spatiotemporal state, executes coordinated active/reactive power decisions, and receives an economic reward together with multiple constraint\-specific cost signals\. The key components are defined as:

𝐬t=𝐡tSTG∈𝒮,\\displaystyle\\mathbf\{s\}\_\{t\}=\\mathbf\{h\}^\{\\text\{STG\}\}\_\{t\}\\in\\mathcal\{S\},\(24a\)𝐚t=\[\(pm,tCB,qm,tCB\)m∈ℳ,\(κi,t,qi,tinv\)i∈𝒩DER\]∈𝒜,\\displaystyle\\mathbf\{a\}\_\{t\}=\[\(p^\{\\text\{CB\}\}\_\{m,t\},q^\{\\text\{CB\}\}\_\{m,t\}\)\_\{m\\in\\mathcal\{M\}\},\(\\kappa\_\{i,t\},q^\{\\text\{inv\}\}\_\{i,t\}\)\_\{i\\in\\mathcal\{N\}^\{\\text\{DER\}\}\}\]\\in\\mathcal\{A\},\(24b\)rt=−φtwh​\[∑i∈𝒩\(pi,tload−pi,tPV\)−∑m∈ℳpm,tCB\]​Δ​t,\\displaystyle r\_\{t\}=\-\\varphi^\{\\text\{wh\}\}\_\{t\}\\Big\[\\sum\_\{i\\in\\mathcal\{N\}\}\(p^\{\\text\{load\}\}\_\{i,t\}\-p^\{\\text\{PV\}\}\_\{i,t\}\)\-\\sum\_\{m\\in\\mathcal\{M\}\}p^\{\\text\{CB\}\}\_\{m,t\}\\Big\]\\Delta t,−φdeg∑m∈ℳ\(\|pm,tCB\|\)Δt,\\displaystyle\\quad\-\\varphi^\{\\text\{deg\}\}\\sum\_\{m\\in\\mathcal\{M\}\}\\big\(\|p^\{\\text\{CB\}\}\_\{m,t\}\|\\big\)\\Delta t,\(24c\)ctVD=∑i∈𝒩\[\(Vi,t−V¯\)\+\+\(V¯−Vi,t\)\+\],\\displaystyle c^\{\\text\{VD\}\}\_\{t\}=\\sum\_\{i\\in\\mathcal\{N\}\}\\Big\[\(V\_\{i,t\}\-\\overline\{V\}\)^\{\+\}\+\(\\underline\{V\}\-V\_\{i,t\}\)^\{\+\}\\Big\],\(24d\)ctVN=∑i∈𝒩𝕀⁡\[\(Vi,t\>V¯\)∨\(Vi,t<V¯\)\],\\displaystyle c^\{\\text\{VN\}\}\_\{t\}=\\sum\_\{i\\in\\mathcal\{N\}\}\\mathbb\{I\}\\Big\[\(V\_\{i,t\}\>\\overline\{V\}\)\\;\\lor\\;\(V\_\{i,t\}<\\underline\{V\}\)\\Big\],\(24e\)ctLO=∑\(i,j\)∈ℒ\[pi​j,t2\+qi​j,t2−S¯i​j\]\+\.\\displaystyle c^\{\\text\{LO\}\}\_\{t\}=\\sum\_\{\(i,j\)\\in\\mathcal\{L\}\}\\Big\[\\sqrt\{p\_\{ij,t\}^\{2\}\+q\_\{ij,t\}^\{2\}\}\-\\overline\{S\}\_\{ij\}\\Big\]^\{\+\}\.\(24f\)In Eq\. \([24a](https://arxiv.org/html/2610.33654#S3.E24.1)\), at timett, the compact spatiotemporal state𝐬t=𝐡tSTG\\mathbf\{s\}\_\{t\}=\\mathbf\{h\}^\{\\text\{STG\}\}\_\{t\}encodes network\- and device\-level variables such as demands, PV generation, voltages, SOC, and market prices\. In Eq\. \([24b](https://arxiv.org/html/2610.33654#S3.E24.2)\), the action vector𝐚t∈𝒜\\mathbf\{a\}\_\{t\}\\in\\mathcal\{A\}comprises the activepm,tCBp^\{\\text\{CB\}\}\_\{m,t\}and reactive powerqm,tCBq^\{\\text\{CB\}\}\_\{m,t\}of each CBESS, as well as the PV curtailment ratioκi,t\\kappa\_\{i,t\}and reactive powerqi,tinvq^\{\\text\{inv\}\}\_\{i,t\}of bus\-level inverters on DER\-rich buses\. For each CBESS, the actor outputs normalized variablesαm,tP,αm,tQ∈\[−1,1\]\\alpha^\{P\}\_\{m,t\},\\alpha^\{Q\}\_\{m,t\}\\in\[\-1,1\], which are mapped to feasible active and reactive power commands before power\-flow evaluation:

pm,tCB\\displaystyle p^\{\\text\{CB\}\}\_\{m,t\}=P¯m,tdis−P¯m,tch2\+P¯m,tdis\+P¯m,tch2​αm,tP,\\displaystyle=\\frac\{\\overline\{P\}^\{\\text\{dis\}\}\_\{m,t\}\-\\overline\{P\}^\{\\text\{ch\}\}\_\{m,t\}\}\{2\}\+\\frac\{\\overline\{P\}^\{\\text\{dis\}\}\_\{m,t\}\+\\overline\{P\}^\{\\text\{ch\}\}\_\{m,t\}\}\{2\}\\alpha^\{P\}\_\{m,t\},\(25\)qm,tCB\\displaystyle q^\{\\text\{CB\}\}\_\{m,t\}=αm,tQ​max⁡\{\(SmCB\)2−\(pm,tCB\)2,0\}\.\\displaystyle=\\alpha^\{Q\}\_\{m,t\}\\sqrt\{\\max\\left\\\{\(S^\{\\text\{CB\}\}\_\{m\}\)^\{2\}\-\(p^\{\\text\{CB\}\}\_\{m,t\}\)^\{2\},0\\right\\\}\}\.\(26\)Here,P¯m,tch\\overline\{P\}^\{\\text\{ch\}\}\_\{m,t\}andP¯m,tdis\\overline\{P\}^\{\\text\{dis\}\}\_\{m,t\}denote the SOC\-aware charging and discharging limits, respectively, andpm,tCB\>0p^\{\\text\{CB\}\}\_\{m,t\}\>0denotes discharging\. This mapping enforces charging–discharging exclusivity and satisfies Eq\. \([1e](https://arxiv.org/html/2610.33654#S2.E1.5)\) by construction\[[38](https://arxiv.org/html/2610.33654#bib.bib42)\]\. In Eq\. \([24c](https://arxiv.org/html/2610.33654#S3.E24.3)\), the rewardrtr\_\{t\}represents the DSO’s economic objective of minimizing operating cost\. It consists of two components: \(i\) the energy cost, computed from the net active power and the time\-varying wholesale electricity priceφtwh\\varphi^\{\\text\{wh\}\}\_\{t\}, and \(ii\) a battery degradation cost that penalizes charging and discharging throughput using a per\-unit degradation coefficientφdeg\\varphi^\{\\text\{deg\}\}\. In terms of cost design, we introduce separate operational\-constraint cost functions\. As for the voltage\-related cost terms, we consider both voltage\-violation degree \(VD\) and voltage\-violation count \(VN\) to stabilize learning under rare\-but\-large events, since VD captures severity, while VN captures spatial frequency\. In Eq\. \([24d](https://arxiv.org/html/2610.33654#S3.E24.4)\), the termctVDc^\{\\text\{VD\}\}\_\{t\}quantifies the aggregated magnitude of voltage deviations beyond admissible bounds across all buses\. In Eq\. \([24e](https://arxiv.org/html/2610.33654#S3.E24.5)\), the termctVNc^\{\\text\{VN\}\}\_\{t\}counts the number of buses whose voltage violates prescribed limits at timett\. In Eq\. \([24f](https://arxiv.org/html/2610.33654#S3.E24.6)\), the termctLOc^\{\\text\{LO\}\}\_\{t\}measures the extent of apparent power exceedance over line load limits for all distribution lines\.

#### III\-B2CMDP Formulation

Given a policyπθ​\(𝐚\|𝐬\)\\pi\_\{\\theta\}\(\\mathbf\{a\}\|\\mathbf\{s\}\), and denote the trajectoryτt=\(𝐬t,𝐚t,rt,\[ctk\]k∈𝒦\)\\tau\_\{t\}=\\Big\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}\_\{t\},r\_\{t\},\[c^\{k\}\_\{t\}\]\_\{k\\in\\mathcal\{K\}\}\\Big\), where𝒦=\{VD,VN,LO\}\\mathcal\{K\}=\\\{\\text\{VD\},\\text\{VN\},\\text\{LO\}\\\}contains the terms of the constraints\. We write the discounted return of the reward and costs as:

JR​\(πθ\)≜𝔼τ∼πθ​\[∑t=0T−1γt​rt\],\\displaystyle J^\{R\}\(\\pi\_\{\\theta\}\)\\triangleq\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\Big\[\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{t\}\\,r\_\{t\}\\Big\],\(27\)JkC​\(πθ\)≜𝔼τ∼πθ​\[∑t=0T−1γt​ctk\],k∈𝒦\.\\displaystyle J^\{C\}\_\{k\}\(\\pi\_\{\\theta\}\)\\triangleq\\mathbb\{E\}\_\{\\tau\\sim\{\\pi\_\{\\theta\}\}\}\\Big\[\\sum\_\{t=0\}^\{T\-1\}\\gamma^\{t\}\\,c^\{k\}\_\{t\}\\Big\],\\quad k\\in\\mathcal\{K\}\.\(28\)
Then the formulation can be written as:

maxπθ⁡JR​\(πθ\),s\.t\.​JkC​\(πθ\)≤dk,k∈𝒦,\\displaystyle\\max\_\{\\pi\_\{\\theta\}\}\\,J^\{R\}\(\\pi\_\{\\theta\}\),\\quad\\text\{s\.t\.\}\\,J^\{C\}\_\{k\}\(\\pi\_\{\\theta\}\)\\leq d\_\{k\},\\,k\\in\\mathcal\{K\},\(29\)wheredkd\_\{k\}is the expected discounted\-cost budget for constraintkk, used to regulate its Lagrangian multiplier and penalize exceedance of the corresponding budget\. Throughout this paper, safe RL refers to CMDP\-based constraint\-aware policy learning under expected discounted cost budgets; it does not imply pointwise feasibility guarantees for every action or operating realization\.

### III\-CConstraint\-Specific Value Decomposition with LDCPPO

We develop LDCPPO for CBESS coordination to optimize economic return subject to expected constraint\-cost budgets\. LDCPPO implements constraint\-specific value decomposition through independent reward and cost critics, preserving distinct learning signals for heterogeneous voltage and line\-loading risks\. Adaptive Lagrangian multipliers combine these signals during actor optimization, while a shared PISTG encoder provides a common physically informed state representation for the lightweight actor and critic heads\.

#### III\-C1Independent Reward and Cost Critics

Different from a mixed\-critic Lagrangian design, LDCPPO estimates the reward return and each physical cost return with separate critics, and combines their advantages only during the actor update\[[39](https://arxiv.org/html/2610.33654#bib.bib5)\]\. This keeps the critic targets independent of the adaptive Lagrangian multipliers and preserves constraint\-specific learning signals for voltage and line\-loading control\. We employ four separate critic networks: one for the reward and three for the costs, parameterized byϕR\\phi^\{R\}and\{ϕk\}k∈𝒦\\\{\\phi^\{k\}\\\}\_\{k\\in\\mathcal\{K\}\}, wherekkcorresponds to the two voltage violation penalties and line\-overload cost, respectively\. Each critic estimates a state\-value function from𝐬t\\mathbf\{s\}\_\{t\}:VϕR​\(𝐬t\)V\_\{\\phi^\{R\}\}\(\\mathbf\{s\}\_\{t\}\)for the reward andVϕk​\(𝐬t\)V\_\{\\phi^\{k\}\}\(\\mathbf\{s\}\_\{t\}\)for costk∈𝒦k\\in\\mathcal\{K\}\.

##### GAE targets and advantages

Generalized advantage estimation \(GAE\) is a popular way to improve PPO learning stability\. First, we calculate temporal difference as:

δtR\\displaystyle\\delta\_\{t\}^\{R\}=rt\+γ​VϕR​\(𝐬t\+1\)−VϕR​\(𝐬t\),\\displaystyle=r\_\{t\}\+\\gamma\\,V\_\{\\phi^\{R\}\}\(\\mathbf\{s\}\_\{t\+1\}\)\-V\_\{\\phi^\{R\}\}\(\\mathbf\{s\}\_\{t\}\),\(30a\)δtk\\displaystyle\\delta\_\{t\}^\{k\}=ctk\+γ​Vϕk​\(𝐬t\+1\)−Vϕk​\(𝐬t\),k∈𝒦\.\\displaystyle=c^\{k\}\_\{t\}\+\\gamma\\,V\_\{\\phi^\{k\}\}\(\\mathbf\{s\}\_\{t\+1\}\)\-V\_\{\\phi^\{k\}\}\(\\mathbf\{s\}\_\{t\}\),\\,k\\in\\mathcal\{K\}\.\(30b\)Next, we compute generalized advantagesAtRA^\{R\}\_\{t\}andAtkA^\{k\}\_\{t\}\(withλ∈\[0,1\]\\lambda\\\!\\in\\\!\[0,1\]\) over truncated rollouts:

AtR\\displaystyle A^\{R\}\_\{t\}=ζ⁡\[∑l=0Lt−1\(γ​λ\)l​δt\+lR\],\\displaystyle=\\zeta\\\!\\left\[\\sum\_\{l=0\}^\{L\_\{t\}\-1\}\(\\gamma\\lambda\)^\{l\}\\,\\delta\_\{t\+l\}^\{R\}\\right\],\(31a\)Atk\\displaystyle A^\{k\}\_\{t\}=ζ⁡\[∑l=0Lt−1\(γ​λ\)l​δt\+lk\],k∈𝒦,\\displaystyle=\\zeta\\\!\\left\[\\sum\_\{l=0\}^\{L\_\{t\}\-1\}\(\\gamma\\lambda\)^\{l\}\\,\\delta^\{k\}\_\{t\+l\}\\right\],\\quad k\\in\\mathcal\{K\},\(31b\)wherellis the dummy summation index \(offset fromtt\),LtL\_\{t\}is the look\-ahead length fromtt, andζ⁡\(⋅\)\\zeta\(\\cdot\)standardizes the advantages to zero mean and unit variance within the batch\. Accordingly, the corresponding value\-regression targets:

R^t=AtR\+VϕR​\(𝐬t\),\\displaystyle\\hat\{R\}\_\{t\}=A^\{R\}\_\{t\}\+V\_\{\\phi^\{R\}\}\(\\mathbf\{s\}\_\{t\}\),\(32a\)C^k,t=Atk\+Vϕk​\(𝐬t\),k∈𝒦\.\\displaystyle\\hat\{C\}\_\{k,t\}=A^\{k\}\_\{t\}\+V\_\{\\phi^\{k\}\}\(\\mathbf\{s\}\_\{t\}\),\\quad k\\in\\mathcal\{K\}\.\(32b\)

##### Critic losses

Given trajectories sampled from the bufferℬ\\mathcal\{B\}, we aim to minimize:

ℒR​\(ϕR\)\\displaystyle\\mathcal\{L\}^\{R\}\(\\phi^\{R\}\)=𝔼τ∼ℬ​\[\(VϕR​\(𝐬t\)−R^t\)2\],\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{B\}\}\\\!\\left\[\\big\(V\_\{\\phi^\{R\}\}\(\\mathbf\{s\}\_\{t\}\)\-\\hat\{R\}\_\{t\}\\big\)^\{2\}\\right\],\(33a\)ℒk​\(ϕk\)\\displaystyle\\mathcal\{L\}^\{k\}\(\\phi^\{k\}\)=𝔼τ∼ℬ​\[\(Vϕk​\(𝐬t\)−C^k,t\)2\],k∈𝒦\.\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{B\}\}\\\!\\left\[\\big\(V\_\{\\phi^\{k\}\}\(\\mathbf\{s\}\_\{t\}\)\-\\hat\{C\}\_\{k,t\}\\big\)^\{2\}\\right\],\\quad k\\in\\mathcal\{K\}\.\(33b\)

#### III\-C2Actor Network

Actions are sampled from the policyπθ\\pi\_\{\\theta\}:

𝐚t∼πθ​\(𝐚t∣𝐬t\)\.\\displaystyle\\mathbf\{a\}\_\{t\}\\sim\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{t\}\\mid\\mathbf\{s\}\_\{t\}\)\.\(34\)To update the actor network, we need to maximize the PPO clipped surrogate:

JPPO\(θ\)=𝔼τ∼ℬ\[min\(ρt\(θ\)AtL,\\displaystyle J^\{\\text\{PPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\mathcal\{B\}\}\\left\[\\min\\Big\(\\rho\_\{t\}\(\\theta\)A^\{L\}\_\{t\},\\right\.clip\(ρt\(θ\),1−ϵ,1\+ϵ\)AtL\)\]\.\\displaystyle\\left\.\\qquad\\qquad\\text\{clip\}\(\\rho\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\,A^\{L\}\_\{t\}\\Big\)\\right\]\.\(35a\)ρt​\(θ\)=πθ​\(𝐚t∣𝐬t\)πθold​\(𝐚t∣𝐬t\),\\displaystyle\\rho\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(\\mathbf\{a\}\_\{t\}\\mid\\mathbf\{s\}\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\mathbf\{a\}\_\{t\}\\mid\\mathbf\{s\}\_\{t\}\)\},\(35b\)AtL≜AtR−∑k∈𝒦λk​Atk,\\displaystyle A^\{L\}\_\{t\}\\triangleq A^\{R\}\_\{t\}\-\\sum\_\{k\\in\\mathcal\{K\}\}\\lambda\_\{k\}\\,A^\{k\}\_\{t\},\(35c\)whereρt​\(θ\)\\rho\_\{t\}\(\\theta\)is the importance ratio,AtLA^\{L\}\_\{t\}is the Lagrangian advantage computed by summing the reward GAE and cost GAEs using Lagrange multipliers\{λk\}k∈𝒦\\\{\\lambda\_\{k\}\\\}\_\{k\\in\\mathcal\{K\}\}\.

#### III\-C3Parameter Update

For each episode or mini\-batch, perform gradient descent for critics/actor and projected dual ascent for multipliers\. Additionally, the Lagrangian multipliers\{λk\}k∈𝒦\\\{\\lambda^\{k\}\\\}\_\{k\\in\\mathcal\{K\}\}are adaptively updated through dual ascent to regulate the expected constraint\-cost budgets, maintaining their non\-negativity and adjusting based on the deviation of constraint values from their respective thresholdsdkd^\{k\}:

ϕR\\displaystyle\\phi^\{R\}←ϕR−ηR​∇ϕRℒR,\\displaystyle\\leftarrow\\phi^\{R\}\-\\eta^\{R\}\\nabla\_\{\\phi^\{R\}\}\\,\\mathcal\{L\}^\{R\},\(36a\)ϕk\\displaystyle\\phi^\{k\}←ϕk−ηk​∇ϕkℒk,k∈𝒦,\\displaystyle\\leftarrow\\phi^\{k\}\-\\eta^\{k\}\\nabla\_\{\\phi^\{k\}\}\\,\\mathcal\{L\}^\{k\},\\quad k\\in\\mathcal\{K\},\(36b\)θ\\displaystyle\\theta←θ−ηAct​∇θℒθ,\\displaystyle\\leftarrow\\theta\-\\eta^\{\\text\{Act\}\}\\nabla\_\{\\theta\}\\,\\mathcal\{L\}\_\{\\theta\},\(36c\)λk\\displaystyle\\lambda^\{k\}←\[λk\+ηLag​\(J^kC−dk\)\]\+,∀k∈𝒦,\\displaystyle\\leftarrow\[\\lambda^\{k\}\+\\eta^\{\\text\{Lag\}\}\(\\hat\{J\}^\{C\}\_\{k\}\-d^\{k\}\)\]^\{\+\},\\forall k\\in\\mathcal\{K\},\(36d\)where∇\\nablais the gradient descent operation, andηR\\eta^\{\\text\{R\}\},\{ηk\}k∈𝒦\\\{\\eta^\{k\}\\\}\_\{k\\in\\mathcal\{K\}\},ηAct\\eta^\{\\text\{Act\}\},ηLag\\eta^\{\\text\{Lag\}\}are the learning rates for different updates\. Note thatJ^kC\\hat\{J\}^\{C\}\_\{k\}is the minibatch mean of discounted cost returnsJkCJ^\{C\}\_\{k\}in Eq\. \([28](https://arxiv.org/html/2610.33654#S3.E28)\), and\[⋅\]\+\[\\cdot\]^\{\+\}enforcesλk≥0\\lambda^\{k\}\\geq 0\.

![Refer to caption](https://arxiv.org/html/2610.33654v1/bus_33.png)Fig\. 4:CBESS deployment in the IEEE 33\-bus system with high DER penetration\.\(a\)Daily PV and load profile\(b\)Aggregated energy scale comparison between one bus and one CBESS
Fig\. 5:System setup of PV, EV, load, and CBESSs\.

## IVSimulations

### IV\-AExperimental Setup

TABLE II:Hyperparameter and network architecture settings\.CategoryParameterValuePISTG encoderPI adjacency constant,ϵ\\epsilon10−610^\{\-6\}Capacity\-weight coefficient,ν\\nu1\.0GCN layers,LL2GCN hidden dimension64Spatial embedding dim\.dSd^\{S\}64DN\-TCN layers2Price\-TCN layers2TCN kernel size3TCN dilations\{1,2\}\\\{1,2\\\}Joint embedding dim\.dSTGd^\{\\mathrm\{STG\}\}80PISTG pre\-trainingPIAE decoder hidden layers\[128,128\]\[128,128\]KCL loss weight,β1\\beta\_\{1\}1\.0Voltage\-drop loss weight,β2\\beta\_\{2\}1\.0Reconstruction loss weight,β3\\beta\_\{3\}0\.5PIAE learning rate1×10−31\\times 10^\{\-3\}LDCPPOActor hidden layers\[128,128\]\[128,128\]Critic hidden layers\[128,128\]\[128,128\]Discount factor,γ\\gamma0\.99GAE parameter,λGAE\\lambda\_\{\\mathrm\{GAE\}\}0\.95PPO clip range,ϵclip\\epsilon\_\{\\mathrm\{clip\}\}0\.2Target KL0\.015Actor learning rate,ηAct\\eta^\{\\mathrm\{Act\}\}3×10−53\\times 10^\{\-5\}Critic learning rate,ηR,ηk\\eta^\{R\},\\eta^\{k\}3×10−53\\times 10^\{\-5\}Dual learning rate,ηLag\\eta^\{\\mathrm\{Lag\}\}5×10−45\\times 10^\{\-4\}Initial multipliers,λ0k\\lambda^\{k\}\_\{0\}0#### IV\-A1Simulator Setup

The case study is based on the IEEE 33\-bus system shown in Fig\.[4](https://arxiv.org/html/2610.33654#S3.F4)\. The slack bus fixed at 1 p\.u\., and bus voltage threshold fixed within \[0\.95,1\.05\] p\.u\. Four CBESS units are placed at buses 6, 11, 16, and 31, each of a capacity of 1000 kWh, an initial SOC of 50%, a charge/discharge rate limit of \[\-250,250\] kW, a reactive power range of \[\-50,50\] kVAr, charge/discharge efficiency of 0\.95, and battery degradation cost of 0\.03 AUD per kWh\[[40](https://arxiv.org/html/2610.33654#bib.bib15)\]\. The operation horizon is one day with 288 steps \(5 min each\)\. Accordingly, the actions in Eq\. \([24b](https://arxiv.org/html/2610.33654#S3.E24.2)\) are updated every 5 min and held until the next step, while faster intra\-interval variations are assumed to be handled by local inverter\-based Volt–VAR control and existing protection functions, and are not explicitly captured by the 5\-min constraint evaluation\.

We set DER buses around these CBESSs\. We construct the 5\-min PV generation and load profiles on each bus by linear interpolation of the Solar Home Electricity Data provided by Ausgrid\[[41](https://arxiv.org/html/2610.33654#bib.bib11)\], which comprises three years of half\-hourly electricity data for 300 randomly selected solar homes\. Moreover, to simulate household consumption peak, we add residential EV charging power for each DER bus over night \(8:00 PM to 6:00 AM\)\. The profiles are demonstrated in Fig\.[5](https://arxiv.org/html/2610.33654#S3.F5)\.

The main hyperparameter and network architecture settings of the PISTG encoder, PISTG pre\-training, and LDCPPO are summarized in Table[II](https://arxiv.org/html/2610.33654#S4.T2)\. The data scale is also specified to improve reproducibility\. With a 5\-min control interval, each daily episode containsT=288T=288decision steps, and 2000 training episodes provide approximately5\.76×1055\.76\\times 10^\{5\}state–action–reward–cost transition samples\. The rolling window lengthW=12W=12used by the PISTG encoder further gives about5\.54×1055\.54\\times 10^\{5\}graph\-temporal training windows\.

#### IV\-A2Comparison Baselines & Computational details

We compare our method with seven DRL baselines: PPO, TD3, SAC, LDCPPO, STG\-LDCPPO, PISTG\-CPO, and PI\-GR\-STG\-LDCPPO\. PPO, TD3, and SAC are classical DRL algorithms widely adopted in continuous control tasks\. LDCPPO and STG\-LDCPPO are included as closely related constrained baselines to isolate the effects of Lagrangian\-based constraint handling, graph\-based state encoding, and physics\-informed training within the proposed framework\. PISTG\-CPO combines the proposed PISTG encoder with a classical constrained policy optimization update\[[42](https://arxiv.org/html/2610.33654#bib.bib43),[28](https://arxiv.org/html/2610.33654#bib.bib44)\], while PI\-GR\-STG\-LDCPPO adapts the physics\-informed graphical representation design\[[27](https://arxiv.org/html/2610.33654#bib.bib29)\]to the same STG\-LDCPPO framework\. For PPO, TD3, and SAC, constraint costs are aggregated into the reward via weighted penalties, resulting in a single scalar objective\. In contrast, LDCPPO, STG\-LDCPPO, PI\-GR\-STG\-LDCPPO, and the proposed method adopt a Lagrangian relaxation framework with constraint\-specific critics, while PISTG\-CPO represents a classical constrained policy optimization baseline\.

Notably, optimization\-based approaches were not included as baselines in this work, which was a choice of scope, which focuses on improving utilities while satisfying multiple constraints by learning\-based policies, rather than a missing consideration\. Also, related work\[[43](https://arxiv.org/html/2610.33654#bib.bib41),[44](https://arxiv.org/html/2610.33654#bib.bib40)\]has shown learning\-based methods can remain competitive while offering much faster online control, showing the promise of RL methods\. Our work aims to evaluate the capability of dealing with key challenges in learning\-based approaches by contributing to novel physics\-informed representation and constraint\-specific critics, which have been demonstrated to be effective\.

Training uses a learning rate of 3e\-5, clipping of 0\.2, target KL of 0\.015, value coefficient of 0\.5, zero entropy bonus, gradient norm clipping of 0\.5, and Gaussian log\-std initialized at \-0\.5, withγ\\gammaset to 0\.99,λk\\lambda^\{k\}set to 0\.95, dual learning rate of 5e\-4, over 2000 episodes of 288 steps \(5\-min resolution\)\. All data collection processes and training are conducted on a server with 8 NVIDIA GeForce 3090 GPUs, each of which has 24G memory\. The CUDA version is 11\.5\.

### IV\-BResult Analysis

![Refer to caption](https://arxiv.org/html/2610.33654v1/train_curves_33bus_2x2.png)Fig\. 6:Training performance on the four key metrics of the compared methods on the IEEE 33\-bus system\. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs\.TABLE III:Test performance comparison over 100 episodes for 5 runs\.MethodOperating costVolt\. vio\. degreeVolt\. vio\. countLine\-overload degreeAUD/dayp\.u\./\(5CLOSE\(5min⋅\\cdotbus\)\)times/\(5CLOSE\(5min⋅\\cdotbus\)\)%/\(5CLOSE\(5min⋅\\cdotline\)\)TD31583\.7±\\pm68\.4\(1\.63±\\pm0\.27\)×10−3\\times 10^\{\-3\}\(1\.02±\\pm0\.16\)×10−2\\times 10^\{\-2\}\(1\.93±\\pm0\.39\)×10−3\\times 10^\{\-3\}SAC1546\.2±\\pm69\.1\(1\.52±\\pm0\.23\)×10−3\\times 10^\{\-3\}\(0\.94±\\pm0\.14\)×10−2\\times 10^\{\-2\}\(1\.82±\\pm0\.36\)×10−3\\times 10^\{\-3\}PPO1424\.5±53\.71424\.5\\pm 53\.7\(1\.28±0\.19\)×10−3\(1\.28\\pm 0\.19\)\\times 10^\{\-3\}\(0\.87±0\.11\)×10−2\(0\.87\\pm 0\.11\)\\times 10^\{\-2\}\(1\.54±0\.28\)×10−3\(1\.54\\pm 0\.28\)\\times 10^\{\-3\}LDCPPO1453\.1±49\.81453\.1\\pm 49\.8\(0\.77±0\.15\)×10−3\(0\.77\\pm 0\.15\)\\times 10^\{\-3\}\(0\.79±0\.10\)×10−2\(0\.79\\pm 0\.10\)\\times 10^\{\-2\}\(1\.32±0\.21\)×10−3\(1\.32\\pm 0\.21\)\\times 10^\{\-3\}STG\-LDCPPO1414\.8±36\.11414\.8\\pm 36\.1\(0\.53±0\.09\)×10−3\(0\.53\\pm 0\.09\)\\times 10^\{\-3\}\(0\.63±0\.06\)×10−2\(0\.63\\pm 0\.06\)\\times 10^\{\-2\}\(1\.15±0\.14\)×10−3\(1\.15\\pm 0\.14\)\\times 10^\{\-3\}PISTG\-CPO1420\.0±37\.41420\.0\\pm 37\.4\(0\.70±0\.08\)×10−3\(0\.70\\pm 0\.08\)\\times 10^\{\-3\}\(0\.63±0\.05\)×10−2\(0\.63\\pm 0\.05\)\\times 10^\{\-2\}\(1\.35±0\.16\)×10−3\(1\.35\\pm 0\.16\)\\times 10^\{\-3\}PI\-GR\-STG\-LDCPPO1412\.3±42\.11412\.3\\pm 42\.1\(0\.49±0\.12\)×10−3\(0\.49\\pm 0\.12\)\\times 10^\{\-3\}\(0\.52±0\.12\)×10−2\(0\.52\\pm 0\.12\)\\times 10^\{\-2\}\(1\.14±0\.19\)×10−3\(1\.14\\pm 0\.19\)\\times 10^\{\-3\}PISTG\-LDCPPO1382\.4±38\.9\\textbf\{1382\.4\}\\pm\\textbf\{38\.9\}\(0\.25±0\.04\)×10−3\\textbf\{\(0\.25\}\\pm\\textbf\{0\.04\)\}\\times 10^\{\-3\}\(0\.37±0\.05\)×10−2\\textbf\{\(0\.37\}\\pm\\textbf\{0\.05\)\}\\times 10^\{\-2\}\(1\.13±0\.14\)×10−3\\textbf\{\(1\.13\}\\pm\\textbf\{0\.14\)\}\\times 10^\{\-3\}#### IV\-B1Overall Performance

Figure[6](https://arxiv.org/html/2610.33654#S4.F6)and Table[III](https://arxiv.org/html/2610.33654#S4.T3)present the training\- and test\-phase performance comparison of eight methods in the CBESS coordination problem, respectively\. Four metrics are evaluated throughout the training process: 1\) operating cost, which represents the economic objective; 2\) voltage\-violation degree, which measures the magnitude of voltage\-limit exceedance at each bus and time step; 3\) voltage\-violation count, which measures the frequency of bus\-voltage violations; and 4\) line\-overload degree, which measures line\-capacity exceedance at each time step\.

![Refer to caption](https://arxiv.org/html/2610.33654v1/voltage_boxplot_2x4.png)Fig\. 7:Bus voltage distribution in test\.Fig\. 8:Line loading rate distribution in test\.Among vanilla DRL methods, PPO, SAC and TD3 converge with non\-negligible steady\-state voltage violations and persistent line\-overload levels\. Introducing LDCPPO reduces empirical constraint violations in terms of violation magnitude, frequency, and line\-overload degree\. Furthermore, incorporating STG\-LDCPPO further suppresses these violations\. The PISTG\-CPO and PI\-GR\-STG\-LDCPPO further show that either using a classical constrained update with PISTG or adding prior physics\-informed graph regularization to STG is helpful but insufficient, because effective CBESS coordination requires both constraint\-specific value estimation and physically consistent spatiotemporal embeddings\. Overall, PISTG\-LDCPPO achieves the lowest empirical constraint violations with competitive operating cost\. Compared with all baselines in Table[III](https://arxiv.org/html/2610.33654#S4.T3), it reduces the voltage\-violation degree by approximately49\.0%49\.0\\%–80\.5%80\.5\\%, the voltage\-violation count by28\.8%28\.8\\%–57\.5%57\.5\\%, the line\-overload degree by0\.9%0\.9\\%–26\.6%26\.6\\%, and the operating cost by about2\.1%2\.1\\%–4\.9%4\.9\\%\. Together, these results confirm the complementary roles of PISTG and constraint\-specific critics\. Their combination achieves the strongest reduction in heterogeneous network\-constraint violations, supporting the effectiveness of the proposed problem\-driven integration\.

\(a\)min\. and max\. bus voltage\(b\)max\. line loading rate
Fig\. 9:Average daily evolution of system\-level safety indicators\.Fig\. 10:Daily SOC trajectories of four CBESS units under PISTG\-LDCPPO and PPO during testing\.
#### IV\-B2Analysis of Voltage and Load

The bus voltage in Fig\.[7](https://arxiv.org/html/2610.33654#S4.F7)and line loading rate boxplots in Fig\.[8](https://arxiv.org/html/2610.33654#S4.F8)jointly illustrate the distribution of network states across buses and lines over the test horizon, with dashed lines marking the operational limits\. The shaded regions further highlight the violating ranges, while the plotted outliers indicate extreme operating states beyond the main distribution\. Compared with the wider dispersion and longer tails observed in the vanilla baselines, the proposed methods progressively tighten both voltage and loading distributions\. PISTG\-LDCPPO produces the most compact and well\-confined distributions across buses and lines, with fewer extreme values and reduced variability\.

The averaged evolutions over 100 test episodes of DN\-wide extremes, i\.e\., minimum/maximum bus voltage in Fig\.[9](https://arxiv.org/html/2610.33654#S4.F9)\(a\) and maximum line loading in Fig\.[9](https://arxiv.org/html/2610.33654#S4.F9)\(b\), provide a system\-level view of safety during operation\. Compared to the baselines that show larger oscillations and more pronounced peaks/troughs in these extreme trajectories, PISTG\-LDCPPO’s voltage extremes are better confined within the admissible band and the maximum loading trajectory violates less from the line load threshold with reduced fluctuation\.

Figure[10](https://arxiv.org/html/2610.33654#S4.F10)compares the daily SOC trajectories of the four CBESS units under PISTG\-LDCPPO and PPO, alongside the PV and load profiles\. During the daytime PV peak, PPO tends to charge aggressively and saturate early, whereas PISTG\-LDCPPO adopts a more moderated charging strategy that better preserves scheduling flexibility\. During the evening peak, PISTG\-LDCPPO discharges more proactively to support network operation, with only a slight sacrifice in economic performance\. This behavior reflects a more grid\-aware dispatch strategy that prioritizes network support and lower operational\-constraint violations\.

#### IV\-B3Scalability Analysis on the IEEE 69\-Bus System

![Refer to caption](https://arxiv.org/html/2610.33654v1/bus_69.png)Fig\. 11:CBESS deployment in the IEEE 69\-bus system with high DER penetration\.![Refer to caption](https://arxiv.org/html/2610.33654v1/train_curves_69bus_2x2_main.png)Fig\. 12:Training performance on the four key metrics of the compared methods on IEEE 69\-bus system\. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs\.TABLE IV:Test performance comparison over 100 episodes for 5 runs on IEEE 69\-bus system\.MethodOperating Cost \(AUD/day\)Volt\. Vio\. Degree \[p\.u\./\(5min⋅\\cdotbus\)\]Volt\. Vio\. Count \[times/\(5min⋅\\cdotbus\)\]Line\-Overload Degree \[%/\(5min⋅\\cdotline\)\]TD33740\.8±\\pm190\.4\(1\.98±\\pm0\.42\)×10−3\\times 10^\{\-3\}\(2\.28±\\pm0\.35\)×10−2\\times 10^\{\-2\}\(1\.85±\\pm0\.42\)×10−3\\times 10^\{\-3\}SAC3455\.6±\\pm210\.7\(2\.12±\\pm0\.48\)×10−3\\times 10^\{\-3\}\(1\.91±\\pm0\.43\)×10−2\\times 10^\{\-2\}\(1\.92±\\pm0\.35\)×10−3\\times 10^\{\-3\}PPO2965\.2±\\pm245\.6\(1\.60±\\pm0\.22\)×10−3\\times 10^\{\-3\}\(1\.22±\\pm0\.20\)×10−2\\times 10^\{\-2\}\(1\.76±\\pm0\.25\)×10−3\\times 10^\{\-3\}LDCPPO3265\.4±\\pm260\.1\(1\.03±\\pm0\.18\)×10−3\\times 10^\{\-3\}\(0\.84±\\pm0\.14\)×10−2\\times 10^\{\-2\}\(1\.43±\\pm0\.15\)×10−3\\times 10^\{\-3\}STG\-LDCPPO3135\.5±\\pm95\.4\(0\.82±\\pm0\.15\)×10−3\\times 10^\{\-3\}\(0\.76±\\pm0\.18\)×10−2\\times 10^\{\-2\}\(1\.22±\\pm0\.12\)×10−3\\times 10^\{\-3\}PI\-GR\-STG\-LDCPPO3122\.1±\\pm172\.7\(0\.67±\\pm0\.12\)×10−3\\times 10^\{\-3\}\(0\.67±\\pm0\.11\)×10−2\\times 10^\{\-2\}\(1\.23±\\pm0\.14\)×10−3\\times 10^\{\-3\}PISTG\-CPO3013\.0±\\pm190\.4\(1\.06±\\pm0\.16\)×10−3\\times 10^\{\-3\}\(0\.91±\\pm0\.15\)×10−2\\times 10^\{\-2\}\(1\.46±\\pm0\.20\)×10−3\\times 10^\{\-3\}PISTG\-LDCPPO3060\.7±\\pm135\.2\(0\.52±\\pm0\.09\)×10−3\\times 10^\{\-3\}\(0\.60±\\pm0\.10\)×10−2\\times 10^\{\-2\}\(1\.15±\\pm0\.14\)×10−3\\times 10^\{\-3\}TABLE V:Inference time and parameter count on IEEE 33\-bus and 69\-bus systems\.Method33\-bus69\-busTimeParamsTimeParams\(ms\)\(M\)\(ms\)\(M\)PPO0\.340\.100\.420\.12LDCPPO0\.490\.240\.580\.28STG\-LDCPPO1\.180\.471\.740\.53PISTG\-LDCPPO1\.220\.471\.780\.53

To examine scalability, we further test the proposed framework on the IEEE 69\-bus distribution system, as shown in Fig\.[11](https://arxiv.org/html/2610.33654#S4.F11)\. Compared with the 33\-bus case, the number of CBESS units increases from 4 to 9; PV is deployed at 53 of the 68 non\-slack buses, giving a PV deployment ratio of77\.9%77\.9\\%, and the PV penetration level is kept at approximately90%90\\%\.

Figure[12](https://arxiv.org/html/2610.33654#S4.F12)and Table[IV](https://arxiv.org/html/2610.33654#S4.T4)evaluate the proposed framework on the IEEE 69\-bus system, where PISTG\-LDCPPO maintains consistent performance and achieves lower empirical network\-constraint violations than the baseline DRL and safe\-DRL methods\. Table[V](https://arxiv.org/html/2610.33654#S4.T5)further shows that its inference time increases only from1\.221\.22ms on the 33\-bus system to1\.781\.78ms on the 69\-bus system, with the parameter count increasing from0\.470\.47M to0\.530\.53M, remaining negligible relative to the 5\-min control interval\. Overall, these results demonstrate that PISTG\-LDCPPO scales effectively to larger distribution networks while maintaining constraint\-aware performance and low computational overhead\.

#### IV\-B4PV Curtailment Analysis

We evaluate PV curtailment test\-time behavior by reporting the curtailed PV energyEcurtE^\{\\mathrm\{curt\}\}, the PV curtailment ratioρcurt\\rho^\{\\mathrm\{curt\}\}, and the high\-PV curtailment shareηHP\\eta^\{\\mathrm\{HP\}\}:

Ecurt\\displaystyle E^\{\\mathrm\{curt\}\}=∑t∈𝒯∑i∈𝒩DERκi,t​p~i,tPV​Δ​t,\\displaystyle=\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{i\\in\\mathcal\{N\}^\{\\mathrm\{DER\}\}\}\\kappa\_\{i,t\}\\tilde\{p\}^\{\\mathrm\{PV\}\}\_\{i,t\}\\Delta t,\(37a\)ρcurt\\displaystyle\\rho^\{\\mathrm\{curt\}\}=∑t∈𝒯∑i∈𝒩DERκi,t​p~i,tPV​Δ​t∑t∈𝒯∑i∈𝒩DERp~i,tPV​Δ​t×100%,\\displaystyle=\\frac\{\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{i\\in\\mathcal\{N\}^\{\\mathrm\{DER\}\}\}\\kappa\_\{i,t\}\\tilde\{p\}^\{\\mathrm\{PV\}\}\_\{i,t\}\\Delta t\}\{\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{i\\in\\mathcal\{N\}^\{\\mathrm\{DER\}\}\}\\tilde\{p\}^\{\\mathrm\{PV\}\}\_\{i,t\}\\Delta t\}\\times 100\\%,\(37b\)ηHP\\displaystyle\\eta^\{\\mathrm\{HP\}\}=∑t∈𝒯HP∑i∈𝒩DERκi,t​p~i,tPV​Δ​t∑t∈𝒯∑i∈𝒩DERκi,t​p~i,tPV​Δ​t×100%,\\displaystyle=\\frac\{\\sum\_\{t\\in\\mathcal\{T\}^\{\\mathrm\{HP\}\}\}\\sum\_\{i\\in\\mathcal\{N\}^\{\\mathrm\{DER\}\}\}\\kappa\_\{i,t\}\\tilde\{p\}^\{\\mathrm\{PV\}\}\_\{i,t\}\\Delta t\}\{\\sum\_\{t\\in\\mathcal\{T\}\}\\sum\_\{i\\in\\mathcal\{N\}^\{\\mathrm\{DER\}\}\}\\kappa\_\{i,t\}\\tilde\{p\}^\{\\mathrm\{PV\}\}\_\{i,t\}\\Delta t\}\\times 100\\%,\(37c\)where𝒯HP\\mathcal\{T\}^\{\\mathrm\{HP\}\}denotes the high\-PV period\. The first two metrics measure the absolute and normalized curtailment levels, whileηHP\\eta^\{\\mathrm\{HP\}\}indicates whether curtailment is mainly concentrated during high\-PV periods\.

TABLE VI:PV curtailment statistics during testing\.MethodCaseEcurtE^\{\\mathrm\{curt\}\}\(kWh/day\)ρcurt\\rho^\{\\mathrm\{curt\}\}\(%\)ηHP\\eta^\{\\mathrm\{HP\}\}\(%\)TD333142\.6±19\.8142\.6\{\\pm\}19\.83\.57±0\.503\.57\{\\pm\}0\.5056\.856\.869386\.4±42\.7386\.4\{\\pm\}42\.74\.55±0\.504\.55\{\\pm\}0\.5055\.355\.3SAC33134\.1±18\.5134\.1\{\\pm\}18\.53\.35±0\.463\.35\{\\pm\}0\.4659\.459\.469338\.7±39\.6338\.7\{\\pm\}39\.63\.98±0\.473\.98\{\\pm\}0\.4759\.159\.1PPO33111\.3±15\.6111\.3\{\\pm\}15\.62\.78±0\.392\.78\{\\pm\}0\.3963\.563\.569226\.9±28\.4226\.9\{\\pm\}28\.42\.67±0\.332\.67\{\\pm\}0\.3365\.865\.8LDCPPO33104\.6±13\.9104\.6\{\\pm\}13\.92\.62±0\.352\.62\{\\pm\}0\.3570\.270\.269281\.5±31\.8281\.5\{\\pm\}31\.83\.31±0\.373\.31\{\\pm\}0\.3771\.471\.4STG\-LDCPPO3386\.5±11\.786\.5\{\\pm\}11\.72\.16±0\.292\.16\{\\pm\}0\.2976\.876\.869254\.3±27\.1254\.3\{\\pm\}27\.12\.99±0\.322\.99\{\\pm\}0\.3277\.677\.6PI\-GR\-STG\-LDCPPO3379\.1±10\.879\.1\{\\pm\}10\.81\.98±0\.271\.98\{\\pm\}0\.2779\.279\.269253\.0±26\.8253\.0\{\\pm\}26\.82\.97±0\.312\.97\{\\pm\}0\.3179\.279\.2PISTG\-CPO3389\.8±12\.589\.8\{\\pm\}12\.52\.25±0\.322\.25\{\\pm\}0\.3273\.373\.369233\.9±26\.5233\.9\{\\pm\}26\.52\.75±0\.312\.75\{\\pm\}0\.3174\.274\.2PISTG\-LDCPPO3368\.2±9\.4\\mathbf\{68\.2\{\\pm\}9\.4\}1\.71±0\.24\\mathbf\{1\.71\{\\pm\}0\.24\}83\.1\\mathbf\{83\.1\}69240\.8±24\.6\\mathbf\{240\.8\{\\pm\}24\.6\}2\.83±0\.29\\mathbf\{2\.83\{\\pm\}0\.29\}82\.6\\mathbf\{82\.6\}Table[VI](https://arxiv.org/html/2610.33654#S4.T6)shows that PISTG\-LDCPPO consistently maintains low PV curtailment across both test systems while achieving the lowest voltage\- and line\-related violations\. Although a few baselines yield slightly lower curtailment in individual cases, they exhibit substantially higher network\-constraint violations\. These results indicate that the proposed policy achieves a better overall balance between local PV utilization and secure network operation\.

#### IV\-B5Ablation of PISTG and Constraint\-Specific Critics

To isolate the contributions of the two main methodological mechanisms, we conduct ablation studies on the PISTG encoder and the constraint\-specific critic design\. Specifically, STG\-LDCPPO removes all physics\-informed components; PISTG\-LDCPPO w/o PI graph replaces the PI adjacencyA~\\tilde\{A\}with the binary topology adjacencyAA; and PISTG\-LDCPPO w/o KCL setsβ1=0\\beta\_\{1\}=0and PISTG\-LDCPPO w/o VDrop setsβ2=0\\beta\_\{2\}=0in Eq\. \([22](https://arxiv.org/html/2610.33654#S3.E22)\)\. The reconstruction loss is retained in all PISTG variants\.

TABLE VII:Ablation of PISTG encoder pre\-training on the IEEE 33\-bus test windows\.Encoder variantVolt\. MAEFlowNorm\. KCLNorm\. V\-drop×10−3\\times 10^\{\-3\}p\.u\.NRMSE %residual×10−2\\times 10^\{\-2\}residual×10−3\\times 10^\{\-3\}STG encoder7\.416\.824\.867\.53PISTG w/o PI graph5\.125\.412\.674\.96PISTG w/o KCL4\.624\.833\.783\.81PISTG w/o VDrop5\.895\.162\.186\.45Full PISTG3\.684\.121\.542\.91Table[VII](https://arxiv.org/html/2610.33654#S4.T7)shows that the full PISTG encoder achieves the best reconstruction accuracy and physics consistency, while removing the PI graph, KCL loss, or voltage\-drop loss degrades the corresponding spatial or physical constraints\. Table[VIII](https://arxiv.org/html/2610.33654#S4.T8)further confirms that these encoder improvements translate into better downstream control: compared with STG\-LDCPPO, PISTG\-LDCPPO reduces the voltage\-violation degree and count by52\.8%52\.8\\%and41\.3%41\.3\\%, respectively\. Overall, the gains arise from the complementary effects of physics\-informed graph propagation and constraint\-specific physical regularization\.

TABLE VIII:Impact of individual physics\-informed ingredients on downstream LDCPPO testing performance\.MethodOperating costVolt\. vio\. degreeVolt\. vio\. countLine\-overload degreeAUD/dayp\.u\./\(5CLOSE\(5min⋅\\cdotbus\)\)times/\(5CLOSE\(5min⋅\\cdotbus\)\)%/\(5CLOSE\(5min⋅\\cdotline\)\)STG\-LDCPPO1414\.8±36\.11414\.8\\pm 36\.1\(0\.53±0\.09\)×10−3\(0\.53\\pm 0\.09\)\\times 10^\{\-3\}\(0\.63±0\.06\)×10−2\(0\.63\\pm 0\.06\)\\times 10^\{\-2\}\(1\.15±0\.14\)×10−3\(1\.15\\pm 0\.14\)\\times 10^\{\-3\}w/o PI graph1396\.7±37\.21396\.7\\pm 37\.2\(0\.34±0\.05\)×10−3\(0\.34\\pm 0\.05\)\\times 10^\{\-3\}\(0\.46±0\.05\)×10−2\(0\.46\\pm 0\.05\)\\times 10^\{\-2\}\(1\.14±0\.13\)×10−3\(1\.14\\pm 0\.13\)\\times 10^\{\-3\}w/o KCL1401\.2±38\.51401\.2\\pm 38\.5\(0\.38±0\.06\)×10−3\(0\.38\\pm 0\.06\)\\times 10^\{\-3\}\(0\.50±0\.05\)×10−2\(0\.50\\pm 0\.05\)\\times 10^\{\-2\}\(1\.14±0\.14\)×10−3\(1\.14\\pm 0\.14\)\\times 10^\{\-3\}w/o VDrop1405\.8±37\.91405\.8\\pm 37\.9\(0\.42±0\.07\)×10−3\(0\.42\\pm 0\.07\)\\times 10^\{\-3\}\(0\.55±0\.06\)×10−2\(0\.55\\pm 0\.06\)\\times 10^\{\-2\}\(1\.15±0\.15\)×10−3\(1\.15\\pm 0\.15\)\\times 10^\{\-3\}Full1382\.4±38\.9\\textbf\{1382\.4\}\\pm\\textbf\{38\.9\}\(0\.25±0\.04\)×10−3\\textbf\{\(0\.25\}\\pm\\textbf\{0\.04\)\}\\times 10^\{\-3\}\(0\.37±0\.05\)×10−2\\textbf\{\(0\.37\}\\pm\\textbf\{0\.05\)\}\\times 10^\{\-2\}\(1\.13±0\.14\)×10−3\\textbf\{\(1\.13\}\\pm\\textbf\{0\.14\)\}\\times 10^\{\-3\}We further isolate the contribution of the constraint\-specific critic design by comparing PISTG\-LDCPPO with PISTG\-LSCPPO \(Lagrangian Single\-Critic PPO\), which uses the same PISTG encoder, actor, Lagrangian update, and training/testing protocol but combines the three constraint costs into a single cost critic\. Figures[13\(a\)](https://arxiv.org/html/2610.33654#S4.F13.sf1)and[13\(b\)](https://arxiv.org/html/2610.33654#S4.F13.sf2), together with Table[IX](https://arxiv.org/html/2610.33654#S4.T9), show that PISTG\-LDCPPO consistently reduces empirical constraint violations, especially voltage\-violation degree and count, on both systems while maintaining comparable operating costs\. This controlled comparison demonstrates the benefit of estimating heterogeneous constraint returns separately rather than through a single aggregated cost\-value estimate\.

\(a\)IEEE 33\-bus system\.\(b\)IEEE 69\-bus system\.
Fig\. 13:Training\-curve comparison between PISTG\-LSCPPO and PISTG\-LDCPPO\.TABLE IX:Comparison between PISTG\-LSCPPO and PISTG\-LDCPPO over 100 test episodes for 5 runs\.SystemMethodOperating costVolt\. vio\. degreeVolt\. vio\. countLine\-overload degreeAUD/dayp\.u\./\(5CLOSE\(5min⋅\\cdotbus\)\)times/\(5CLOSE\(5min⋅\\cdotbus\)\)%/\(5CLOSE\(5min⋅\\cdotline\)\)33\-busPISTG\-LSCPPO1408\.6±41\.51408\.6\\pm 41\.5\(0\.52±0\.08\)×10−3\(0\.52\\pm 0\.08\)\\times 10^\{\-3\}\(0\.58±0\.07\)×10−2\(0\.58\\pm 0\.07\)\\times 10^\{\-2\}\(1\.20±0\.15\)×10−3\(1\.20\\pm 0\.15\)\\times 10^\{\-3\}33\-busPISTG\-LDCPPO1382\.4±38\.9\\textbf\{1382\.4\}\\pm\\textbf\{38\.9\}\(0\.25±0\.04\)×10−3\\textbf\{\(0\.25\}\\pm\\textbf\{0\.04\)\}\\times 10^\{\-3\}\(0\.37±0\.05\)×10−2\\textbf\{\(0\.37\}\\pm\\textbf\{0\.05\)\}\\times 10^\{\-2\}\(1\.13±0\.14\)×10−3\\textbf\{\(1\.13\}\\pm\\textbf\{0\.14\)\}\\times 10^\{\-3\}69\-busPISTG\-LSCPPO3118\.4±148\.73118\.4\\pm 148\.7\(0\.88±0\.14\)×10−3\(0\.88\\pm 0\.14\)\\times 10^\{\-3\}\(0\.79±0\.12\)×10−2\(0\.79\\pm 0\.12\)\\times 10^\{\-2\}\(1\.28±0\.16\)×10−3\(1\.28\\pm 0\.16\)\\times 10^\{\-3\}69\-busPISTG\-LDCPPO3060\.7±135\.2\\textbf\{3060\.7\}\\pm\\textbf\{135\.2\}\(0\.52±0\.09\)×10−3\\textbf\{\(0\.52\}\\pm\\textbf\{0\.09\)\}\\times 10^\{\-3\}\(0\.60±0\.10\)×10−2\\textbf\{\(0\.60\}\\pm\\textbf\{0\.10\)\}\\times 10^\{\-2\}\(1\.15±0\.14\)×10−3\\textbf\{\(1\.15\}\\pm\\textbf\{0\.14\)\}\\times 10^\{\-3\}
#### IV\-B6Sensitivity Analysis of PIAM

To further validate the physics\-informed adjacency matrix \(PIAM\), we conduct two sensitivity studies\. The first study evaluates different edge\-weight constructions, and the second study examines the effect of the capacity\-weight coefficientν\\nuin PIAM\.

Figure[14](https://arxiv.org/html/2610.33654#S4.F14)compares four adjacency designs: binary topology, impedance\-only PI, capacity\-only PI, and full PI adjacency\. The full PI adjacency achieves the best overall performance, while the two single\-factor variants also outperform the binary topology baseline\. This confirms that jointly using line impedance and capacity provides a more informative physical graph prior than topology\-only or single\-factor weighting\.

Fig\. 14:Sensitivity analysis of different PIAM edge\-weight constructions\.Figure[15](https://arxiv.org/html/2610.33654#S4.F15)shows that PIAM is robust to the capacity\-weight coefficientν\\nu\. The performance varies from0\.50\.5to2\.02\.0, with the best overall result atν=1\.0\\nu=1\.0, indicating that PIAM is benefiting from a balanced use of impedance and capacity information\.

Fig\. 15:Sensitivity analysis of the PIAM capacity\-weight coefficientν\\nu\.TABLE X:Physical correlation analysis of learned representations\.EncoderVolt\. vio\. degreeVolt\. vio\. countLine\-overload degreeρ\\rhoR2R^\{2\}ρ\\rhoR2R^\{2\}ρ\\rhoR2R^\{2\}STG encoder0\.710\.490\.670\.440\.540\.29PISTG encoder0\.890\.780\.840\.690\.580\.32
#### IV\-B7Physical Correlation Analysis of Learned Embeddings

To further verify whether the proposed PISTG encoder learns physically meaningful representations, we conduct a physical correlation analysis on the learned latent space\. Letztz\_\{t\}denote the latent vector generated by an encoder before the policy head\. We examine whetherztz\_\{t\}preserves information related to three physical stress indicators:

χtVD\\displaystyle\\chi\_\{t\}^\{\\mathrm\{VD\}\}=∑i∈𝒩\[\(Vi,t−V¯\)\+\+\(V¯−Vi,t\)\+\],\\displaystyle=\\sum\_\{i\\in\\mathcal\{N\}\}\\left\[\(V\_\{i,t\}\-\\overline\{V\}\)^\{\+\}\+\(\\underline\{V\}\-V\_\{i,t\}\)^\{\+\}\\right\],\(38\)χtVN\\displaystyle\\chi\_\{t\}^\{\\mathrm\{VN\}\}=∑i∈𝒩𝕀⁡\[\(Vi,t\>V¯\)∨\(Vi,t<V¯\)\],\\displaystyle=\\sum\_\{i\\in\\mathcal\{N\}\}\\mathbb\{I\}\\left\[\(V\_\{i,t\}\>\\overline\{V\}\)\\vee\(V\_\{i,t\}<\\underline\{V\}\)\\right\],\(39\)χtLO\\displaystyle\\chi\_\{t\}^\{\\mathrm\{LO\}\}=∑\(i,j\)∈ℒ\[pi​j,t2\+qi​j,t2−Si​j\]\+\.\\displaystyle=\\sum\_\{\(i,j\)\\in\\mathcal\{L\}\}\\left\[\\sqrt\{p\_\{ij,t\}^\{2\}\+q\_\{ij,t\}^\{2\}\}\-S\_\{ij\}\\right\]^\{\+\}\.\(40\)For each indicatork∈\{VD,VN,LO\}k\\in\\\{\\mathrm\{VD\},\\mathrm\{VN\},\\mathrm\{LO\}\\\}, we fit a simple linear probeχ^tk=wk⊤​zt\+bk\\widehat\{\\chi\}\_\{t\}^\{k\}=w\_\{k\}^\{\\top\}z\_\{t\}\+b\_\{k\}and report the Pearson correlation coefficientρ\\rhoand coefficient of determinationR2R^\{2\}on testing trajectories\. The probe is only used for interpretation and is not involved in policy training or control execution\.

As shown in Table[X](https://arxiv.org/html/2610.33654#S4.T10), the PISTG embedding is more strongly correlated with voltage\-related physical stress than the ordinary STG embedding\. This is consistent with Table[III](https://arxiv.org/html/2610.33654#S4.T3), where PISTG\-LDCPPO achieves lower voltage violations than STG\-LDCPPO\. The smaller improvement in line\-overload degree also matches the weaker line\-overload correlation\. These results suggest that the PISTG latent space is better aligned with power\-flow\-related stress indicators, especially voltage stress\.

## VConclusion and Future Work

This paper investigates the coordinated operation of CBESSs in a DN with high DER penetration, aiming to improve empirical network\-constraint compliance and economic efficiency\. Our study targets three key challenges: mixed constraint feedback under multi\-CBESS constraint\-aware learning, insufficient physical awareness in purely data\-driven control policies, and nonlocal spatiotemporal interactions under large\-scale CBESS dispatch\. To address the first challenge, we develop a constraint\-specific value\-decomposition mechanism that preserves separate long\-term learning signals for heterogeneous operational constraints\. To address the latter two, we develop a dual\-level physics\-informed spatiotemporal representation that embeds distribution\-network physics into both graph propagation and latent\-state learning, enabling the policy to capture physically meaningful nonlocal spatiotemporal dependencies\. Extensive experiments demonstrate that the proposed method substantially reduces empirical voltage\-limit and line\-loading violations while maintaining competitive operating cost compared with representative DRL and constrained\-RL baselines under scenarios with high DER penetration\.

Future work will extend the proposed framework toward more practical deployment scenarios, including three\-phase unbalanced and reconfigurable distribution networks, cooperative multi\-agent CBESS coordination under local observations and limited communication, and operation at shorter decision intervals with higher\-resolution measurements, and under noisy, delayed, or incomplete measurements\. We will further develop uncertainty\-aware models and an execution\-time safety\-assurance layer that validates learned actions through online power\-flow analysis and, when necessary, applies corrective measures such as inverter reactive\-power support, conservative CBESS redispatch, or limited PV curtailment\.

## References

- \[1\]International Energy Agency \(IEA\)Net zero by 2050: a roadmap for the global energy sector\.External Links:[Link](https://www.iea.org/reports/net-zero-by-2050)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[2\]M\. Bloch\(2025\)Australian home battery uptake accelerates\.External Links:[Link](https://www.solarquotes.com.au/blog/australia-battery-uptake-mb3139)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[3\]N\. Thompson\(2024\)State of EVs 2024\.External Links:[Link](https://electricvehiclecouncil.com.au/reports/state-of-evs-2024/)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[4\]R\. Tonkoski, D\. Turcotte, and T\. H\. M\. EL\-Fouly\(2012\)Impact of high PV penetration on voltage profiles in residential neighborhoods\.IEEE Transactions on Sustainable Energy3\(3\),pp\. 518–527\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2012.2191425)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[5\]\(2024\)Consumer energy resources and the transformation of the NEM\.Energy Security Board \(ESB\)\.External Links:[Link](https://www.energy.gov.au/energy-and-climate-change-ministerial-council/energy-ministers-publications/consumer-energy-resources-and-transformation-of-nem)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[6\]Q\. Yang and H\. Wang\(2021\)Exploring blockchain for the coordination of distributed energy resources\.In2021 55th Annual Conference on Information Sciences and Systems \(CISS\),pp\. 1–6\.Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[7\]M\. G\. Kashani, M\. Mobarrez, and S\. Bhattacharya\(2019\)Smart inverter volt\-Watt control design in high PV\-penetrated distribution systems\.IEEE Transactions on Industry Applications55\(2\),pp\. 1147–1156\.External Links:[Document](https://dx.doi.org/10.1109/TIA.2018.2878844)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[8\]M\. Zeraati, M\. E\. Hamedani Golshan, and J\. M\. Guerrero\(2019\)A consensus\-based cooperative control of PEV battery and PV active power curtailment for voltage regulation in distribution networks\.IEEE Transactions on Smart Grid10\(1\),pp\. 670–680\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2017.2749623)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[9\]M\. Zeraati, M\. E\. Hamedani Golshan, and J\. M\. Guerrero\(2018\)Distributed control of battery energy storage systems for voltage regulation in distribution networks with high PV penetration\.IEEE Transactions on Smart Grid9\(4\),pp\. 3582–3593\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2016.2636217)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p1.1)\.
- \[10\]AusgridCommunity batteries\.External Links:[Link](https://www.ausgrid.com.au/In-your-community/Community-Batteries)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[11\]Department of Climate Change, Energy, the Environment and Water, Australian GovernmentCommunity batteries for household solar program\.External Links:[Link](https://www.dcceew.gov.au/energy/renewable/community-batteries)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[12\]E\. Barbour, D\. Parra, Z\. Awwad, and M\. C\. González\(2018\)Community energy storage: a smart choice for the smart grid?\.Applied energy212,pp\. 489–497\.Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[13\]B\. J\. Kalkbrenner\(2019\)Residential vs\. community battery storage systems–consumer preferences in germany\.Energy Policy129,pp\. 1355–1363\.Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[14\]A\. Mohanty, A\. Ramasamy, R\. Verayiah, and S\. Mohanty\(2024\)Neighborhood and community battery projects: a systematic analysis of their current state and future prospects\.Journal of Energy Storage95,pp\. 112525\.Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[15\]Australian Renewable Energy Agency \(ARENA\)Energy queensland \- community batteries project\.External Links:[Link](https://arena.gov.au/projects/energy-queensland-community-batteries-project/)Cited by:[§I\-A](https://arxiv.org/html/2610.33654#S1.SS1.p2.1)\.
- \[16\]T\. Tewari, A\. Mohapatra, and S\. Anand\(2021\)Coordinated control of OLTC and energy storage for voltage regulation in distribution network with high PV penetration\.IEEE Transactions on Sustainable Energy12\(1\),pp\. 262–272\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2020.2991017)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.3.1.1.1)\.
- \[17\]Y\. Li, M\. Han, Z\. Yang, and G\. Li\(2021\)Coordinating flexible demand response and renewable uncertainties for scheduling of community integrated energy systems with an electric vehicle charging station: a bi\-level approach\.IEEE Transactions on Sustainable Energy12\(4\),pp\. 2321–2331\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2021.3090463)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.4.1.1.1)\.
- \[18\]S\. Dorahaki, S\. M\. Muyeen, and N\. Amjady\(2024\)A stackelberg game theory model for integrated community energy storage systems\.IEEE Access12\(\),pp\. 152908–152920\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2024.3481155)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.4.1.1.1)\.
- \[19\]Y\. Sun, H\. Hui, T\. Qi, and L\. Chen\(2024\)Multitime scale optimization of urban Micro\-Grids considering high penetration of PVs and heterogeneous energy storage systems\.IEEE Internet of Things Journal11\(14\),pp\. 24428–24438\.External Links:[Document](https://dx.doi.org/10.1109/JIOT.2024.3354803)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.4.1.1.1)\.
- \[20\]Y\. Wang, H\. Wang, M\. Wagner, and A\. Liebman\(2023\)Community battery energy storage systems for enhancing distribution system operation: a multi\-objective optimization approach\.In2023 IEEE International Conference on Energy Technologies for Future Grids \(ETFG\),Vol\.,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/ETFG55873.2023.10408412)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.4.1.1.1)\.
- \[21\]C\. Jiang and H\. Wang\(2024\)Network\-aware value stacking of community battery via asynchronous distributed optimization\.In2024 IEEE Power & Energy Society General Meeting \(PESGM\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/PESGM51994.2024.10761093)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.4.1.1.1)\.
- \[22\]S\. Wang, L\. Du, X\. Fan, and Q\. Huang\(2021\)Deep reinforcement scheduling of energy storage systems for real\-time voltage regulation in unbalanced LV networks with high PV penetration\.IEEE Transactions on Sustainable Energy12\(4\),pp\. 2342–2352\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2021.3092961)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.5.1.1.1)\.
- \[23\]M\. Al\-Saffar and P\. Musilek\(2020\)Reinforcement learning\-based distributed BESS management for mitigating overvoltage issues in systems with high PV penetration\.IEEE Transactions on Smart Grid11\(4\),pp\. 2980–2994\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2020.2972208)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.6.1.1.1)\.
- \[24\]Y\. Su, J\. Teh, S\. Fang, Z\. Dai, and P\. Wang\(2025\)Two\-stage optimal dispatch framework of active distribution networks with hybrid energy storage systems via deep reinforcement learning and real\-time feedback dispatch\.Journal of Energy Storage108,pp\. 115169\.Cited by:[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.6.1.1.1)\.
- \[25\]Y\. Tao, Z\. Y\. Dong, J\. Qiu, S\. Lai, X\. Sun, J\. Ruan, J\. Lin, and J\. Zhao\(2025\)Human\-in\-the\-loop reinforcement learning method for volt/var control in active distribution network with safe operation mechanism\.IEEE Transactions on Sustainable Energy\(\),pp\. 1–13\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2025.3581180)Cited by:[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.7.1.1.1)\.
- \[26\]D\. Cao, J\. Zhao, W\. Hu, F\. Ding, Q\. Huang, Z\. Chen, and F\. Blaabjerg\(2021\)Data\-driven multi\-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of PVs\.IEEE Transactions on Smart Grid12\(5\),pp\. 4137–4150\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2021.3072251)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.8.1.1.1)\.
- \[27\]D\. Cao, J\. Zhao, J\. Hu, Y\. Pei, Q\. Huang, Z\. Chen, and W\. Hu\(2024\)Physics\-informed graphical representation\-enabled deep reinforcement learning for robust distribution system voltage control\.IEEE Transactions on Smart Grid15\(1\),pp\. 233–246\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2023.3267069)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.9.1.1.1),[§IV\-A2](https://arxiv.org/html/2610.33654#S4.SS1.SSS2.p1.1)\.
- \[28\]A\. S\. Omar and R\. El\-Shatshat\(2025\)Energy management of multi\-energy communities: a hierarchical MIQP\-constrained deep reinforcement learning approach\.IEEE Transactions on Sustainable Energy16\(3\),pp\. 2236–2250\.External Links:[Document](https://dx.doi.org/10.1109/TSTE.2025.3550563)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p3.1),[TABLE I](https://arxiv.org/html/2610.33654#S1.T1.4.10.1.1.1),[§IV\-A2](https://arxiv.org/html/2610.33654#S4.SS1.SSS2.p1.1)\.
- \[29\]D\. Zhao, H\. Wang, J\. Huang, and X\. Lin\(2020\)Virtual energy storage sharing and capacity allocation\.IEEE Transactions on Smart Grid11\(2\),pp\. 1112–1123\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2019.2932057)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1)\.
- \[30\]N\. Tucker and M\. Alizadeh\(2022\)An online scheduling algorithm for a community energy storage system\.IEEE Transactions on Smart Grid13\(6\),pp\. 4651–4664\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2022.3179251)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p1.1)\.
- \[31\]J\. Fan and H\. Wang\(2023\)Deep reinforcement learning for community battery scheduling under uncertainties of load, PV generation, and energy prices\.In2023 IEEE 7th Conference on Energy Internet and Energy System Integration \(EI2\),Vol\.,pp\. 4871–4876\.External Links:[Document](https://dx.doi.org/10.1109/EI259745.2023.10513300)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1)\.
- \[32\]H\. Song, Y\. Liu, J\. Zhao, J\. Liu, and G\. Wu\(2021\)Prioritized replay dueling DDQN based grid\-edge control of community energy storage system\.IEEE Transactions on Smart Grid12\(6\),pp\. 4950–4961\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2021.3099133)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1)\.
- \[33\]J\. Li, C\. Wang, and H\. Wang\(2023\)Deep reinforcement learning for wind and energy storage coordination in wholesale energy and ancillary service markets\.Energy and AI14,pp\. 100280\.Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1)\.
- \[34\]J\. Li, C\. Wang, Y\. Zhang, and H\. Wang\(2024\)Temporal\-aware deep reinforcement learning for energy storage bidding in energy and contingency reserve markets\.IEEE Transactions on Energy Markets, Policy and Regulation2\(3\),pp\. 392–406\.Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1)\.
- \[35\]K\. Xiong, Q\. Wei, and Y\. Liu\(2025\)Community microgrid energy co\-scheduling based on deep reinforcement learning and contribution mechanisms\.IEEE Transactions on Smart Grid16\(2\),pp\. 1051–1061\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2024.3461320)Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p2.1)\.
- \[36\]H\. J\. Kim and D\. Kim\(2026\)Safe multi\-critic reinforcement learning framework for coordinated energy management and voltage regulation in active distribution networks\.Sustainable Energy, Grids and Networks,pp\. 102179\.Cited by:[§I\-B](https://arxiv.org/html/2610.33654#S1.SS2.p3.1)\.
- \[37\]M\.E\. Baran and F\.F\. Wu\(1989\)Network reconfiguration in distribution systems for loss reduction and load balancing\.IEEE Transactions on Power Delivery4\(2\),pp\. 1401–1407\.External Links:[Document](https://dx.doi.org/10.1109/61.25627)Cited by:[§II\-A4](https://arxiv.org/html/2610.33654#S2.SS1.SSS4.p1.1)\.
- \[38\]H\. T\. Nguyen and D\. Choi\(2022\)Three\-stage inverter\-based peak shaving and volt\-var control in active distribution networks using online safe deep reinforcement learning\.IEEE Transactions on Smart Grid13\(4\),pp\. 3266–3277\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2022.3166192)Cited by:[§III\-B1](https://arxiv.org/html/2610.33654#S3.SS2.SSS1.p1.3)\.
- \[39\]Y\. Yang, C\. Huang, and H\. Wang\(2026\)Why dedicated critics: eliminating target drift in multi\-constraint RL\.InForty\-third International Conference on Machine Learning,Cited by:[§III\-C1](https://arxiv.org/html/2610.33654#S3.SS3.SSS1.p1.1)\.
- \[40\]\(2021\)Battery storage: the new, clean peaker\.Clean Energy Council\.External Links:[Link](https://assets.cleanenergycouncil.org.au/documents/resources/reports/battery-storage-the-new-clean-peaker.pdf)Cited by:[§IV\-A1](https://arxiv.org/html/2610.33654#S4.SS1.SSS1.p1.1)\.
- \[41\]E\. L\. Ratnam, S\. R\. Weller, C\. M\. Kellett, and A\. T\. Murray\(2017\)Residential load and rooftop pv generation: an australian distribution network dataset\.International Journal of Sustainable Energy36\(8\),pp\. 787–806\.External Links:[Document](https://dx.doi.org/10.1080/14786451.2015.1100196)Cited by:[§IV\-A1](https://arxiv.org/html/2610.33654#S4.SS1.SSS1.p2.1)\.
- \[42\]J\. Achiam, D\. Held, A\. Tamar, and P\. Abbeel\(2017\)Constrained policy optimization\.InInternational conference on machine learning,pp\. 22–31\.Cited by:[§IV\-A2](https://arxiv.org/html/2610.33654#S4.SS1.SSS2.p1.1)\.
- \[43\]H\. Li and H\. He\(2022\)Learning to operate distribution networks with safe deep reinforcement learning\.IEEE Transactions on Smart Grid13\(3\),pp\. 1860–1872\.External Links:[Document](https://dx.doi.org/10.1109/TSG.2022.3142961)Cited by:[§IV\-A2](https://arxiv.org/html/2610.33654#S4.SS1.SSS2.p2.1)\.
- \[44\]T\. Wu, A\. Scaglione, and D\. Arnold\(2024\)Constrained reinforcement learning for predictive control in real\-time stochastic dynamic optimal power flow\.IEEE Transactions on Power Systems39\(3\),pp\. 5077–5090\.External Links:[Document](https://dx.doi.org/10.1109/TPWRS.2023.3326121)Cited by:[§IV\-A2](https://arxiv.org/html/2610.33654#S4.SS1.SSS2.p2.1)\.

![[Uncaptioned image]](https://arxiv.org/html/2610.33654v1/huangchenghao.jpg)Chenghao Huangreceived the B\.E\. degree in Software Engineering and the M\.S\. degree in Computer Science and Engineering from the University of Electronic Science and Technology of China, Chengdu, China, in 2020 and 2023, respectively\. He is currently pursuing the Ph\.D\. degree in the Faculty of Information Technology, Monash University, Melbourne, VIC, Australia\. His research interests deep learning, federated learning, reinforcement learning and smart grids\.![[Uncaptioned image]](https://arxiv.org/html/2610.33654v1/YueYang.JPG)Yue Yangreceived the B\.A\. in Mathematics and Computer Science from Mount Holyoke College, Massachusetts, USA, in 2017\. She received her Ph\.D\. in Information Technology from Monash University, Melbourne, VIC, Australia, in 2026\. Her research interests include include reinforcement learning, large language model training and inference, with their applications\.![[Uncaptioned image]](https://arxiv.org/html/2610.33654v1/Hao_Wang.png)Hao Wang\(M’16\) received his Ph\.D\. in Information Engineering from The Chinese University of Hong Kong, Hong Kong, in 2016\. He was a Postdoctoral Research Fellow at Stanford University, Stanford, CA, USA, and a Washington Research Foundation Innovation Fellow at the University of Washington, Seattle, WA, USA\.He is currently a Senior Lecturer in the Department of Data Science and AI, Faculty of IT, Monash University, Melbourne, VIC, Australia\. He received Washington Research Foundation Innovation Fellowship 2016\-2018, ARC Discovery Early Career Researcher Award \(DECRA\) 2023\-2025, First Prize Paper Award from IEEE Industry Applications Magazine 2024, Best Vision Paper Award from ACM SIGSPATIAL 2024, Best Conference Paper from IEEE PESGM 2024, Best Paper Award at IEEE SmartGridComm 2025, Best Special Theme Paper at EMNLP 2025, Best Paper Award at IEEE EPEC 2025, IJPEM\-GT Award from International Journal of Precision Engineering and Manufacturing\-Green Technology in 2025\. His research interests include optimization, machine learning, and data analytics for power and energy systems\.