Learning to Run Power Networks: Effective AlphaZero-inspired Topological Control

# Learning to Run Power Networks: Effective AlphaZero-inspired Topological Control
Source: [https://arxiv.org/html/2608.14114](https://arxiv.org/html/2608.14114)
## Learning to Run Power Networks: Effective AlphaZero\-inspired Topological ControlCCS:Computing methodologies Reinforcement learningCCS:Hardware Smart grid

###### Abstract\.

As the integration of volatile renewable energy sources increases the strain on modern power grids, the use of Reinforcement Learning \(RL\) for autonomous topological reconfiguration has emerged as a promising research field to keep strained grids stable and operational\. Compared to traditional redispatching measures, topological actions offer a cheaper and more cost\-effective way to manage grid congestion\. However, their implementation is hindered by a vast combinatorial action space and strict operational constraints\. This paper investigates the effectiveness of model\-based AlphaZero\-inspired approaches that utilize Monte Carlo Tree Search \(MCTS\) for proactive grid management\. We systematically evaluate how reward functions, observation density, and search guidance influence an agent’s survivability\. Our results demonstrate that the optimized AlphaZero approach achieves a peak survivability of 98\.43%, significantly outperforming the proximal policy optimization \(PPO\) variant\. We find that conducting the MCTS without guidance from a prior learned policy or value function can enhance training efficiency, and that a straightforward binary survival reward provides more effective search guidance than complex, multi\-objective functions\. Our findings demonstrate that while AlphaZero is a powerful framework for topological control, pure reinforcement learning is not sufficient; rather, an effective and reliable system requires a ‘minimalist’ integration of domain\-specific heuristics, binary rewards, and a restricted observation space of line loads\.

###### Keywords:

Learning to Run a Power Network \(L2RPN\), Deep Reinforcement learning, Monte Carlo Tree Search \(MCTS\), AlphaZero, Topological control

Availability of Data and Material:

## 1\.Introduction

The global transition toward carbon neutrality has catalyzed a fundamental shift in power system dynamics\. While essential for reaching climate goals, the integration of volatile renewable energy sources \(RES\), such as solar and wind, introduces two main challenges to the grid: First, these generators are volatile, threatening the balance of supply and demand and thereby the stability of the grid\. Second, generation is often located far from consumption centers, as seen with offshore wind farms, increasing the strain on the transmission system and leading to congested and potentially overloaded lines\. These conditions can trigger cascading failures and catastrophic blackouts, paralyzing critical infrastructure as recently demonstrated in Spain and Portugal\([33](https://arxiv.org/html/2608.14114#bib.bib6)\)\.

Conventionally, the use of Reinforcement Learning \(RL\) for autonomous topological reconfiguration has emerged as a promising research field to keep strained grids stable and operational\. Historically, grid congestion has been managed through redispatching, which is adjusting the setpoints of generators\([11](https://arxiv.org/html/2608.14114#bib.bib12)\), or curtailing renewable energy generation or feed\-in management\([10](https://arxiv.org/html/2608.14114#bib.bib14);[28](https://arxiv.org/html/2608.14114#bib.bib15)\)\. While effective, these methods are economically costly and often underutilize existing infrastructure\. Topological actions, such as busbar reconfiguration and line switching, offer a more cost\-effective alternative by rerouting power through underloaded sections of the grid\. However, the practical implementation of topological control is hindered by the combinatorial explosion of possible grid configurations, which exceeds the capabilities of traditional exhaustive search or manual intervention\([17](https://arxiv.org/html/2608.14114#bib.bib26)\)\.

To address this complexity, theLearning to Run a Power Network\(L2RPN\) challenge was established as a multi\-year competition series to benchmark autonomous agents in realistic grid environments\([22](https://arxiv.org/html/2608.14114#bib.bib8)\)\. These challenges have facilitated a paradigm shift from traditional redispatching toward topological control as the primary tool for grid management\. While model\-free reinforcement learning \(RL\) agents, such as Proximal Policy Optimization \(PPO\), have set strong reactive baselines, they inherently lack explicit look\-ahead capabilities\([25](https://arxiv.org/html/2608.14114#bib.bib22)\)\. In safety\-critical power systems, a purely reactive agent struggles to handle cascading thermal failures because it cannot evaluate the downstream, multi\-step consequences of a topological action prior to execution\.

Consequently, there is an increasing interest shift toward model\-based frameworks, specifically those inspired by AlphaZero, which couple deep neural networks with Monte Carlo Tree Search \(MCTS\) to proactively plan stable grid configurations\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)\. AlphaZero uniqu\-ely bridges this gap by utilizing MCTS to perform systematic, non\-parametric exploration of the discrete combinatorial action space, while the neural network progressively learns to guide the search, mitigating the need for computationally exhaustive full\-tree rollouts\. While real\-world transmission networks encompass thousands of nodes, systematically analyzing the architectural sensitivities of these model\-based controllers requires a controlled environment\. In this work, we utilize the standard IEEE\-14 bus system\. Despite its relatively small physical footprint, its combination of substations and lines still yields a vast, non\-trivial combinatorial action space of 405 raw actions \(reduced to 203 via symmetry constraints\)\. This configuration serves as an ideal, computationally viable sandbox to isolate how reward structures, search guidance, and observation configurations impact MCTS performance before deploying such frameworks to larger scales\.

In addition, we provide a systematic evaluation of AlphaZero\-inspired approach proposed by\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)\. While AlphaZero has revolutionized board games\([26](https://arxiv.org/html/2608.14114#bib.bib13)\), its sensitivity to design factors in physical domains like power systems remains under\-explored\. We bridge this research gap by analyzing the impact of three critical design dimensions: observation density, reward shaping, and action space pruning\. Furthermore, we investigate the influence of MCTS guidance variants, ranging from heuristic\-based leaf evaluation to learned Q\-functions, and tune hyperparameters to balance computational efficiency with grid survivability\. Our findings contribute to a deeper understanding of the trade\-offs between search depth and training stability in autonomous grid management and can help implement AlphaZero approaches more effectively\.

The remainder of this paper is organized as follows: Section[2](https://arxiv.org/html/2608.14114#S2)reviews the previous literature in L2RPN; Section[3](https://arxiv.org/html/2608.14114#S3)details the methodology, including the implementation of the AlphaZero\-inspired approach and the design choices evaluated; Section[4](https://arxiv.org/html/2608.14114#S4)presents the experimental setup and results; Section[5](https://arxiv.org/html/2608.14114#S5)discusses the findings and their implications; and Section[6](https://arxiv.org/html/2608.14114#S6)summarizes the contributions and outlines future research directions\.

## 2\.Related Works

The application of Deep Reinforcement Learning \(DRL\) to power systems has been formalized by the L2RPN challenges, which have taken place annually between 2019 and 2023\([22](https://arxiv.org/html/2608.14114#bib.bib8)\)\. These competitions utilize realistic grid environments to benchmark autonomous agents’ ability to maintain stability via topological control, focusing on real\-time operational windows to prevent cascading failures\([30](https://arxiv.org/html/2608.14114#bib.bib7)\)\.

### 2\.1\.Evolution of the L2RPN Challenge

The inaugural 2019 IJCNN challenge utilized the IEEE 14\-bus system and established the core objective of operating a grid using only bus\-splitting and line\-switching actions\. The winning solutions from this first competition introduced several foundational concepts that have since become default standards in the field:

- •Action Space Reduction: The A3C\-2019 solution first identified the need to eliminate redundant busbar symmetry actions, a strategy now included by default in the Grid2Op package to manage combinatorial complexity\([24](https://arxiv.org/html/2608.14114#bib.bib9)\)\.
- •Curriculum learning\([2](https://arxiv.org/html/2608.14114#bib.bib10)\): To improve training effectiveness, early winners employed curriculum strategies, training neural networks on simpler tasks before transitioning to full grid complexity\.
- •Activation Thresholds: The DDQN\-2019 solution introduced a “warning flag” based on line loading levels\([16](https://arxiv.org/html/2608.14114#bib.bib11)\)\. This threshold determines when the agent should intervene versus choosing a “do\-nothing” action, a mechanism used in all subsequent L2RPN solutions\.
- •Guided Search and Simulation: Early agents began using guided exploration and the simulation of the top N predicted actions to select the best move\([21](https://arxiv.org/html/2608.14114#bib.bib31)\)\. This established a primitive form of look\-ahead that serves as the conceptual precursor to the more advanced Monte Carlo Tree Search \(MCTS\) employed in our work\.

### 2\.2\.Model\-Free Expert System Paradigms

The current landscape of L2RPN solutions is dominated by two primary paradigms\. First, extensive research has focused on model\-free RL, particularly Proximal Policy Optimization \(PPO\)\([6](https://arxiv.org/html/2608.14114#bib.bib29);[19](https://arxiv.org/html/2608.14114#bib.bib30);[30](https://arxiv.org/html/2608.14114#bib.bib7)\)\. These methods have been iteratively refined with heuristics and curriculum frameworks, such as the Teacher\-Tutor\-Junior\-Senior \(TTJS\) framework\([12](https://arxiv.org/html/2608.14114#bib.bib28)\)\. Notable examples include the 2023 winners, La Javaness\([7](https://arxiv.org/html/2608.14114#bib.bib27)\)and Artelys\([15](https://arxiv.org/html/2608.14114#bib.bib25)\), who utilized optimized PPO variants\.

Second, several successful approaches utilize expert systems or combine them with brute\-force methods\. These non\-RL solutions have proven to be highly competitive, achieving multiple top\-three placements in recent challenges\([23](https://arxiv.org/html/2608.14114#bib.bib24);[1](https://arxiv.org/html/2608.14114#bib.bib1);[13](https://arxiv.org/html/2608.14114#bib.bib2)\)\. While these findings established a robust benchmark for reactive agents, they remain confined to model\-free or rule\-based paradigms and do not account for the look\-ahead capabilities inherent in tree\-search methods\.

### 2\.3\.Transition to Model\-Based Approaches

The limitations of reactive agents have catalyzed interest in model\-based frameworks, specifically those inspired by the AlphaZero framework which utilizes Monte Carlo Tree Search \(MCTS\)\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)\. By simulating potential sequences of actions, these agents can proactively plan stable grid configurations\. The progression of top\-performing methodologies is summarized in Table[1](https://arxiv.org/html/2608.14114#S2.T1)\. This approach emerged as a dominant force in recent challenge iterations, achieving third and first place respectively in the 2021 ICAPS and 2022 WCCI competitions\.

Table 1\.Historical evolution of top\-performing agents and methodologies in the Learning to Run a Power Network \(L2RPN\) challenges\. The progression shows a shift from early model\-free RL toward specialized architectures involving Curriculum Learning and model\-based Tree Search\.While the 2022 WCCI winner successfully employed MCTS, a significant research gap remains regarding the sensitivity of such agents to specific design factors\. Currently, there is a lack of publicly available codebases for AlphaZero\-inspired grid controllers, and a significant research gap exists regarding the sensitivity of such model\-based agents to design factors such as reward shaping, MCTS guidance methods, and observation space configurations\. This paper builds upon these foundations by providing a systematic guide to effectively implementing AlphaZero controllers in safety\-critical grid environments\.

## 3\.Methodology

For this work, we use the Grid2op\([22](https://arxiv.org/html/2608.14114#bib.bib8)\)framework as intended for the L2RPN competition\. The experiments are conducted on the IEEE\-14 Grid, a standard benchmark used in the first iteration of the challenge\([22](https://arxiv.org/html/2608.14114#bib.bib8);[30](https://arxiv.org/html/2608.14114#bib.bib7)\)\. This grid consists of 14 substations, 20 lines, 3 transformers and 5 generators\. We utilize 1004 chronics, each 8064 time steps long\. To ensure generalization, performance for Rainbow\-PPO and AlphaZero is measured on a held\-out set of10%10\\%of the chronics\.

### 3\.1\.Non\-AlphaZero Baselines

We retrace the progression of L2RPN agents to establish a variation of baselines:

- •DQN and PPO: Implemented via the Stable\-Baselines3 library\([25](https://arxiv.org/html/2608.14114#bib.bib22)\)\. PPO is evaluated in the Redispatch \(continuous\) and Topology \(discrete\) paradigms\.
- •Curriculum Learning \(TTJS\): Integrates a Teacher\-Tutor\-Junior\-Senior framework\([27](https://arxiv.org/html/2608.14114#bib.bib23);[17](https://arxiv.org/html/2608.14114#bib.bib26)\)\. The agent is initialized via Imitation Learning before transitioning to active RL\. \(See Appendix[A\.1](https://arxiv.org/html/2608.14114#A1.SS1)\)\.
- •Rainbow PPO: An optimized PPO agent utilizing the RLlib library\([30](https://arxiv.org/html/2608.14114#bib.bib7)\)\. This includes adapted reward functions and supporting heuristics, serving as our state\-of\-the\-art model\-free baseline\.

### 3\.2\.AlphaZero and MCTS Approach

Our primary approach adapts the AlphaZero framework, utilizing Monte Carlo Tree Search \(MCTS\) to navigate the grid’s combinatorial topology\([26](https://arxiv.org/html/2608.14114#bib.bib13);[11](https://arxiv.org/html/2608.14114#bib.bib12)\)\. In the search tree, nodes represent critical states where grid load exceeds an activation threshold\.

MCTS Guidance and Configuration: The choice of these three specific search guidance variants is explicitly motivated by the need to isolate the underlying source of control intelligence within a physical, physics\-governed domain:

1. \(1\)The Imitation Learning \(IL\) Variant \(no prior guidance\):Evaluates the baseline efficacy of pure look\-ahead search depth and structural discovery, independent of a learned value landscape\.
2. \(2\)The Original Heuristic Approach:Examines the performance gains achieved by narrow, domain\-specific physics heuristics that act as hardcoded rules to evaluate leaf states\.
3. \(3\)The Learned Q\-Function setup:Tests the classical AlphaZero paradigm to determine whether a parameterized neural network can effectively approximate the highly non\-linear, stochastically driven boundaries of power grid stability better than hardcoded rules\.

The specific settings for thresholds and search limits are detailed in the Appendix[A](https://arxiv.org/html/2608.14114#A1)\.

##### Action Space Ablations

To maintain a manageable search space, we evaluate three action reduction strategies: Symmetry \(SYM\), \(N−0N\-0\)\([29](https://arxiv.org/html/2608.14114#bib.bib34)\), and \(N−1N\-1\)\([9](https://arxiv.org/html/2608.14114#bib.bib5)\)reductions\. Their impact on action space size is summarized in Table[2](https://arxiv.org/html/2608.14114#S3.T2)\.

Table 2\.Impact of Reduction Strategies on Action Space Size for IEEE\-14
##### Observation Space Configurations

We also evaluate four incremental observation feature sets, ranging from Minimal \(line loads only\) to Complete \(full telemetry\), as detailed in Table[3](https://arxiv.org/html/2608.14114#S3.T3)\.

Table 3\.Overview of Observation Space Configurations
##### Reward Function Configurations

A critical factor in MCTS efficiency is the alignment of the reward signal with the search objective\. We implement six distinct reward functions categorized by their complexity and goal orientation: \(1\) the originalAlphaZero Reward\([11](https://arxiv.org/html/2608.14114#bib.bib12)\), \(2\) the binaryD3QN\-2022 Survival Reward\([8](https://arxiv.org/html/2608.14114#bib.bib20)\), \(3\) the multi\-objectiveD3QN\-2020 Composite Reward\([34](https://arxiv.org/html/2608.14114#bib.bib19)\), \(4\) an efficiency\-basedLoss Reward\([32](https://arxiv.org/html/2608.14114#bib.bib16)\), and \(5\) linear safety\-margin rewards \(MaxRhoandPPO Reward\)\([14](https://arxiv.org/html/2608.14114#bib.bib21)\)\. Detailed mathematical formulations for each are provided in Appendix[A\.3](https://arxiv.org/html/2608.14114#A1.SS3)\.

##### MCTS Guidance Variants

To optimize the search efficiency of the AlphaZero agent, we implement three distinct methods for guiding the MCTS process: an Imitation Learning \(IL\) Variant, an Original Heuristic Approach, and a Learned Q\-Function setup\. These variants differ in how they prioritize node selection and evaluate leaf nodes\. The IL Variant operates without a prior learned policy or value function, relying exclusively on search depth and transition rewards to identify surviving trajectories\. This essentially treats the MCTS as a systematic discovery mechanism\. Full descriptions of each approach are provided in Appendix[A\.4](https://arxiv.org/html/2608.14114#A1.SS4)\.

##### Baseline Configuration

To compare the impact of different design choices, we define a Baseline AlphaZero Configuration consisting of a0\.980\.98activation threshold, a maximum of 250 MCTS simulations per step, and the original AlphaZero reward function\. All subsequent ablation studies on rewards, observations, and actions are compared against this standard\. Detailed parameters for search limits and safety heuristics are provided in the Appendix[B\.2](https://arxiv.org/html/2608.14114#A2.SS2)\.

## 4\.Experiments

### 4\.1\.Comparative Performance and Training Efficiency

In this section, we benchmark the AlphaZero\-inspired approach against established RL paradigms to validate its competitiveness and analyze the trade\-offs between survival capability and computational overhead\. The performance of each agent’s is assessed based on the ability to maintain grid stability over the maximum horizon of 8,064 steps\.

##### The Model\-Free Landscape

As shown in Table[4](https://arxiv.org/html/2608.14114#S4.T4), vanilla DQN and PPO\-Redispatch fail to navigate the complex grid constraints, surviving only 1\.1% and 20\.7% of the horizon, respectively\. Transitioning to a topological action space \(PPO Topological\) doubles performance, while Curriculum Learning \(TTJS\) provides a significant jump to 51\.2%\. These results confirm that topological reconfiguration is the superior tool for grid management\.

Table 4\.Peak performances of the different RL approaches compared to the Do Nothing agent\. For the PPO Rainbow, AlphaZero Baseline and Alphzero D3QN\-2022 we averaged the peaks over 4 runs\. Additionally, we can also see the average standard deviation over the 101 training chronics for these\.Figure 1\.Comparison of AlphaZero with the different PPO variants\. The results for the AlphaZero baseline described in[subsection B\.2](https://arxiv.org/html/2608.14114#A2.SS2), the AlphaZero with the D3QN\-2022 reward function as well as the PPO rainbow variant are evaluated over 4 runs and plotted with their standard deviation using a sliding window for smoothing\.
##### The AlphaZero Advantage

Our AlphaZero baseline achieves 92\.83% survivability, rivaling the state\-of\-the\-art Rainbow PPO \(91\.80%\)\. Crucially, by integrating the D3QN\-2022 binary reward, AlphaZero reaches a peak survivability of 98\.43%, effectively solving the IEEE\-14 benchmark\.

##### The Efficiency Trade\-off

The average survived steps are shown in[Figure 1](https://arxiv.org/html/2608.14114#S4.F1)\. The baseline AlphaZero agent achieves an average peak performance of 7486 which is comparable to the Rainbow PPO agent from\([30](https://arxiv.org/html/2608.14114#bib.bib7)\)which achieved 7403 steps\. Although AlphaZero provides superior reliability, it requires significantly more training steps \(∼\\sim40M\) compared to Rainbow PPO \(∼\\sim100k\)\. This suggests that while MCTS is highly effective for “running” the network, its “learning” phase is computationally intensive and benefits greatly from the design optimizations explored in the next section\.

As demonstrated in[Table 4](https://arxiv.org/html/2608.14114#S4.T4), modifying the AlphaZero design by adopting the simple binary reward from the D3QN approach\([8](https://arxiv.org/html/2608.14114#bib.bib20)\)significantly enhanced the performance, reaching an average peak of 7937 steps \(98\.43% survivability\)\. Beyond pure performance, this modification notably improves training efficiency, allowing the agent to converge more stably\. This result not only significantly outperforms the state\-of\-the\-art Rainbow PPO baseline but also confirms that objective alignment through sparse, survival\-oriented signals is a critical factor for effective MCTS\-based control\. These findings provide a concrete path for improving upon the original baseline AlphaZero agent as suggested in\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)\.

### 4\.2\.Systematic Design Guidelines for Effective AlphaZero

To provide actionable insights for implementing AlphaZero\-inspired controllers, we perform a series of ablation studies on the core architectural components\.

##### Objective Alignment \(Reward Functions\)

Contrary to the intuition that complex, multi\-objective rewards provide better gradients, our results in[Figure 2](https://arxiv.org/html/2608.14114#S4.F2)show that he simple binary survival rewardRD3QN\-2022R\_\{\\text\{D3QN\-2022\}\}significantly outperforms all others, achieving a peak of 7937 steps\. Complex rewards like D3QN\-2020 and Loss\-based efficiency signals introduced noise and slowed convergence, reaching an averaged peak performance of 4484 steps and the Loss reward showed much slower training speed as they failed to reach the 40 million training steps in the given time frame\. Competitive results were reached by the originally proposed AlphaZero as well as the Loss reward as they achieved a peak performance of 7485 and 6705 respectively\.

Figure 2\.Training progression of the AlphaZero agent using the different reward functions presented in Appendix[A\.3](https://arxiv.org/html/2608.14114#A1.SS3)\. The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation\.
##### Navigation of Action Space

Surprisingly, the most restrictive reduction strategy \(N\-1\) did not yield the best results\. The Symmetry \(SYM\) reduction, which maintains the highest degree of topological flexibility while removing redundant actions, proved to be the fastest and most stable \([Figure 3](https://arxiv.org/html/2608.14114#S4.F3)\)\. MCTS is robust enough to handle moderate action spaces; over\-restricting the topology \(e\.g\., N\-0 or N\-1\) can limit the agent’s ability to find “survival paths” in extreme congestion\.

Figure 3\.Training progression of the AlphaZero agent using the different action spaces presented in Section[3\.2](https://arxiv.org/html/2608.14114#S3.SS2)\. The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation\.
##### Information Density \(Observation Space\)

Our analysis shows that the Minimal observation space configuration, which uses only line loads, consistently outperformed configurations with complete telemetry\. The result yields the fastest learning rate and the highest training stability, as can be found in[Figure 4](https://arxiv.org/html/2608.14114#S4.F4)\. Its peak performance achieved a survival time of 7340 steps\. The Custom, Reduced and Complete configurations achieve similar peak performances with 7285, 7328 and 7287 steps, respectively\. However, they all learn slower and are less consistent\. Providing full nodal injections and voltage phase angles appeared to overcomplicate the state\-action mapping, leading to slower learning and inconsistency\.

Figure 4\.Training progression of the AlphaZero agent using the different observation spaces presented in Section[3\.2](https://arxiv.org/html/2608.14114#S3.SS2)\. The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation\.
##### The Guidance Paradox

The simplified no\-guidance variant, which uses neither a prior policy nor a value\-function guidance, achieved strong performance with the highest training efficiency\. It reached an average peak survival of 7220 steps within only 25 million training steps\. In contrast, while the average peak performance is compatible with 7250 steps, the original heuristic\-based version required a lot more training steps to achieve strong performance\. Finally, the integration of a learned Q\-function proved to be the least effective\. This configuration resulted in an extremely slow execution runtime as even with an increased runtime of 72 hours, it only reached 20\-30 million training steps due to the additional overhead\. While the performance for the achieved number of training steps seems competitive with the other variants, the extremely long run\-time makes this approach unattractive\. Unbiased MCTS search, when paired with an effective reward signal, is often more efficient than search guided by potentially inaccurate or high\-latency learned functions\. The results are shown in[Figure 5](https://arxiv.org/html/2608.14114#S4.F5)\.

Figure 5\.Comparison of the three MCTS guidance methods of using learned prior and value function, learned prior and heuristic value function or no guidance at all\.

## 5\.Discussion

This section analyzes the design principles for the AlphaZero inspired controller, identifying the relationship between tree\-search mechanics and the physical constraints of the power system, including reward structures, action space reduction, and observation density\.

##### Optimizing Information Density \(Observation Space\)

Our findings demonstrate that AlphaZero agent performance is inversely proportional to observation complexity, with the minimal configuration \(line loads only\) consistently yielding the fastest convergence and highest training stability\. This result challenges the intuitive assumption that denser telemetry, such as voltage phase angles, active power injections, and local demand profiles, inherently enables superior grid control\. One hypothesis for this behavior is that expanding the observation space drastically increases the state\-space dimensionality, introducing what can be termed as “feature noise”\. In the context of topological reconfiguration, line loading acts as the most direct physical proxy for systemic grid stress and imminent thermal failure\. By saturating the neural network with granular, high\-dimensional inputs, the model risks overfitting to stochastic demand fluctuations and localized noise rather than learning the structural bottleneck features critical for long\-term survival\.

This interpretation aligns with recent findings in Reinforcement Learning for Optimal Power Flow \(OPF\), where it has been shown that over\-complicating or duplicating telemetry features during the training phase can yield highly counterintuitive results and degrade policy optimization\([5](https://arxiv.org/html/2608.14114#bib.bib32)\)\. Rather than providing a richer context, excessive feature density complicates the state\-action mapping for the neural network, ultimately impeding the value propagation within the MCTS framework\. Therefore, we recommend a more conservative, simulation\-verified approach to feature selection in safety\-critical grid domains, prioritizing core physical constraints over complete telemetry saturation\.

##### Objective Alignment and Sparse Signals \(Reward Function\)

Contrary to our observations regarding the observation space, the results visualized in[Figure 2](https://arxiv.org/html/2608.14114#S4.F2)show that the superiority of the binary survival reward \(RD3QN\-2022R\_\{\\text\{D3QN\-2022\}\}\) over complex, load\-based rewards \(MaxRho, D3QN\-2020\) highlights a critical “Objective Alignment” challenge in MCTS\. Dense rewards attempt to guide the search via a smooth gradient, but in safety\-critical environments, they often introduce conflicting subgoals, e\.g\., optimizing for efficiency at the cost of stability\.

While the energy loss\-based and the original AlphaZero reward \(based on maximal line load and the number of lines that are offline\) were effective, the simple binary reward proved superior in both training speed and final performance\. By providing a clear “pass/fail” signal, the agent avoided distraction from possible subgoals\. Additionally, this strategy aligns effectively with the MCTS tree algorithm, as it provides a clean, unambiguous signal for value propagation; especially since the final action selection in MCTS is purely based on the maximum steps reached within the tree\. In contrast, other reward structures involving continuous or penalty\-heavy metrics performed poorly, likely because they introduced a “dense but noisy” gradient\. MCTS is a discovery\-driven algorithm; if the agent is penalized for minor line\-load fluctuations that do not lead to failure, the search may prune branches that contain the decisive topological maneuver needed for long\-term survival\. A binary signal ensures that the value propagation within the search tree remains focused solely on the terminal survival objective, preventing the distraction observed with penalty\-heavy reward structures\.

##### Action Space Flexibility vs\. Over\-Pruning

For the IEEE\-14 grid, we found that strict action space reduction is counterproductive\. The Symmetry \(SYM\) reduction strategy, which retains 203 actions, outperformed more restrictiveN−1N\-1strategies\. This suggests that the combinatorial explosion of a small grid is not yet a bottleneck for MCTS\. Instead, the bottleneck is the loss of reachability: over\-pruning removes the unconventional busbar configurations that are often necessary to reroute flow during extreme contingencies\. While larger grids will eventually require stricter dimensionality management, these results suggest that MCTS’s inherent search capability is robust enough to handle moderate action spaces without losing efficiency\.

##### The Guidance Paradox: Search vs\. Prior Policy

The evaluation of guidance variants reveals a Guidance Paradox: the no guidance variant was the most training\-efficient\. In traditional AlphaZero, a prior policy is essential to prune the tree\. However, in the power grid domain where state transitions are governed by physical power flow, the rewards collected directly within the search tree are often more reliable than a nascent, learnedQQ\-function\. While a learned prior can stabilize the agent in the long run, the computational overhead of frequent neural network updates and the risk of policy collapse, where a poor early policy misguides the search, make unbiased MCTS a more effective starting point for autonomous grid controllers\.

## 6\.Conclusion and Outlook

This study provides a systematic evaluation of AlphaZero\-inspired topological control, demonstrating that while the framework is highly capable, pure RL is not sufficient for the operational demands of modern power grids\. Our results highlight that achieving a peak survivability of 98\.43% on the IEEE\-14 benchmark is not a result of increased model complexity, but rather of strategic simplification and the integration of domain\-specific heuristics\.

We have established a minimalist design blueprint for effective AlphaZero implementation: utilizing a minimal observation space \(line loads only\) and a binary survival\-based reward\. These choices ensure objective alignment between the neural network and the Monte Carlo Tree Search \(MCTS\), preventing the agent from being distracted by the “feature noise” and conflicting subgoals inherent in multi\-objective reward structures\. Crucially, this blueprint constrains the agent exclusively to topological actions\. In real\-world deployment, grid operators would prioritize these topological reconfigurations because they do not incur direct market costs\. However, a production\-grade controller would treat topology as the first line of defense to minimize expensive redispatch, falling back on generation adjustments only when topological maneuvers are exhausted\.

While AlphaZero successfully outperformed the state\-of\-the\-art Rainbow PPO in survival time, this gain comes with a significant trade\-off in training efficiency\. The superior performance of our search\-based adaptation, alongside the historical success of non\-RL expert systems in the L2RPN challenges, suggests that the “intelligence” in grid control may lie more in the search and physics\-based heuristics than in the reinforcement learning policy itself\. This raises fundamental questions for the community: as we move toward larger grids, should we prioritize learning complex policies, or should we focus on optimizing the efficiency of the search and the accuracy of the underlying heuristics?

To bridge the gap between high\-performance research and real\-world grid deployment, future work must prioritize several key areas\. First, addressing the computational overhead of MCTS is essential; integrating AlphaZero into a Curriculum Learning framework, where the agent masters small\-scale topological maneuvers before tackling cascading failures, could significantly accelerate convergence\. Furthermore, an Options Framework could allow agents to treat complex bus\-switching sequences as temporally extended abstract actions, enabling reasoning over high\-level grid states rather than individual busbar changes\.

Scalability and transparency also remain primary concerns for larger networks\. In general, the importance of reflecting domain knowledge in feature design has been shown for RL applications in OPF\([31](https://arxiv.org/html/2608.14114#bib.bib35)\), which provides valuable input for optimizing our minimalist observation framework in future work\. Adopting Multi\-Agent Reinforcement Learning \(MARL\) or hierarchical frameworks would allow modular AlphaZero instances to manage regional substations, reducing the global action space to a manageable level\([3](https://arxiv.org/html/2608.14114#bib.bib3);[18](https://arxiv.org/html/2608.14114#bib.bib4);[9](https://arxiv.org/html/2608.14114#bib.bib5)\)\. Finally, for autonomous agents to be adopted by grid operators, their decisions must be interpretable\. Future research should prioritize Explainable RL \(XRL\) techniques, such as SHAP or LIME, to provide human\-interpretable justifications for topological reconfigurations\([5](https://arxiv.org/html/2608.14114#bib.bib32);[4](https://arxiv.org/html/2608.14114#bib.bib33)\)\. This transparency is vital for transforming a “black\-box” controller into a trusted decision\-support tool, ultimately leading to a hybrid approach that respects the physical laws of the system through heuristics and look\-ahead search while using RL to refine decision efficiency\.

###### Acknowledgements\.

We gratefully acknowledge funding from the Helmholtz Association under grant No\. VH\-NG\-1727 and the Networking Fund through Helmholtz AI\. We also thank EnBW Research Department for their valuable support/discussion\. The authors acknowledge support by the state of Baden\-Württemberg through bwHPC\.

## References

- Alibaba Research \(2022\)Alibaba ResearchL2RPN WCCI 2022 Competition Repository\.Note:GitHub[https://github\.com/AlibabaResearch/l2rpn\-wcci\-2022/tree/main](https://github.com/AlibabaResearch/l2rpn-wcci-2022/tree/main), Accessed 08 May 2026Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p2.1)\.
- Bengioet al\.\(2009\)Y\. Bengio, J\. Louradour, R\. Collobert, and J\. WestonCurriculum learning\.InProceedings of the 26th annual international conference on machine learning,pp\. 41–48\.Cited by:[2nd item](https://arxiv.org/html/2608.14114#S2.I1.i2.p1.1)\.
- Boguslawskiet al\.\(2024\)E\. Boguslawski, A\. Leite, M\. Dussartre, B\. Donnot, and M\. SchoenauerEmulation of Zonal Controllers for the Power System Transport Problem\.RJCIA41\.Cited by:[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Buttet al\.\(2024\)H\. S\. Butt, Q\. Huang, and B\. SchäferExplainable reinforcement learning for optimizing electricity costs in building energy management\.In2024 3rd International Conference on Energy Transition in the Mediterranean Area \(SyNERGY MED\),pp\. 1–6\.Cited by:[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Butt and Schafer \(2024\)H\. S\. Butt and B\. SchaferWhy reinforcement learning in energy systems needs explanations\.InProceedings of the 2024 Workshop on Explainability Engineering,pp\. 26–30\.Cited by:[§5](https://arxiv.org/html/2608.14114#S5.SS0.SSS0.Px1.p2.1),[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Chauhanet al\.\(2023\)A\. Chauhan, M\. Baranwal, and A\. BasumataryPowRL: a reinforcement learning framework for robust management of power networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 14757–14764\.Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1)\.
- D\. A\. E\. Lab \(2023\)D\. A\. E\. LabLearning to Run a Power Network \- Delft 2023 Competition\.Note:[https://codalab\.lisn\.upsaclay\.fr/competitions/12420](https://codalab.lisn.upsaclay.fr/competitions/12420)CodaLab; Accessed 08 May 2026Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1)\.
- Damjanovicet al\.\(2022\)I\. Damjanovic, I\. Pavić, M\. Puljiz, and M\. BrcicDeep reinforcement learning\-based approach for autonomous power flow control using only topology changes\.Energies15\(19\),pp\. 6920\.Cited by:[2nd item](https://arxiv.org/html/2608.14114#A1.I2.i2.p1.1),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.14114#S4.SS1.SSS0.Px3.p2.1)\.
- de Molet al\.\(2025\)B\. de Mol, D\. Barbieri, J\. Viebahn, and D\. GrossiCentrally coordinated multi\-agent reinforcement learning for power grid topology control\.InProceedings of the 16th ACM International Conference on Future and Sustainable Energy Systems,pp\. 460–475\.Cited by:[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Doornhof \(2026\)J\. DoornhofRenewable curtailment compensation costs in germany decrease 22% in 2025\.Note:[https://www\.cleanenergywire\.org/news/renewable\-curtailment\-compensation\-costs\-germany\-decrease\-22\-2025](https://www.cleanenergywire.org/news/renewable-curtailment-compensation-costs-germany-decrease-22-2025)Accessed 01 May 2026Cited by:[§1](https://arxiv.org/html/2608.14114#S1.p2.1)\.
- Dorferet al\.\(2022\)M\. Dorfer, A\. R\. Fuxjäger, K\. Kozak, P\. M\. Blies, and M\. WassererPower grid congestion management via topology optimization with AlphaZero\.arXiv preprint arXiv:2211\.05612\.Cited by:[1st item](https://arxiv.org/html/2608.14114#A1.I2.i1.p1.1),[§A\.2](https://arxiv.org/html/2608.14114#A1.SS2.p2.1),[§1](https://arxiv.org/html/2608.14114#S1.p2.1),[§1](https://arxiv.org/html/2608.14114#S1.p4.1),[§1](https://arxiv.org/html/2608.14114#S1.p5.1),[§2\.3](https://arxiv.org/html/2608.14114#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.6.4),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2608.14114#S4.SS1.SSS0.Px3.p2.1)\.
- H\. T\. EI Innovation Lab, Huawei Cloud \(2020\)H\. T\. EI Innovation Lab, Huawei CloudNeurIPS Competition 2020: Learning to Run a Power Network \(L2RPN\) \- Robustness Track\.Note:[https://github\.com/leizhu0608/L2RPN\-2020\-Robustness\-Track\-Solution](https://github.com/leizhu0608/L2RPN-2020-Robustness-Track-Solution)Accessed 08 May 2025Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1)\.
- Honda Research Institute Europe \(2022\)Honda Research Institute EuropeL2RPN Trial and Error \(TAE\) Agent\.Note:GitHub[https://github\.com/HRI\-EU/l2rpn\_tae\_agent](https://github.com/HRI-EU/l2rpn_tae_agent), Accessed 08 May 2026Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p2.1)\.
- La Javaness \(2026\)La JavanessL2RPN with PPO: a winning solution for the L2RPN challenge\.Note:GitHub[https://github\.com/lajavaness/l2rpn\-with\-ppo](https://github.com/lajavaness/l2rpn-with-ppo), Accessed 10 March 2026Cited by:[5th item](https://arxiv.org/html/2608.14114#A1.I2.i5.p1.1),[6th item](https://arxiv.org/html/2608.14114#A1.I2.i6.p1.1),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px3.p1.1)\.
- Lairet al\.\(2023\)N\. Lair, A\. Bossavy, P\. Champion, and V\. RenaultArtificial agents designed to run a power network \- White Paper\.Note:Artelys[https://www\.artelys\.com/app/uploads/2024/04/White\_paper\_L2RPN\_2023\_\.pdf](https://www.artelys.com/app/uploads/2024/04/White_paper_L2RPN_2023_.pdf), Accessed 08 May 2026Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1)\.
- Lanet al\.\(2020\)T\. Lan, J\. Duan, B\. Zhang, D\. Shi, Z\. Wang, R\. Diao, and X\. ZhangAI\-based autonomous line flow control via topology adjustment for maximizing time\-series atcs\.In2020 IEEE Power & Energy Society General Meeting \(PESGM\),pp\. 1–5\.Cited by:[3rd item](https://arxiv.org/html/2608.14114#S2.I1.i3.p1.1),[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.2.4)\.
- Lehnaet al\.\(2023\)M\. Lehna, J\. Viebahn, A\. Marot, S\. Tomforde, and C\. ScholzManaging power grids through topology actions: a comparative study between advanced rule\-based and reinforcement learning agents\.Energy and AI14,pp\. 100276\.Cited by:[§A\.1](https://arxiv.org/html/2608.14114#A1.SS1.p1.1),[§1](https://arxiv.org/html/2608.14114#S1.p2.1),[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.7.4),[2nd item](https://arxiv.org/html/2608.14114#S3.I1.i2.p1.1)\.
- Losapioet al\.\(2024\)G\. Losapio, D\. Beretta, M\. Mussi, A\. M\. Metelli, and M\. RestelliState and Action Factorization in Power Grids\.arXiv preprint arXiv:2409\.04467\.Cited by:[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Manczaket al\.\(2023\)B\. Manczak, J\. Viebahn, and H\. van HoofHierarchical reinforcement learning for power network topology control\.arXiv preprint arXiv:2311\.02129\.Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1)\.
- Marotet al\.\(2022\)A\. Marot, B\. Donnot, K\. Chaouache, A\. Kelly, Q\. Huang, R\. Hossain, and J\. L\. CremerLearning to run a power network with trust\.Electric Power Systems Research212,pp\. 108487\.Cited by:[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.5.4)\.
- Marotet al\.\(2021\)A\. Marot, B\. Donnot, G\. Dulac\-Arnold, A\. Kelly, A\. O’Sullivan, J\. Viebahn, M\. Awad, I\. Guyon, P\. Panciatici, and C\. RomeroLearning to run a power network challenge: a retrospective analysis\.InNeurIPS 2020 competition and demonstration track,pp\. 112–132\.Cited by:[4th item](https://arxiv.org/html/2608.14114#S2.I1.i4.p1.1)\.
- Marotet al\.\(2020\)A\. Marot, B\. Donnot, C\. Romero, B\. Donon, M\. Lerousseau, L\. Veyrin\-Forrer, and I\. GuyonLearning to run a power network challenge for training topology controllers\.Electric Power Systems Research189,pp\. 106635\.Cited by:[§1](https://arxiv.org/html/2608.14114#S1.p3.1),[§2](https://arxiv.org/html/2608.14114#S2.p1.1),[§3](https://arxiv.org/html/2608.14114#S3.p1.1)\.
- Martinez \(2021\)H\. MartinezL2RPN\.Note:GitHub[https://github\.com/horacioMartinez/L2RPN](https://github.com/horacioMartinez/L2RPN), Accessed 08 May 2026Cited by:[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p2.1)\.
- Matavalamet al\.\(2022\)A\. R\. R\. Matavalam, K\. P\. Guddanti, Y\. Weng, and V\. AjjarapuCurriculum based reinforcement learning of grid topology controllers to prevent thermal cascading\.IEEE Transactions on Power Systems38\(5\),pp\. 4206–4220\.Cited by:[1st item](https://arxiv.org/html/2608.14114#S2.I1.i1.p1.1)\.
- RTE\-France \(2026\)RTE\-FranceL2RPN baselines: a repository to host baselines for L2RPN competitions\.Note:GitHub[https://github\.com/rte\-france/l2rpn\-baselines](https://github.com/rte-france/l2rpn-baselines), Accessed 22 March 2026Cited by:[§1](https://arxiv.org/html/2608.14114#S1.p3.1),[1st item](https://arxiv.org/html/2608.14114#S3.I1.i1.p1.1)\.
- Silveret al\.\(2018\)D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel, T\. Lillicrap, K\. Simonyan, and D\. HassabisA general reinforcement learning algorithm that masters chess, shogi, and go through self\-play\.Science362\(6419\),pp\. 1140–1144\.External Links:[Document](https://dx.doi.org/10.1126/science.aar6404)Cited by:[§A\.2](https://arxiv.org/html/2608.14114#A1.SS2.p2.2),[§1](https://arxiv.org/html/2608.14114#S1.p5.1),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.p1.1)\.
- Sintes and Dang \(2024\)J\. Sintes and V\. T\. DangHow we built the winning real\-time autonomous agent for power grid management in the L2RPN challenge 2023\.Note:Medium[https://lajavaness\.medium\.com/how\-we\-built\-the\-winning\-real\-time\-autonomous\-agent\-for\-power\-grid\-management\-in\-the\-l2rpn\-41ab3cfaddbd](https://lajavaness.medium.com/how-we-built-the-winning-real-time-autonomous-agent-for-power-grid-management-in-the-l2rpn-41ab3cfaddbd), Accessed 08 May 2026Cited by:[§A\.1](https://arxiv.org/html/2608.14114#A1.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.7.4),[2nd item](https://arxiv.org/html/2608.14114#S3.I1.i2.p1.1)\.
- SMARD \(2024\)SMARDThe development of congestion management\.Note:[https://www\.smard\.de/page/en/topic\-article/212250/217910/the\-development\-of\-congestion\-management](https://www.smard.de/page/en/topic-article/212250/217910/the-development-of-congestion-management)Accessed 01 May 2026Cited by:[§1](https://arxiv.org/html/2608.14114#S1.p2.1)\.
- Subramanianet al\.\(2021\)M\. Subramanian, J\. Viebahn, S\. H\. Tindemans, B\. Donnot, and A\. MarotExploring grid topology reconfiguration using a simple deep reinforcement learning approach\.In2021 IEEE Madrid PowerTech,pp\. 1–6\.Cited by:[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px1.p1.1)\.
- van der Saret al\.\(2025\)E\. van der Sar, A\. Zocca, and S\. BhulaiOptimizing power grid topologies with reinforcement learning: a survey of methods and challenges\.Foundations and Trends in Electric Energy Systems9\(1\),pp\. 1–119\.Cited by:[§A\.3](https://arxiv.org/html/2608.14114#A1.SS3.p1.1),[§2\.2](https://arxiv.org/html/2608.14114#S2.SS2.p1.1),[§2](https://arxiv.org/html/2608.14114#S2.p1.1),[3rd item](https://arxiv.org/html/2608.14114#S3.I1.i3.p1.1),[§3](https://arxiv.org/html/2608.14114#S3.p1.1),[§4\.1](https://arxiv.org/html/2608.14114#S4.SS1.SSS0.Px3.p1.1)\.
- Wolgast and Nieße \(2024\)T\. Wolgast and A\. NießeLearning the optimal power flow: environment design matters\.Energy and AI17,pp\. 100410\.Cited by:[§6](https://arxiv.org/html/2608.14114#S6.p5.1)\.
- Yoonet al\.\(2021\)D\. Yoon, S\. Hong, B\. Lee, and K\. KimWinning the l2rpn challenge: power grid management via semi\-markov afterstate actor\-critic\.InInternational Conference on Learning Representations,Cited by:[4th item](https://arxiv.org/html/2608.14114#A1.I2.i4.p1.1),[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.3.4),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px3.p1.1)\.
- Young \(2025\)H\. YoungWhat the blackout in Spain, Portugal says about renewables\.Note:[https://www\.dw\.com/en/spain\-portugal\-blackout\-renewables\-wind\-solar\-energy\-grid\-v2/a\-72606531](https://www.dw.com/en/spain-portugal-blackout-renewables-wind-solar-energy-grid-v2/a-72606531)Accessed 08 March 2026Cited by:[§1](https://arxiv.org/html/2608.14114#S1.p1.1)\.
- Zhihonget al\.\(2020\)Y\. Zhihonget al\.A winning approach of NeurIPS 2020 L2RPN comp\.Note:GitHub[https://github\.com/lujasone/NeurIPS\_2020\_L2RPN\_Comp\_An\_Approach](https://github.com/lujasone/NeurIPS_2020_L2RPN_Comp_An_Approach), Accessed 10 March 2026Cited by:[3rd item](https://arxiv.org/html/2608.14114#A1.I2.i3.p1.1),[§3\.2](https://arxiv.org/html/2608.14114#S3.SS2.SSS0.Px3.p1.1)\.
- Zhouet al\.\(2021\)B\. Zhou, H\. Zeng, Y\. Liu, K\. Li, F\. Wang, and H\. TianAction set based policy optimization for safe power grid management\.InJoint European Conference on Machine Learning and Knowledge Discovery in Databases,pp\. 168–181\.Cited by:[Table 1](https://arxiv.org/html/2608.14114#S2.T1.5.4.4)\.

## Appendix AAppendix

### A\.1\.Curriculum Learning \(TTJS\) Framework

Using the same setup and hyperparameters as PPO Topology, we supplemented it with an additional Curriculum Learning setup as provided by the La Javaness\([27](https://arxiv.org/html/2608.14114#bib.bib23)\)combined with the Curriculum Agent\([17](https://arxiv.org/html/2608.14114#bib.bib26)\)repository\. These apply a Teacher\-Tutor\-Junior \(TTJS\) framework where the “Teachers” run a brute\-force search to collect a large amount of trajectories \(approximately 50GB\)\. The “Tutors” then filter these for high\-quality experiences to reduce the action space accordingly and initialize the Junior neural network via IL\. Only after this, does the Senior apply the actual RL utilizing the previously initialized neural network\.

### A\.2\.MCTS Mathematical Framework and Structure

Whenever the AlphaZero agent reaches a state with a maximin line load that is above our activation threshold, the iterative MCTS algorithm is triggered\. In each iteration, we traverse the tree from the root, which represents the current state of the power grid, until we reach the leaf layer, where a new node is then appended\. The traversal follows the Predictor Upper Confidence Bound applied to Trees \(PUCT\) formula:

\(1\)at=argmaxa​\(Q⁡\(s,a\)\+cp​u​c​t⋅P⁡\(s,a\)⋅∑bN⁡\(s,b\)1\+N⁡\(s,a\)\),a\_\{t\}=\\text\{argmax\}\_\{a\}\\left\(Q\(s,a\)\+c\_\{puct\}\\cdot P\(s,a\)\\cdot\\frac\{\\sqrt\{\\sum\_\{b\}N\(s,b\)\}\}\{1\+N\(s,a\)\}\\right\),where:

- •Q⁡\(s,a\)Q\(s,a\)is the action\-value \(the mean reward of choosing actionaain statess\)\.
- •P⁡\(s,a\)P\(s,a\)is the prior probability of choosing actionaaas provided by the policy network\.
- •N⁡\(s,a\)N\(s,a\)is the visit count for the specific edge, while ∑bN⁡\(s,b\)\\sum\_\{b\}N\(s,b\)is the total visit count of the parent node\.
- •cp​u​c​tc\_\{puct\}is a constant that controls the level of exploration\.

\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)suggests determining the value of the leaf nodes added to the MCTS using the heuristic function:

\(2\)vt=∑j=tt\+hγj−t​rjv\_\{t\}=\\sum\_\{j=t\}^\{t\+h\}\\gamma^\{j\-t\}r\_\{j\}wherehhis a sufficiently large horizon depending on the discount factorγ\\gamma\([26](https://arxiv.org/html/2608.14114#bib.bib13)\), and the rewardrjr\_\{j\}at the current time stepjj\. To manage computational complexity, an early stopping mechanism is included\. This mechanism terminates the search once a specific number of recovery nodes are identified\. Specifically, whents​k​i​p​p​e​dt\_\{skipped\}steps are skipped between critical nodes, a “recovery node” is added to mark an action that solved the congestion and led to a period of grid stability\. The search is interrupted oncetstoppingt\_\{\\text\{stopping\}\}recovery nodes have been identified or we have exhausted a predetermined budget of simulations\. The training target is then determined by evaluating which branch of the tree reached the greatest depth in terms of survival steps\. The action corresponding to this deepest subtree is selected as the training target for the current root state\. These state\-action pairs are appended to a replay buffer, which is used to update the neural network once a sufficient number of chronics have been processed to ensure a diverse and representative dataset\.

### A\.3\.MCTS Reward Function

Previous studies have employed a wide variety of reward functions\. In this work, we cover the most relevant categories, including safety margins, maximum line loading, survival time, and energy efficiency, as well as composite multi\-objective strategies\. The full mathematical definitions for the AlphaZero Reward, D3QN\-2022 Survival Reward, D3QN\-2020 Composite Reward, Loss Reward, MaxRho Reward, and PPO Reward are summarized in\([30](https://arxiv.org/html/2608.14114#bib.bib7)\)and given as follows:

- •AlphaZero Reward\([11](https://arxiv.org/html/2608.14114#bib.bib12)\)Originally utilized with the AlphaZero agent, this reward is designed to penalize high line loading and network fragmentation\. It uses an exponential decay function to provide a smooth gradient for the MCTS: \(3\)RAlphaZero=e−u⁡\(t\)−0\.5⋅nofflineR\_\{\\text\{AlphaZero\}\}=e^\{\-u\(t\)\-0\.5\\cdot n\_\{\\text\{offline\}\}\}The penalty termu⁡\(t\)u\(t\)is calculated based on the maximum line loading ratioρmax\\rho\_\{\\max\}: \(4\)u⁡\(t\)=\{max⁡\(ρmax−0\.5,0\)if​ρmax≤1∑i∈𝒪\(ρi−0\.5\)if​ρmax\>1u\(t\)=\\begin\{cases\}\\max\(\\rho\_\{\\max\}\-0\.5,0\)&\\text\{if \}\\rho\_\{\\max\}\\leq 1\\\\ \\sum\_\{i\\in\\mathcal\{O\}\}\(\\rho\_\{i\}\-0\.5\)&\\text\{if \}\\rho\_\{\\max\}\>1\\end\{cases\}where𝒪\\mathcal\{O\}denotes the set of overflowing lines \(i\.e\., all linesiifor whichρi\>1\\rho\_\{i\}\>1\)\. The termnofflinen\_\{\\text\{offline\}\}accounts for the number of lines currently disconnected\.
- •D3QN\-2022 Survival Reward\([8](https://arxiv.org/html/2608.14114#bib.bib20)\)A binary reward signal focusing exclusively on survival: \(5\)RD3QN\-2022=\{1if grid is operational at step​t0if game over or illegal actionR\_\{\\text\{D3QN\-2022\}\}=\\begin\{cases\}1&\\text\{if grid is operational at step \}t\\\\ 0&\\text\{if game over or illegal action\}\\end\{cases\}
- •D3QN\-2020 Composite Reward\([34](https://arxiv.org/html/2608.14114#bib.bib19)\) This is a multi\-objective reward function that balances grid stability, safety margins, and topological similarity: \(6\)RD3QN\-2020=w1⋅Rsandbox\+w2⋅Roverflow\+w3⋅Rdistance\+w4⋅Rcapacity\\begin\{split\}\\begin\{aligned\} R\_\{\\text\{D3QN\-2020\}\}=&\\ w\_\{1\}\\cdot R\_\{\\text\{sandbox\}\}\+w\_\{2\}\\cdot R\_\{\\text\{overflow\}\}\\\\ &\+w\_\{3\}\\cdot R\_\{\\text\{distance\}\}\+w\_\{4\}\\cdot R\_\{\\text\{capacity\}\}\\end\{aligned\}\\end\{split\}The weights arew1w\_\{1\}=30,w2w\_\{2\}=200,w3w\_\{3\}=20,w4w\_\{4\}=3\.Rsandbox​\(t\)R\_\{\\text\{sandbox\}\}\(t\)is based on the costs for redispatch and power loss, whileRoverflow​\(t\)R\_\{\\text\{overflow\}\}\(t\)rewards a lower magnitude of line overflows\.Rdistance​\(t\)R\_\{\\text\{distance\}\}\(t\)is determined by the distance from the reference topology \(where a smaller distance yields a higher reward\), andRcapacity​\(t\)R\_\{\\text\{capacity\}\}\(t\)rewards global safety margins based on the available line capacity\.
- •Loss Reward \(Efficiency\-Based\)\([32](https://arxiv.org/html/2608.14114#bib.bib16)\)Targets the minimization of active power losses: \(7\)RLoss​\(t\)=∑Ploads∑Pgens−0\.9R\_\{\\text\{Loss\}\}\(t\)=\\frac\{\\sum P\_\{\\text\{loads\}\}\}\{\\sum P\_\{\\text\{gens\}\}\}\-0\.9
- •MaxRho Reward \(LJN Agent\)\([14](https://arxiv.org/html/2608.14114#bib.bib21)\)A linear reward aimed at maximizing the safety margin: \(8\)RMaxRho​\(t\)=2\.0−ρmaxR\_\{\\text\{MaxRho\}\}\(t\)=2\.0\-\\rho\_\{\\text\{max\}\}
- •PPO Reward \(LJN Agent\)\([14](https://arxiv.org/html/2608.14114#bib.bib21)\)Combines the MaxRho principle with a progressive step\-penalty: \(9\)RPPO​\(t\)=\(2\.0−ρmax\)−penalty\+bonusactionR\_\{\\text\{PPO\}\}\(t\)=\(2\.0\-\\rho\_\{\\text\{max\}\}\)\-\\text\{penalty\}\+\\text\{bonus\}\_\{\\text\{action\}\}Thepenaltyterm is defined as: \(10\)penalty=\{0if​ρmax≤0\.81if​0\.8<ρmax≤0\.92if​0\.9<ρmax≤0\.953if​0\.95<ρmax≤1\.05if​ρmax\>1\.0\\text\{penalty\}=\\begin\{cases\}0&\\text\{if \}\\rho\_\{\\text\{max\}\}\\leq 0\.8\\\\ 1&\\text\{if \}0\.8<\\rho\_\{\\text\{max\}\}\\leq 0\.9\\\\ 2&\\text\{if \}0\.9<\\rho\_\{\\text\{max\}\}\\leq 0\.95\\\\ 3&\\text\{if \}0\.95<\\rho\_\{\\text\{max\}\}\\leq 1\.0\\\\ 5&\\text\{if \}\\rho\_\{\\text\{max\}\}\>1\.0\\end\{cases\}Abonusaction=0\.1\\text\{bonus\}\_\{\\text\{action\}\}=0\.1is applied for the “Do Nothing” action\.

### A\.4\.MCTS Guidance Variants: Detailed Descriptions

To optimize the search efficiency of the AlphaZero agent, we examine three distinct methods for guiding the MCTS process:

- •IL Variant:This configuration operates without using the learned policy or a value function within the MCTS\. It relies exclusively on the search depth and the transition rewards encountered during tree expansion to identify surviving trajectories, essentially treating the MCTS as systematic discovery mechanism\.
- •Original Heuristic Approach:This version uses the policy prior to narrow the search breadth by focusing on high\-probability actions\. Additionally, it employs the heuristic value function from[Equation 2](https://arxiv.org/html/2608.14114#A1.E2)to estimate the quality of newly added leaf nodes\.
- •Learned Q\-Function:In this setup, node selection and evaluation are driven by a combination of the prior policy and learned Q\-values\. By replacing the heuristic estimation with predictions by the neural network, the agent attempts to leverage learned experience to prioritize more promising search branches\.

## Appendix BMCTS

### B\.1\.MCTS Hyperparameter Configurations

To investigate the impact of search depth and efficiency, the following configurations for Early Stopping and Horizon size were evaluated:

##### Early Stopping Variants

- •Variant 1:tskipped=200,tstopping=50t\_\{\\text\{skipped\}\}=200,t\_\{\\text\{stopping\}\}=50
- •Variant 2:tskipped=200,tstopping=20t\_\{\\text\{skipped\}\}=200,t\_\{\\text\{stopping\}\}=20
- •Variant 3:tskipped=50,tstopping=10t\_\{\\text\{skipped\}\}=50,t\_\{\\text\{stopping\}\}=10
- •Variant 4:tskipped=10,tstopping=5t\_\{\\text\{skipped\}\}=10,t\_\{\\text\{stopping\}\}=5

##### Horizon Size and Reset Thresholds

We compared lookahead horizons of 20, 100, 200, and 500 steps applied to[Equation 2](https://arxiv.org/html/2608.14114#A1.E2)\. Additionally, the topology reset threshold was examined across values of 75%,80%80\\%,85%85\\%,90%90\\%, and95%95\\%defined by the maximum line load observed within the grid\.

Figure 6\.Evaluation of different horizon sizes for the heuristic based AlphaZero baseline approach\.Figure 7\.Evaluation of different thresholds for the application of the heuristic that leads the agent back to the reference topology\.Figure 8\.Evaluation of values for thets​t​o​p​p​i​n​gt\_\{stopping\}andts​k​i​p​p​e​dt\_\{skipped\}hyperparameters which define the early stopping mechanism for the MCTS tree\.Figure 9\.Evaluation of the effect caused by different choices for the batch size which determines the number of chronics we collect data from between training iterations\.
#### B\.1\.1\.MCTS Hyperparameters

The sensitivity of MCTS hyperparameters reveals a consistent trend: while final agent performance is relatively robust across configurations, parameter selection significantly impacts the stability and efficiency of the training process\. Overall, these results indicate that the primary role of MCTS hyperparameters tuning is not to lift the performance ceiling, but rather to enhance computational efficiency and training robustness\.

##### Search and Stopping Criteria

As shown in[Figure 8](https://arxiv.org/html/2608.14114#A2.F8), the medium early stopping configuration \(ts​k​i​p​p​e​dt\_\{skipped\}= 50 andts​t​o​p​p​i​n​gt\_\{stopping\}= 10\) avoided the training outliers seen in more aggressive or conservative variants\. An exception can be seen in the batch size experiment, where we learn that smaller batch sizes can lead to worse performance\. Our evaluation of early\-stopping parameters suggests that aggressive stopping criteria can be adopted to significantly reduce computational overhead without degrading final survival times\. Since the final survival times did not degrade under more aggressive early\-stopping criteria, these settings offer a clear path to increasing the real\-time feasibility of MCTS\-based controllers\. However, the superior stability of the “medium” configuration \(Variant 2\) indicates that some degree of search depth is necessary to prevent the high\-variance training spikes observed in the more minimal variants\.

##### Horizon and Thresholds

Regarding look\-ahead depth, shorter horizons facilitated a more consistent learning trajectory\. The longest horizon \(500 steps\) exhibited the pronounced instability, likely due to the diminishing validity of stationary reward assumptions over extended durations as grid dynamics evolve and stochasticity increases\. Furthermore, a topology reset threshold of 0\.85 emerged as the most robust setting\. Higher thresholds, which delay intervention until the grid is more severely stressed, resulted in more frequent performance “down\-spikes”, suggesting that overly delayed interventions lead to more volatile grid states\.

##### Batch Size Efficiency

Finally, the batch size experiments demonstrate that insufficiently small batches lead to degraded performance, likely due to a lack of gradient diversity\. While performance remained comparable across larger scales, the medium\-sized batches provided the optimal balance between wall\-clock training time and convergence stability\.

### B\.2\.Detailed Baseline Configuration

To compare all of the previously mentioned AlphaZero design options we will use a baseline configuration with the following settings:

- •Activation Threshold:0\.98 \(max\. line load of98%98\\%\)\.
- •Search Limits:Maximum of 250 MCTS simulations per step\.
- •Temporal Parameters:ts​k​i​p​p​e​dt\_\{skipped\}= 200 andts​t​o​p​p​i​n​gt\_\{stopping\}= 50\.
- •Reward Function:AlphaZero reward\.
- •Safety Heuristics:Automatic reconnection of a disconnected line after coldown and reversion to the initial reference topology when in safe state \(below 75% max\. line load\)\.
- •Observation Space:Minimal \- restricted to line loads only\.