SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning

# SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning
Source: [https://arxiv.org/html/2608.09967](https://arxiv.org/html/2608.09967)
Claudia Goldman2 1Benin School of Computer Science and Engineering, The Hebrew University of Jerusalem 2Hebrew University Business School, Data Science Department, The Hebrew University of Jerusalem \{tamar\.gozlan, claudia\.goldman\}@mail\.huji\.ac\.il

###### Abstract

Deep reinforcement learning \(DRL\) agents achieve strong performance in complex environments, yet their decision\-making processes remain difficult to interpret\. We introduceSPOT\(Sampling Policy Observation Tree\), a novel model\-agnostic, sampling\-based framework for interpreting DRL policies\. Given access to the policy and an environment simulator, SPOT constructs an interpretable finite\-horizon tree by sampling actions and recursively simulating the resulting successor states\. The tree provides an empirical representation of the policy’s action preferences and their possible downstream evolution\. We provide formal guarantees establishing SPOT’s asymptotic recovery of the policy’s unique most probable action and characterizing its disagreement behavior under high\-entropy policies\. We demonstrate SPOT in the SUMO\-RL traffic\-signal control domain\. The case study illustrates how its tree\-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single\-timestep feature\-attribution methods\.

## 1Introduction

Despite its success in complex decision\-making tasks, the application of deep reinforcement learning \(DRL\) in real\-world settings remains limited due to its reliance on large amounts of data, sensitivity to changing environments, and lack of interpretability \(e\.g\.,Lekadiret al\.\([2025](https://arxiv.org/html/2608.09967#bib.bib25)\)\)\. Existing explainable AI \(XAI\) methods for DRL largely focus on feature\-level attribution, identifying which inputs influence a given action\. While useful, these approaches are inherently limited to single timesteps and do not capture how decisions affect future states\. As a result, they fail to explain long\-term consequences or detect suboptimal decisions whose impact emerges over time\.

We introduceSPOT\(Sampling Policy Observation Tree\), a novel XAI framework that addresses these limitations by modeling the distribution of future trajectories induced by the agent’s policy\. SPOT constructs an interpretable tree through action sampling and state expansion, enabling trajectory\-aware explanations that explicitly compare the outcomes of alternative actions\.

We consider a human\-in\-the\-loop setting where the agent acts as a decision\-support system\. Explanations must therefore be actionable, enabling operators to assess whether to follow or override recommendations\. SPOT provides domain\-readable summaries that highlight key signals to support such decisions\.

We demonstrate SPOT in a SUMO\-RL traffic control environment, showing that it uncovers behaviors missed by single\-timestep explanations and enables more informed decisions\.

## 2Related Work

Deep reinforcement learning \(DRL\) combines reinforcement learningSutton and Barto \([2018](https://arxiv.org/html/2608.09967#bib.bib12)\)with deep neural networks to learn policies from high\-dimensional observations\. This integration allows RL to scale to complex environments by leveraging powerful function approximation capabilitiesDonget al\.\([2020](https://arxiv.org/html/2608.09967#bib.bib1)\); Arulkumaranet al\.\([2017](https://arxiv.org/html/2608.09967#bib.bib2)\)\.

### 2\.1Taxonomy of DRL Methods

DRL methods are commonly categorized into value\-based, policy\-based and actor\-critic methods\. This classification reflects different approaches to solving Markov Decision Processes \(MDPs\), where the goal is to maximize expected cumulative rewardDonget al\.\([2020](https://arxiv.org/html/2608.09967#bib.bib1)\)\. The MDP is given by the tuple<S,A,P,R,γ\><S,A,P,R,\\gamma\>, whereSSrepresents the state space,AA, the actions set,PPthe transition probabilities \(that are not explicitly known by the DRL solution\), the reward functionRRand a discount parameterγ\\gamma\.

#### Scope of Taxonomy\.

Our taxonomy focuses on*pure DRL*methods, where the core learning and decision\-making components are parameterized by deep neural networks\. Hybrid systems are excluded because their additional mechanisms fall outside a unified DRL taxonomy\. However, our explainability framework can still be applied to the DRL component within such systems\.

![Refer to caption](https://arxiv.org/html/2608.09967v1/DRL_methods_7.png)Figure 1:Taxonomy of deep reinforcement learning methods, categorized into value\-based, policy\-based, and actor\-critic approaches\. Based onDonget al\.\([2020](https://arxiv.org/html/2608.09967#bib.bib1)\)\.

### 2\.2Value\-Based Methods

Value\-based reinforcement learning methods aim to estimate value functions that quantify the expected return of states or state\-action pairs\. The state\-value functionV​\(s\)V\(s\)represents the expected cumulative reward when starting from statess, while the action\-value functionQ​\(s,a\)Q\(s,a\)represents the expected return of taking actionaain statessand following a given policy thereafter\.

Formally, the state\-value function, following a policyπ\\pi, is defined as:

Vπ​\(s\)=𝔼π​\[∑t=0∞γt​rt∣s0=s\],V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\\mid s\_\{0\}=s\\right\],and the action\-value function as:

Qπ​\(s,a\)=𝔼π​\[∑t=0∞γt​rt∣s0=s,a0=a\]\.Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\\mid s\_\{0\}=s,a\_\{0\}=a\\right\]\.
Q\-learning forms the foundation of modern value\-based reinforcement learning, where the optimal policy is obtained by selecting actions that maximize the learnedQQ\-function\. Most subsequent advancements build upon this core formulation, introducing improvements in stability and performance without fundamentally altering the underlying algorithmMcKenzie and McDonnell \([2022](https://arxiv.org/html/2608.09967#bib.bib5)\)\.

A major breakthrough occurred with the introduction of the Deep Q\-Network \(DQN\), which combines Q\-learning with deep neural networks to approximate value functions in high\-dimensional state spacesMnihet al\.\([2013](https://arxiv.org/html/2608.09967#bib.bib4)\)\. DQN enabled end\-to\-end learning from raw inputs and achieved human\-level performance on Atari benchmarks\. Following this milestone, a rich line of work has emerged extending DQN, with each variant designed to address specific shortcomings such as overestimation bias, inefficient sampling, or representation limitations\. Notable examples include Double DQNvan Hasseltet al\.\([2016](https://arxiv.org/html/2608.09967#bib.bib13)\), prioritized experience replay, and dueling architectures\. While these methods differ in their design objectives, they largely retain the core Q\-learning framework\.

### 2\.3Policy\-Based Methods

Policy\-based methods directly parameterize the policyπ​\(a\|s\)\\pi\(a\|s\)and optimize it using gradient\-based methods\. The theoretical foundation of policy gradient methods was established by the REINFORCE algorithmWilliams \([1992](https://arxiv.org/html/2608.09967#bib.bib3)\), which provides an unbiased estimator of the gradient of expected reward\.

∇θJ​\(θ\)=𝔼​\[\(r−b\)​∇θlog⁡πθ​\(a\|s\)\]\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\\left\[\(r\-b\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)\\right\]
This expression defines how the policy parametersθ\\thetashould be updated to maximize expected return\. The term∇θlog⁡πθ​\(a\|s\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)captures the sensitivity of the policy to parameter changes, indicating how a small adjustment inθ\\thetaaffects the probability of selecting actionaain statess\. The scalar term\(r−b\)\(r\-b\)represents the advantage of the action, measuring how much better or worse the observed returnrris compared to a baselinebb\.

Intuitively, this update rule increases the probability of actions that yield higher\-than\-expected returns and decreases the probability of actions that perform worse than expected\. Using the baselinebbreduces the variance of the gradient estimate, leading to more stable learning\.

In practice, modern implementations of policy gradient methods are often referred to as Vanilla Policy GradientSchulmanet al\.\([2017](https://arxiv.org/html/2608.09967#bib.bib7)\), which follows the same fundamental principle as REINFORCE but typically incorporates practical improvements such as batch updates, reward\-to\-go estimation, and normalization techniques to improve training stability and efficiency\.

A key insight of policy gradient methods is that they perform stochastic gradient ascent on the expected return without requiring an explicit model of the environment\. In expectation, the update direction aligns with the true gradient of the objective function, ensuring that the policy improves over time\.

### 2\.4Actor\-Critic Methods

Actor\-critic methods combine the advantages of value\-based and policy\-based approaches by maintaining two components: anactorthat represents the policy, and acriticthat estimates a value functionGrondmanet al\.\([2012](https://arxiv.org/html/2608.09967#bib.bib6)\)\. These approaches jointly learn a policy and a value function, unlike earlier methods that learn only one\. The critic evaluates the current policy by estimating the expected return, while the actor updates the policy parameters based on this feedback\. This interaction reduces the high variance inherent in policy gradient methods while retaining their favorable convergence properties\.

Typically, the policy update is guided by the temporal\-difference \(TD\) error which serves as an estimate of the advantage of the taken action:

δt=rt\+1\+γ​V​\(st\+1\)−V​\(st\),\\delta\_\{t\}=r\_\{t\+1\}\+\\gamma V\(s\_\{t\+1\}\)\-V\(s\_\{t\}\),
The resulting policy gradient update takes the form:

∇θJ​\(θ\)=𝔼​\[δt​∇θlog⁡πθ​\(a\|s\)\],\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\\left\[\\delta\_\{t\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)\\right\],where∇θlog⁡πθ​\(a\|s\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)captures the sensitivity of the policy to parameter changes, andδt\\delta\_\{t\}indicates whether the action performed better or worse than expected\.

As in policy\-based methods, this update reinforces actions yielding higher\-than\-expected returns while suppressing suboptimal ones\. Using a learned value function as a baseline reduces the variance of the gradient estimates, leading to more stable and efficient learning\.

### 2\.5Explainable Deep Reinforcement Learning Methods

Explainability in deep reinforcement learning \(XRL\) aims to make an agent’s decisions, strategies, and failure modes more transparent to human observers\. This need is particularly critical in DRL, where policies are typically represented by deep neural networks, making the decision\-making process difficult to interpret in complex environments\. Recent work categorizes XRL methods according to the level at which explanations are generated, including feature\-level, state\-level, dataset\-level, and model\-level approachesChenget al\.\([2025](https://arxiv.org/html/2608.09967#bib.bib8)\)\.

#### Feature\-level explanations\.

Feature\-level methods aim to identify which components of the input most influence the agent’s decisions\. For example, SHAPLundberg and Lee \([2017](https://arxiv.org/html/2608.09967#bib.bib20)\)provides a model\-agnostic framework based on Shapley values to attribute importance to input features\. While these methods offer insight into which features influence decisions, they remain limited to single timesteps and primarily capture low\-level correlations, without explaining how decisions affect future outcomes\.

#### State\-level explanations\.

State\-level methods explain decisions by identifying important states in the agent’s trajectory and analyzing their impact on performance\. Prior work includes offline methods that analyze pre\-collected trajectories, such as AIRSYuet al\.\([2023](https://arxiv.org/html/2608.09967#bib.bib15)\)\. These methods depend on the quality of collected data and may fail to capture unseen states\. Online approaches, such as LazyMDPJacqet al\.\([2022](https://arxiv.org/html/2608.09967#bib.bib19)\), identify critical steps by modifying actions during interaction\. Another line of work uses causal models to explain decisions; for example,Madumalet al\.\([2019](https://arxiv.org/html/2608.09967#bib.bib17)\)learn structural causal models during training to capture cause\-effect relationships between variables and actions, while assuming a directed acyclic graph \(DAG\) specifying the causal relationships between variables is given\. These models enable the generation of contrastive explanations \(e\.g\., “why this action and not another”\) by reasoning over counterfactual scenarios\. While effective for highlighting influential states, these approaches may not generalize well to long\-term or global policy behavior\.

#### Dataset\-level explanations\.

Dataset\-level methods explain agent behavior by quantifying how training data influences the learned policy\. These approaches provide a global perspective by identifying influential trajectories or experiences\. For example, Data Shapley methodsGhorbani and Zou \([2019](https://arxiv.org/html/2608.09967#bib.bib14)\)assign importance scores to data points based on their contribution to policy performance\. While such methods help diagnose training dynamics and guide data collection, they are computationally expensive and do not provide insight into individual decisions\.

#### Model\-level explanations\.

Model\-level methods provide structured, high\-level explanations by analyzing the learned policy or the underlying environment model\. For example,Finkelsteinet al\.\([2022](https://arxiv.org/html/2608.09967#bib.bib18)\)generate explanations by applying transformations to a Markov Decision Process \(MDP\) until the agent’s behavior aligns with an observer’s expectations\. These transformations reveal which aspects of the environment, such as constraints or dynamics, explain discrepancies between expected and observed behavior\. While such approaches provide global insight into policy behavior, they often require access to a model of the environment and rely on additional assumptions \(e\.g\., a human\-observer policy\)\.

#### Summary\.

Overall, existing XRL methods differ in both*what*they explain and*how*they explain it\. Feature\- and state\-level methods are often useful for local, post\-hoc interpretation, whereas dataset\-level methods summarize broader behavior, and model\-level methods provide more structured and often more semantically meaningful accounts of decision making\.

## 3SPOT: Sampling Policy Observation Tree

We propose*SPOT*\(Sampling Policy Observation Tree\), an online sampling\-based structure designed to expose and interpret the decision\-making process of deep reinforcement learning agents\. SPOT constructs a tree rooted at the current state and expands it using samples drawn from the agent’s policy, thereby providing an interpretable representation of action preferences and their relative likelihoods\.

### 3\.1Model\-Agnostic Policy Extraction

SPOT assumes access to a stochastic policyπ​\(a∣s\)\\pi\(a\\mid s\)defined over a finite action space𝒜=\{a1,…,ak\}\\mathcal\{A\}=\\\{a\_\{1\},\\dots,a\_\{k\}\\\}\. For policy\-based and actor\-critic methods, this policy is directly available as the output of the actor network\. For value\-based methods, which estimate an action\-value functionQ​\(s,a\)Q\(s,a\), we derive a stochastic policy using a softmax transformationSutton and Barto \([2018](https://arxiv.org/html/2608.09967#bib.bib12)\):

π​\(ai∣s\)=exp⁡\(Q​\(s,ai\)\)∑j=1kexp⁡\(Q​\(s,aj\)\)\\pi\(a\_\{i\}\\mid s\)=\\frac\{\\exp\\left\(Q\(s,a\_\{i\}\)\\right\)\}\{\\sum\_\{j=1\}^\{k\}\\exp\\left\(Q\(s,a\_\{j\}\)\\right\)\}
Since a stochastic policy can be derived for each class of deep reinforcement learning methods, SPOT is model\-agnostic and can be applied in all settings\.

### 3\.2Tree Construction

SPOT assumes access to an agent’s policyπ\\piand an environment simulatorℰ\\mathcal\{E\}capable of generating successor states\. Given a current statess, SPOT initializes a root node representingssand drawsNNindependent action samples:

A1,…,AN∼π\(⋅∣s\)\.A\_\{1\},\\ldots,A\_\{N\}\\sim\\pi\(\\cdot\\mid s\)\.
Let𝒜=\{a1,…,ak\}\\mathcal\{A\}=\\\{a\_\{1\},\\ldots,a\_\{k\}\\\}denote the action space\. Rather than creating a separate branch for every sample, SPOT aggregates identical actions\. The visit count of actionaia\_\{i\}is

Ci:=∑n=1N𝟏​\{An=ai\},∑i=1kCi=N\.C\_\{i\}:=\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\\\{A\_\{n\}=a\_\{i\}\\\},\\qquad\\sum\_\{i=1\}^\{k\}C\_\{i\}=N\.
For each actionaia\_\{i\}withCi\>0C\_\{i\}\>0, the simulator generates the successor state

si′=ℰ​\(s,ai\)\.s^\{\\prime\}\_\{i\}=\\mathcal\{E\}\(s,a\_\{i\}\)\.SPOT then creates a child node representingsi′s^\{\\prime\}\_\{i\}and labels the corresponding branch withaia\_\{i\}andCiC\_\{i\}\. The procedure is applied recursively\. At each node of depthd<Kd<K, SPOT samplesNNactions from the policy at the node’s state, aggregates the samples by action, and simulates the corresponding transitions\. Expansion continues until the predefined maximum depthKKis reached, producing a finite\-horizon tree of the policy’s local decision behavior\. Algorithm[1](https://arxiv.org/html/2608.09967#alg1)summarizes the construction procedure\.

Algorithm 1SPOT Tree Construction0:Initial state

ss, policy

π\\pi, simulator

ℰ\\mathcal\{E\}
0:Sample count

NN, maximum depth

KK
0:SPOT tree

𝒯\\mathcal\{T\}rooted at

ss
1:Initialize

𝒯\\mathcal\{T\}with a root node representing

ss
2:for

d=0,…,K−1d=0,\\ldots,K\-1do

3:foreach node

vvat depth

dddo

4:Let

svs\_\{v\}denote the state represented by

vv
5:Draw

A1,…,AN∼π\(⋅∣sv\)A\_\{1\},\\ldots,A\_\{N\}\\sim\\pi\(\\cdot\\mid s\_\{v\}\)
6:foreach distinct sampled action

aia\_\{i\}do

7:

Ci←∑n=1N𝟏​\{An=ai\}C\_\{i\}\\leftarrow\\sum\_\{n=1\}^\{N\}\\mathbf\{1\}\\\{A\_\{n\}=a\_\{i\}\\\}
8:

si′←ℰ​\(sv,ai\)s^\{\\prime\}\_\{i\}\\leftarrow\\mathcal\{E\}\(s\_\{v\},a\_\{i\}\)
9:Add a child representing

si′s^\{\\prime\}\_\{i\}to

vv
10:Label the branch with

\(ai,Ci\)\(a\_\{i\},C\_\{i\}\)
11:endfor

12:endfor

13:endfor

14:return

𝒯\\mathcal\{T\}

This expansion is applied recursively\. At each node of depthdd, we drawNNsamples from the policy at the corresponding state and construct its child nodes accordingly\. The process continues until a predefined maximum depthKKis reached, yielding a multi\-level tree that captures the agent’s decision\-making over a finite lookahead horizon\.

### 3\.3Theoretical Guarantees

We provide theoretical justification for SPOT by analyzing the relationship between its empirical estimates and the underlying policy\.

###### Theorem 1\(Consistency with the Greedy Policy\)\.

Leta⋆=arg⁡maxa∈𝒜⁡π​\(a∣s\)a^\{\\star\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\}\\pi\(a\\mid s\)be the unique greedy action, and letCiC\_\{i\}denote the visit count of actionaia\_\{i\}obtained fromNNi\.i\.d\. samples drawn fromπ\(⋅∣s\)\\pi\(\\cdot\\mid s\)\. Then,

Pr⁡\(arg⁡maxai∈𝒜⁡Ci=a⋆\)→1as​N→∞\.\\Pr\\\!\\left\(\\arg\\max\_\{a\_\{i\}\\in\\mathcal\{A\}\}C\_\{i\}=a^\{\\star\}\\right\)\\to 1\\quad\\text\{as \}N\\to\\infty\.

This result establishes that SPOT recovers the greedy action in the limit of large sample sizes \(proof in the Supplementary Material\)\.

###### Theorem 2\(Disagreement Under Maximum Uncertainty\)\.

Let𝒜=\{a1,…,ak\}\\mathcal\{A\}=\\\{a\_\{1\},\\dots,a\_\{k\}\\\}, and define

TN:=arg⁡maxai∈𝒜⁡Ci,T\_\{N\}:=\\arg\\max\_\{a\_\{i\}\\in\\mathcal\{A\}\}C\_\{i\},with ties broken uniformly at random\. As the policy approaches the uniform distribution,

π​\(ai∣s\)→1k,\\pi\(a\_\{i\}\\mid s\)\\to\\frac\{1\}\{k\},the disagreement probability satisfies

Pr⁡\(TN≠a⋆\)→k−1k\.\\Pr\(T\_\{N\}\\neq a^\{\\star\}\)\\to\\frac\{k\-1\}\{k\}\.

This result characterizes the behavior of SPOT under high\-entropy policies, where action preferences become indistinguishable \(proof in the Supplementary Material\)\.

### 3\.4Extending SPOT for Rich Policy Analysis

While SPOT represents action preferences through visit counts, the tree structure naturally supports the integration of additional signals that enrich the interpretability of the agent’s decision\-making process\.

#### Node\-Level Information\.

Each node in the SPOT tree can store auxiliary quantities beyond visit counts, including the critic’s value estimateV​\(s\)V\(s\), action\-value estimatesQ​\(s,a\)Q\(s,a\), advantage estimatesA​\(s,a\)A\(s,a\), and observed rewards along sampled trajectories\.

These quantities are computed during tree expansion through environment interaction\. For each sampled action, the environment is advanced from the node’s state to obtain a successor states′s^\{\\prime\}and an immediate rewardr​\(s,a\)r\(s,a\)\. When a value function is available in the DRL model, the value estimatesV​\(s\)V\(s\)andV​\(s′\)V\(s^\{\\prime\}\)are obtained directly from this function\. The advantageA​\(s,a\)A\(s,a\)can be computed using a one\-step temporal\-difference estimate\.

Incorporating these signals provides a richer characterization of each decision, capturing not only action likelihoods but also their expected utility and contribution to long\-term outcomes\.

#### Tree\-Based Analysis\.

The SPOT structure enables the extraction of explanations from the agent’s decision\-making process by analyzing the information encoded in the tree\. Similar to planning\-based approaches, where tree structures expose the agent’s reasoning over possible future trajectories, SPOT provides access to both local decisions and forward\-looking behavior\.

In particular, the explanation methods introduced byGoldmanet al\.\([2025](https://arxiv.org/html/2608.09967#bib.bib21)\)are directly applicable in the SPOT setting\.*Plan Next*explanations describe the agent’s expected future behavior by identifying high\-probability trajectories in the tree, effectively indicating what the agent is likely to do next\.*Contrastive*explanations compare alternative branches to explain why a selected action is preferred over other feasible actions, highlighting the differences between competing choices\. Finally,*Value Component*explanations decompose value\-related quantities stored in the tree, such as accumulated rewards or value estimates, to explain the contribution of different factors to the agent’s decision, when such quantities admit a meaningful decomposition\.

By supporting these explanation methods, SPOT enables a structured analysis of both the immediate and long\-term consequences of decisions for DeepRL solutions\.

#### Global Behavior Summarization\.

Beyond local explanations, the SPOT framework can be used to construct a global view of an agent’s behavior by identifying and analyzing*critical states*\. Prior work has shown that states in which different actions lead to substantially different outcomes are particularly informative for understanding an agent’s strategyAmir and Amir \([2018](https://arxiv.org/html/2608.09967#bib.bib22)\)\.

In SPOT, critical states can be identified by measuring the divergence between the outcomes of alternative actions\. States where different actions lead to large differences in expected return or future trajectories are flagged as high\-impact decision points\. For example, a state where one action leads to low congestion while another results in significant delay is considered critical\. To avoid redundancy, we select critical states that represent different regions of the state space \(e\.g\., different traffic patterns or congestion levels\)\. Aggregating the SPOT trees rooted at these states provides a structured summary of the agent’s policy; this component is not implemented in the current work and will be reported separately\.

## 4Case Study: Traffic Signal Control

In this section, we demonstrate SPOT on traffic\-signal control environment built with SUMOBehrischet al\.\([2011](https://arxiv.org/html/2608.09967#bib.bib10)\)\. The underlying decision\-making problem is to dynamically select the traffic signal phase at each timestep so as to minimize cumulative vehicle waiting time at a signalized intersection\. In our setting the agent does not act autonomously\. Instead, a human operator remains in the loop and uses the agent’s recommendations to guide decision\-making\. In this context, explanations are not merely descriptive but must support actionable oversight: the operator must be able to assess whether to follow or override the agent’s decision\. We show that SPOT produces temporally grounded, domain\-readable explanations that surface critical operational signals that are invisible to single\-timestep attribution methods like SHAPLundberg and Lee \([2017](https://arxiv.org/html/2608.09967#bib.bib20)\)\. We first describe the explanation extraction pipeline \(Section[4\.1](https://arxiv.org/html/2608.09967#S4.SS1)\), then the experimental setup \(Section[4\.2](https://arxiv.org/html/2608.09967#S4.SS2)\), and finally a disruption scenario designed to expose the blind spot of attribution\-only methods \(Section[4\.3](https://arxiv.org/html/2608.09967#S4.SS3)\)\.

### 4\.1Explanation Extraction Pipeline

Given a SPOT tree for each decision step, the explanation pipeline converts raw numerical tree data into human\-readable text through four successive stages\.

#### Stage 1 — Observation decoding\.

The 21\-dimensional observation vector produced by SUMO\-RL is decoded into domain concepts over eight incoming lanes, including the active signal phase, a minimum\-green indicator, and per\-lane traffic features\. \(A detailed description of these components is provided in the Supplementary Material\)\.

#### Stage 2 — Path extraction and trend detection\.

Two paths are extracted from each tree\. The*chosen path*follows the highest\-visit child at every depth, tracing the most likely future trajectory induced by the policy\. This corresponds to the*Plan\-Next \(PN\)*explanation in Goldman et al\. \[[2025](https://arxiv.org/html/2608.09967#bib.bib21)\], capturing what the agent is expected to do next\. The*counterfactual path*starts from the depth\-1 node with the second\-highest visit count and then follows the highest\-visit branch thereafter, representing the most plausible alternative trajectory\. This corresponds to the*Contrastive \(CON\)*explanation in Goldman et al\. \[[2025](https://arxiv.org/html/2608.09967#bib.bib21)\], capturing what would have happened had a different action been taken\.

For each path, we extract six scalar sequences along the trajectory: critic values, immediate rewards, advantages, policy confidence, per\-lane queue lengths, and per\-lane densities\. For each sequence, we compute a qualitative trend label through the following steps:

1. 1\.Net change\.We summarize each sequence by its overall change between the first and last timestep: Δ=xT−x0\.\\Delta=x\_\{T\}\-x\_\{0\}\.This captures whether the quantity increases, decreases, or remains stable along the path\. To interpret the magnitude of this change, we compare\|Δ\|\|\\Delta\|to two thresholds scaled by the root critic valueV0V\_\{0\}: δsmall=0\.2​\|V0\|,δlarge=0\.6​\|V0\|\.\\delta\_\{\\text\{small\}\}=0\.2\|V\_\{0\}\|,\\qquad\\delta\_\{\\text\{large\}\}=0\.6\|V\_\{0\}\|\.Changes smaller thanδsmall\\delta\_\{\\text\{small\}\}are treated as negligible \(*flat*\), changes betweenδsmall\\delta\_\{\\text\{small\}\}andδlarge\\delta\_\{\\text\{large\}\}are considered*small*, and changes larger thanδlarge\\delta\_\{\\text\{large\}\}are considered*large*\. Combining magnitude with the sign ofΔ\\Deltayields five coarse categories:*large positive*,*small positive*,*flat*,*small negative*, and*large negative*\.
2. 2\.Trajectory shape\.We then examine step\-by\-step changes along the sequence to determine whether the trajectory is: \(i\) monotonic \(consistently increasing or decreasing\), or \(ii\) non\-monotonic \(changing direction\)\.
3. 3\.Final classification\.Combining the net change and the trajectory shape produces one of seven trend categories:*monotone\-large\-positive*,*monotone\-small\-positive*,*monotone\-large\-negative*,*monotone\-small\-negative*,*non\-monotone\-net\-positive*,*non\-monotone\-net\-negative*, or*flat*\.

#### Stage 3 — Signal\-strength filtering\.

Not all detected trends are informative enough to be included in the explanation\. We therefore apply a signal\-strength filter that retains only trends that exhibit sufficient magnitude or clear directional structure, and suppresses weak or ambiguous signals\.

The filter removes noise \(small fluctuations or inconsistent patterns\) and keeps only trends that provide actionable insight into the system’s behavior\. Table 2 in the Supplementary Material summarizes the filtering criteria\.

#### Stage 4 — Stance scoring and actionable summary\.

The chosen path \(*chosen\_dir*\) is compared against the alternative path \(*alt\_dir*\) by aggregating evidence across multiple aspects into a single scalar score\. Each aspect contributes a signed vote, where positive values support*chosen\_dir*and negative values support*alt\_dir*\. The final score is obtained by summing all votes and is mapped to a recommendation: high positive scores indicateTrust, intermediate scores indicateWatch, and strongly negative scores indicateIntervene\. Full scoring rules and thresholds are provided in the Supplementary Material file\.

#### Actionable summary\.

At each decision step, the system produces a concise, one\-paragraph summary for the operator\. It includes a stance label \(Trust,Watch, orIntervene\), the key signals driving the decision, and an indication of the agent’s confidence\. When the stance isIntervene, the summary also identifies the preferred alternative phase and explains why it is favored\.

### 4\.2Experimental Setup

#### Simulation environment\.

We use the SUMO\-RL Gymnasium wrapper111[https://github\.com/LucasAlegre/sumo\-rl](https://github.com/LucasAlegre/sumo-rl)with a single four\-way signalized intersection\. Each episode runs for 3,600 simulated seconds\. The observation space follows the default SUMO\-RL encoding:

obs=\[\\displaystyle\\texttt\{obs\}=\[phase\_one\_hot,min\_green,\\displaystyle\\texttt\{phase\\\_one\\\_hot\},\\;\\texttt\{min\\\_green\},lane densities,lane queues\]\\displaystyle\\texttt\{lane densities\},\\;\\texttt\{lane queues\}\]which in our network yields a 21\-dimensional state vector\. The reward is the change in cumulative vehicle waiting time \(diff\-waiting\-time\), so negative reward corresponds to increasing delay\. The action space consists of four discrete signal phases \(Figure[2](https://arxiv.org/html/2608.09967#S4.F2)\), each enabling a set of non\-conflicting traffic movements\. The minimum green time is 5 s, and every phase transition incurs a 2 s yellow interval\. Consequently, the agent acts every 5 simulated seconds \(delta\_time=5\)\.

![Refer to caption](https://arxiv.org/html/2608.09967v1/sumo_rl_phases.png)Figure 2:Signal phases \(agent’s actions\) used in the SUMO\-RL environment\.
#### Policy\.

We train a Proximal Policy Optimization \(PPO\) agent for100,000100\{,\}000timesteps using Stable\-Baselines3222[https://stable\-baselines3\.readthedocs\.io/](https://stable-baselines3.readthedocs.io/)with standard hyperparameters, including a discount factor ofγ=0\.99\\gamma=0\.99and a clipping range of0\.20\.2\. The agent uses an MLP actor–critic policy and is trained in the single\-agent SUMO\-RL environment defined by the two\-way single\-intersection network and its associated traffic\-demand route file\. At inference time, the agent acts deterministically according to

at=arg⁡maxa⁡π​\(a∣st\)\.a\_\{t\}=\\arg\\max\_\{a\}\\pi\(a\\mid s\_\{t\}\)\.

#### SPOT Configuration\.

At each decision step, SPOT constructs a lookahead tree by samplingN=100N=100stochastic actions from the trained policy at every node and expanding the tree to depthK=3K=3\.

### 4\.3Accident Scenario: Demonstrating SPOT’s Future\-Aware Reasoning

#### Motivation\.

Methods such as SHAP provide explanations based solely on the current observation\. As a result, disruptions that do not immediately affect the observed state—but lead to future degradation—remain undetected\. To highlight this limitation, we design a scenario in which a lane blockage is placed upstream of the sensor range\. This ensures that the current feature vector appears normal, while the system’s performance deteriorates over time\.

#### Scenario design\.

Att=600t=600\(step 120\), we introduce a disruption by placing a stationary vehicle on the northbound lane upstream of the sensor coverage region \(Figure[3](https://arxiv.org/html/2608.09967#S4.F3)\)\. This causes a queue to form outside the observed region, making the resulting congestion unobservable in the agent’s input\. The disruption lasts 600 seconds \(steps 120–240\), allowing congestion to accumulate and its long\-term impact to become apparent\.

![Refer to caption](https://arxiv.org/html/2608.09967v1/lane_disruption_1.png)Figure 3:Lane\-level view of the disruption\. The blockage is located upstream of the sensor zone, causing a queue that remains outside the observed region\.In SUMO\-RL, traffic is measured only near the stop line\. Since the disabled vehicle is positioned upstream, the resulting queue lies outside the sensor’s field of view\. Consequently, the corresponding queue feature remains*light*throughout the accident window, even as congestion builds upstream\.

This limitation also applies to the SPOT tree: each node is evaluated using the same sensor\-based observation, so the blocked lane appears*light*at all depths\. SPOT does not detect the disruption directly, but instead captures its downstream effects\.

#### Example SPOT Explanation\.

To illustrate SPOT explanations, Figure[4](https://arxiv.org/html/2608.09967#S4.F4)shows the control panel output for a representative decision during the accident window \(step 122\)\. Despite the agent strongly favoring the current phase, SPOT recommendsIntervene, identifying an alternative phase that reduces expected future waiting time\. Notably, this recommendation arises even though the observed state appears normal, highlighting SPOT’s ability to reason beyond immediate observations\.

In contrast, Figure[5](https://arxiv.org/html/2608.09967#S4.F5)shows a later decision \(step 172\), where SPOT recommendsTrust, indicating that the agent’s selected phase is well\-supported\. Here, the lookahead analysis confirms that the current action leads to favorable downstream outcomes\.

![Refer to caption](https://arxiv.org/html/2608.09967v1/spot_122_2.png)Figure 4:SPOT control panel for step 122\. The system recommendsintervene, suggesting a switch to an alternative phase that improves expected future traffic conditions\.![Refer to caption](https://arxiv.org/html/2608.09967v1/spot_172_2.png)Figure 5:SPOT control panel for step 172\. The system recommendsTrust, indicating that the agent’s selected phase is supported by favorable expected future outcomes\.Figure[6](https://arxiv.org/html/2608.09967#S4.F6)shows the SPOT tree corresponding to step 122, where each node encodes visit counts, rewards, critic values, and advantages\. The tree reveals that the alternative action leads to more favorable downstream outcomes, even when immediate signals are inconclusive\.

![Refer to caption](https://arxiv.org/html/2608.09967v1/ROOT_122_updated.png)Figure 6:SPOT for step 122\.![Refer to caption](https://arxiv.org/html/2608.09967v1/shap_step122_paper.png)Figure 7:SHAP feature attribution for step 122\.For comparison, Figure[7](https://arxiv.org/html/2608.09967#S4.F7)shows feature\-level explanations obtained using SHAP\. Although SHAP identifies which features drive the agent’s immediate decision, it provides no information about the consequences of that decision\. Because it operates on a single observation, SHAP cannot account for how traffic conditions evolve over time or how alternative actions would affect future states\. In the presence of the upstream disruption—unobservable in the current feature vector—SHAP fails to detect the emerging congestion and instead explains a decision that appears locally reasonable\.

## 5Conclusion

We introducedSPOT, an explainable AI framework for deep reinforcement learning that constructs a sampling\-based tree to capture the distribution of future trajectories induced by a policy\. The approach is model\-agnostic and applies across value\-based, policy\-based, and actor\-critic methods\.

We provided theoretical guarantees showing that SPOT faithfully reflects the underlying policy\. Empirically, we demonstrated SPOT in a SUMO\-RL traffic control setting, where it reveals behaviors not captured by single\-timestep explanations\. In particular, SPOT uncovers robustness issues arising from partially observable disruptions by reasoning over future trajectories, enabling more informed intervention in a human\-in\-the\-loop setting\.

As future work, we plan to conduct user studies to evaluate the actionability of SPOT explanations and their impact on human decision\-making\.

## References

- D\. Amir and O\. Amir \(2018\)HIGHLIGHTS: summarizing agent behavior to people\.InProceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS\),pp\. 1168–1176\.Cited by:[§3\.4](https://arxiv.org/html/2608.09967#S3.SS4.SSS0.Px3.p1.1)\.
- K\. Arulkumaran, P\. M\. Deisenroth, M\. Brundage, and A\. A\. Bharath \(2017\)Deep reinforcement learning: a brief survey\.IEEE Signal Processing Magazine34\(6\),pp\. 26–38\.External Links:[Document](https://dx.doi.org/10.1109/MSP.2017.2743240)Cited by:[§2](https://arxiv.org/html/2608.09967#S2.p1.1)\.
- M\. Behrisch, L\. Bieker, J\. Erdmann, and D\. Krajzewicz \(2011\)SUMO \- simulation of urban mobility: an overview\.InProceedings of the Third International Conference on Advances in System Simulation \(SIMUL\),Cited by:[§4](https://arxiv.org/html/2608.09967#S4.p1.1)\.
- Z\. Cheng, J\. Yu, and X\. Xing \(2025\)A survey on explainable deep reinforcement learning\.arXiv preprint arXiv:2502\.06869\.Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.p1.1)\.
- H\. Dong, Z\. Ding, and S\. Zhang \(2020\)Deep reinforcement learning: fundamentals, research, and applications\.Springer Nature[https://doi\.org/10\.1007/978\-981\-15\-4095\-0](https://doi.org/10.1007/978-981-15-4095-0)\.Note:[https://doi\.org/10\.1007/978\-981\-15\-4095\-0](https://doi.org/10.1007/978-981-15-4095-0)Cited by:[Figure 1](https://arxiv.org/html/2608.09967#S2.F1),[§2\.1](https://arxiv.org/html/2608.09967#S2.SS1.p1.6),[§2](https://arxiv.org/html/2608.09967#S2.p1.1)\.
- M\. Finkelstein, L\. Liu, N\. L\. Schlot, Y\. Kolumbus, D\. C\. Parkes, J\. S\. Rosenschein, and S\. Keren \(2022\)Explainable reinforcement learning via model transforms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px4.p1.1)\.
- A\. Ghorbani and J\. Zou \(2019\)Data shapley: equitable valuation of data for machine learning\.ICML\.Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px3.p1.1)\.
- C\. V\. Goldman, R\. Bustin, W\. Qi, Z\. Xing, R\. McPhearson\-White, and S\. Rogers \(2025\)The value of real\-time automated explanations in stochastic planning\.Artificial Intelligence343,pp\. 104323\.External Links:[Document](https://dx.doi.org/10.1016/j.artint.2025.104323)Cited by:[§3\.4](https://arxiv.org/html/2608.09967#S3.SS4.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2608.09967#S4.SS1.SSS0.Px2.p1.1)\.
- I\. Grondman, L\. Busoniu, G\. A\. D\. Lopes, and R\. Babuska \(2012\)A survey of actor\-critic reinforcement learning: standard and natural policy gradients\.IEEE Transactions on Systems, Man, and Cybernetics, Part C42\(6\),pp\. 1291–1307\.External Links:[Document](https://dx.doi.org/10.1109/TSMCC.2012.2218595)Cited by:[§2\.4](https://arxiv.org/html/2608.09967#S2.SS4.p1.1)\.
- A\. Jacq, J\. Ferret, O\. Pietquin, and M\. Geist \(2022\)Lazy\-mdps: towards interpretable reinforcement learning by learning when to act\.Autonomous Agents and Multi\-Agent Systems\.Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px2.p1.1)\.
- K\. Lekadir, A\. F\. Frangi, A\. R\. Porras, B\. Glocker, C\. Cintas, C\. P\. Langlotz, E\. Weicken, F\. W\. Asselbergs, F\. Prior, G\. S\. Collins, G\. Kaissis, G\. Tsakou, I\. Buvat, J\. Kalpathy\-Cramer, J\. Mongan, J\. A\. Schnabel, K\. Kushibar, K\. Riklund, K\. Marias, L\. M\. Amugongo, L\. A\. Fromont, L\. Maier\-Hein, L\. Cerdá\-Alberich, L\. Martí\-Bonmatí, M\. J\. Cardoso, M\. Bobowicz, M\. Shabani, M\. Tsiknakis, M\. A\. Zuluaga, M\. Fritzsche, M\. Camacho, M\. G\. Linguraru, M\. Wenzel, M\. D\. Bruijne, M\. G\. Tolsgaard, M\. Goisauf, M\. C\. Abadía, N\. Papanikolaou, N\. Lazrak, O\. Pujol, R\. Osuala, S\. Napel, S\. Colantonio, S\. Joshi, S\. Klein, S\. Aussó, W\. A\. Rogers, Z\. Salahuddin, and M\. P\. A\. Starmans \(2025\)FUTURE\-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare\.BMJ388,pp\. r340\.External Links:[Document](https://dx.doi.org/10.1136/bmj.r340)Cited by:[§1](https://arxiv.org/html/2608.09967#S1.p1.1)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.09967#S4.p1.1)\.
- P\. Madumal, T\. Miller, L\. Sonenberg, and F\. Vetere \(2019\)Explainable reinforcement learning through a causal lens\.arXiv preprint arXiv:1905\.10958\.Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px2.p1.1)\.
- M\. C\. McKenzie and M\. D\. McDonnell \(2022\)Modern value\-based reinforcement learning: a chronological review\.IEEE Access10,pp\. 134704–134720\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2022.3228647)Cited by:[§2\.2](https://arxiv.org/html/2608.09967#S2.SS2.p3.1)\.
- V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. Graves, I\. Antonoglou, D\. Wierstra, and M\. Riedmiller \(2013\)Playing atari with deep reinforcement learning\.arXiv preprint arXiv:1312\.5602\.Cited by:[§2\.2](https://arxiv.org/html/2608.09967#S2.SS2.p4.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.3](https://arxiv.org/html/2608.09967#S2.SS3.p5.1)\.
- R\. S\. Sutton and A\. G\. Barto \(2018\)Reinforcement learning: an introduction\.Second edition,MIT Press\.Note:Second edition, in progressCited by:[§2](https://arxiv.org/html/2608.09967#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.09967#S3.SS1.p1.3)\.
- H\. van Hasselt, A\. Guez, and D\. Silver \(2016\)Deep reinforcement learning with double q\-learning\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),Vol\.30\.Cited by:[§2\.2](https://arxiv.org/html/2608.09967#S2.SS2.p4.1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3\-4\),pp\. 229–256\.Cited by:[§2\.3](https://arxiv.org/html/2608.09967#S2.SS3.p1.1)\.
- J\. Yu, W\. Guo, Q\. Qin, G\. Wang, T\. Wang, and X\. Xing \(2023\)AIRS: explanation for deep reinforcement learning based security applications\.InUSENIX Security Symposium,Cited by:[§2\.5](https://arxiv.org/html/2608.09967#S2.SS5.SSS0.Px2.p1.1)\.