Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents

# Evaluating RL Explainability Methodsby How Much They Help Fix Bugs in Agents
Source: [https://arxiv.org/html/2608.17524](https://arxiv.org/html/2608.17524)
Ram RachumAffiliation:University of California, BerkeleyAffiliation:Tufts UniversityEmail:[ram\.rachum@berkeley\.edu](mailto:[email protected])Bálint GyevnárAffiliation:Carnegie Mellon UniversityEmail:[bgyevnar@cmu\.edu](mailto:[email protected])Reuth MirskyAffiliation:Tufts UniversityEmail:[reuth\.mirsky@tufts\.edu](mailto:[email protected])Cameron AllenAffiliation:University of California, BerkeleyEmail:[camallen@berkeley\.edu](mailto:[email protected])

###### Abstract

This preliminary paper outlines a planned evaluation benchmark for Explainable Reinforcement Learning \(XRL\) methods\. Current evaluations rely on functionally\-grounded metrics like faithfulness and compactness, and on human\-grounded proxies like subjective ratings or prediction accuracy\. We suggest evaluating XRL methods by how effectively their generated explanations help to diagnose and fix malfunctioning reinforcement learning \(RL\) agents\. We proposeEvalXRL, a benchmark in which a Large Language Model \(LLM\) coding agent uses different XRL methods to diagnose a held\-out malfunction in an RL agent, and then repair it\. Our proposed benchmark iterates across \(environment×\\timesmalfunction×\\timesXRL method\) tuples and uses the reward signal of the RL agents to form a final score for each XRL method\. The coding agent may use the methodinteractively: invoke the XRL method, process its output, form new hypotheses on what is broken, and invoke the method again with parameters adjusted for testing these hypotheses\. This closed\-loop structure may be described as a simplified version of the scientific method\. Some XRL methods provide self\-evaluations that follow this pattern; we propose the first head\-to\-head comparison of multiple XRL methods in closed\-loop usage\.

## 1Introduction

XRL methods produce explanations that answer several distinct needs\. Developers want to know why their agent is failing; end\-users want to know whether an agent is trustworthy; regulators want accountability in systems that affect the public\. All these stakeholders want an explanation, but each wants to get something different out of it, so each has different criteria for what they will consider a good explanation[48](https://arxiv.org/html/2608.17524#bib.bib31)\.

The landscape of XRL methods is as varied as the makeup of its target audience\. Methods differ in what kind of questions they answer, in what information they use to build answers, and in the form their answers take\. A saliency map[24](https://arxiv.org/html/2608.17524#bib.bib41)marks which pixels of an observation drove a single action; HIGHLIGHTS[3](https://arxiv.org/html/2608.17524#bib.bib44)returns short video clips of the policy’s most important moments; reward decomposition[34](https://arxiv.org/html/2608.17524#bib.bib19)reports per\-component contributions to a scalar value; and VIPER[9](https://arxiv.org/html/2608.17524#bib.bib21)distills the whole policy into a readable decision tree\. Given this Cambrian explosion of outputs, the XRL community would benefit from evaluation methods that take in each of these heterogeneous artifacts and produce a homogeneous array of comparable scores\. Hopefully, these scores would answer the questions “how useful are the explanations produced by each method?” and “which methods are the most useful?” keeping in mind that the different types of XRL users we described above might each require a different evaluation scheme\.

Evaluating explanations is a difficult problem, because an explanation is considered good to the extent that it helps the userunderstand; therefore, each XRL evaluation paradigm can be interpreted as an answer to the epistemological question “what does it mean tounderstand?”[41](https://arxiv.org/html/2608.17524#bib.bib64);[18](https://arxiv.org/html/2608.17524#bib.bib63)Table[1](https://arxiv.org/html/2608.17524#S1.T1)surveys three existing paradigms for XRL evaluation, each framed as an implicit answer to this question\.

Table 1:Existing paradigms for XRL evaluation, framed as implicit answers to“what does it mean to understand?”We are proposing an evaluation paradigm whose respective answer is:“to understand a mechanism is to be able to fix it when it breaks\.”This functional view of explanation is long\-standing in cognitive science[43](https://arxiv.org/html/2608.17524#bib.bib7), with a recent ability\-based formulation for AI[16](https://arxiv.org/html/2608.17524#bib.bib8); as an evaluation paradigm it is partial like the others, but measurable\. Therefore, our main focus is the first motivation we listed: the developer’s need to know why their agents are failing\. This is often calleddebugging[26](https://arxiv.org/html/2608.17524#bib.bib1), and we will use the terms debugging anddiagnosissynonymously\. Diagnosing failures is an arduous but necessary phase in the endeavor of developing increasingly capable RL agents\. In this paper, we propose a method to evaluate which XRL methods produce explanations that are most useful for diagnosis\. The framing follows the spirit of[20](https://arxiv.org/html/2608.17524#bib.bib3)’sapplication\-groundedevaluation, with the developer’s downstream task as the metric\. It echoes recent calls for downstream\-validated, fair\-fight interpretability evaluation[47](https://arxiv.org/html/2608.17524#bib.bib4);[52](https://arxiv.org/html/2608.17524#bib.bib5)and for treating intervention outcomes as ground truth[8](https://arxiv.org/html/2608.17524#bib.bib6)\. To our knowledge, though, this has not been executed as a benchmark, and not for explainable RL\.

This approach offers several benefits\. First, we can wire the RL agent’s reward signal directly to the output of our evaluation\. In other words, we define the success of an XRL method as the performance of an RL agent that was repaired using information from that method\. This unmediated connection may be a good strategy for removing proxy\-driven bias from our scores\. Most excitingly, diagnosis and repair are interactive processes, in which an explainability method may be used multiple times; we elaborate in design choice 2 below\.

We proposeEvalXRL, a benchmark for evaluating XRL methods by how well they help a developer repair online\-RL agents\. It operationalizes task completion as follows: \(1\) give the developer a deliberately broken RL agent and an XRL method; \(2\) ask them to repair it; \(3\) measure how well the repaired agent performs, as reported by its reward signal or a held\-out scoring function\. The score is always continuous, never a binary “did you find the bug?” judgment; we never ask the developer to articulate the cause, only to make the agent perform better\. Three design choices follow:

1. 1\.Deliberately broken RL agents as evaluation substrate\.We construct agents with known, controlled malfunctions whose diagnosis requires reasoning about behavioral consequences\. Holding the malfunction set fixed across all XRL methods lets us measure each method’s diagnostic utility on like\-for\-like cells\.
2. 2\.The developer may use the method multiple times, and adjust its parameters\.Diagnosis is a closed\-loop process: the developer invokes the XRL method, processes its output, forms new hypotheses about what is broken, and re\-invokes the method with parameters adjusted to test these specific hypotheses\. This is in contrast to one\-shot or pre\-planned multi\-shot use \(open\-loop\), where parameters are fixed without reference to intermediate results\. We make closed\-loop use a first\-class property of the evaluation: the repair score is computed after arbitrarily many XRL method invocations within the per\-session budget, not after one\. Some XRL methods provide self\-evaluations in a closed\-loop pattern[4](https://arxiv.org/html/2608.17524#bib.bib27);[69](https://arxiv.org/html/2608.17524#bib.bib69);[19](https://arxiv.org/html/2608.17524#bib.bib70);[7](https://arxiv.org/html/2608.17524#bib.bib11); we propose the first head\-to\-head comparison of multiple XRL methods in closed\-loop usage\.
3. 3\.We use an LLM coding agent as the developer\.Running our proposed procedure with LLM agents is orders of magnitude cheaper and faster than running it with human subjects, and avoids complex approvals for human research\. In our default configuration the developer is a frontier LLM \(e\.g\., Claude Opus 4\.8\) wrapped in a minimalist coding\-agent scaffold \(e\.g\.,mini\-swe\-agent[61](https://arxiv.org/html/2608.17524#bib.bib74)\), running in a sandboxed Docker environment with full programmatic control: it can read source, run experiments, modify parameters, retrain the agent, and submit a fix\. We measure variance acrossNNruns per cell\. We discuss the tradeoffs of more elaborate scaffolds in Section[5](https://arxiv.org/html/2608.17524#S5)\.

A measurement of how effective an XRL method is for helping an LLM agent fix a bug may not carry over accurately enough to the use case of helping a human, and we sketch a calibration between the two in Section[5](https://arxiv.org/html/2608.17524#S5)\. However, we argue that besides serving as a proxy for humans, this measurement may be useful in and of itself\.[12](https://arxiv.org/html/2608.17524#bib.bib39)surveyed deployed XAI systems and found that the dominant user is the ML engineer who is debugging a model, not a layperson; and LLM coding agents are increasingly doing engineer\-shaped work, with clear evidence that engineering of ML systems is being automated with LLM assistance[35](https://arxiv.org/html/2608.17524#bib.bib15);[42](https://arxiv.org/html/2608.17524#bib.bib16), often referred to as “vibe coding”\. This suggests that LLM coding agents may take a larger role in debugging work that was previously done by human engineers\. If “vibe debugging” becomes prevalent, we advocate that it should be studied in controlled settings\.

We present EvalXRL as a benchmarkdesignand a set of hypotheses, not yet as a validated leaderboard\. The pilot run that would test H1–H3 is the immediate next step, and is the contribution of a follow\-up paper\. The rest of this paper describes the design, the hypotheses, and open questions we want the community’s input on\.

## 2Related Work

The closest prior work to EvalXRL is[26](https://arxiv.org/html/2608.17524#bib.bib1), who divide XRL evaluation into two contexts:debugging\(developers verifying or investigating an agent before/after deployment\) andteaming\(humans and agents collaborating during deployment\)\. Their five debugging metrics are all prediction\-based: next\-action, goal, sub\-goal, counterfactual policy, and time taken\. Task completion appears in their layout only on the teaming side\. The cell their layout does not contain istask completion as a debugging metric: not predicting what the agent will do, but fixing what the broken agent fails to do and then measuring the fixed agent\. EvalXRL is an attempt to fill that cell\.

#### Cross\-method XRL comparisons\.

Recent work has compared multiple XRL methods on the same task with the same users via objective accuracy\.[57](https://arxiv.org/html/2608.17524#bib.bib28)crossed HIGHLIGHTS with reward decomposition on Highway\-env and Pacman;[5](https://arxiv.org/html/2608.17524#bib.bib26)found counterfactual\-outcome explanations and reward decomposition complementary on a highway agent;[64](https://arxiv.org/html/2608.17524#bib.bib2)evaluated four mechanisms on Pacman; and[22](https://arxiv.org/html/2608.17524#bib.bib49)compared counterfactual versus critical\-state trajectories under distribution shift\. EvalXRL extends this comparison style from prediction and identification tasks to repair tasks\.

#### Repair as the XAI use case\.

[12](https://arxiv.org/html/2608.17524#bib.bib39)found that ML engineers debugging models are XAI’s dominant real\-world consumers\. In supervised XAI,[1](https://arxiv.org/html/2608.17524#bib.bib18)introduceddebugging teststhat evaluate explanation methods against known model bugs; we adapt this framing to RL by scoring XRL methods on downstream repair success\. The closest precursor to EvalXRL is[58](https://arxiv.org/html/2608.17524#bib.bib46), who constructed Frogger agents with controlled deficits and ran a user study where participants identified each agent’s capabilities and limitations from XRL\-generated visual summaries\.[54](https://arxiv.org/html/2608.17524#bib.bib47)extended this to flaw\-detection: non\-experts who watched counterfactual states from a Space Invaders deep RL agent whose ship\-position pixels had been masked raised flaw identification from 57% to 90%\. Earlier,[31](https://arxiv.org/html/2608.17524#bib.bib45)generated natural\-language policy explanations to help operators pinpoint controller faults, and[7](https://arxiv.org/html/2608.17524#bib.bib11)had non\-experts iteratively patch a Super Mario Bros agent with interactive explanations\.[4](https://arxiv.org/html/2608.17524#bib.bib27)’s ASQ\-IT lets users iteratively query an RL agent’s behavior, with a user study showing that interactive querying improves users’ ability to identify faulty behavior; this directly motivates EvalXRL’s closed\-loop design choice \(Section[1](https://arxiv.org/html/2608.17524#S1)\)\. None of these studies compare XRL methods head\-to\-head on a controlled malfunction set\.[55](https://arxiv.org/html/2608.17524#bib.bib10)ran the closest task\-completion study, but with a hand\-designed agent rather than a learned RL policy\.[63](https://arxiv.org/html/2608.17524#bib.bib17)’s LEGIBLE demonstrates that the diagnose\-and\-repair loop is already viable inside RL, lifting cumulative reward by up to 273% on highway\-fast without retraining, though as a single method it offers no head\-to\-head comparison\.

#### Dialogic and hypothesis\-driven XAI\.

The argument that explanation is fundamentally dialogical rather than one\-shot has been made repeatedly\.[48](https://arxiv.org/html/2608.17524#bib.bib31)reviews the cognitive\-science evidence that explanation is an abductive\-reasoning loop and a social process\.[45](https://arxiv.org/html/2608.17524#bib.bib66)derive a grounded dialog model from398398explanation dialogs in which follow\-up questions and re\-explanation are first\-class moves\.[37](https://arxiv.org/html/2608.17524#bib.bib65)’s interviews with2626domain experts \(medical and policy researchers\) report that all but one cited the impossibility of conversing with explanations as a primary frustration\.[49](https://arxiv.org/html/2608.17524#bib.bib71)argues for a paradigm shift to hypothesis\-driven decision support, what he callsevaluative AI;[39](https://arxiv.org/html/2608.17524#bib.bib72)provide a Weight\-of\-Evidence implementation reporting improved trust calibration over a recommendation\-driven baseline, with mixed accuracy effects across populations\.[69](https://arxiv.org/html/2608.17524#bib.bib69)’s LangXRL operationalizes a related idea in RL: users edit history information and observe counterfactual changes in agent behavior, with larger gains on abnormality identification and downstream actionability against a static\-text baseline\.[19](https://arxiv.org/html/2608.17524#bib.bib70)extend the structured\-iteration argument with AAR/AI, an after\-action\-review protocol that frames RL agent assessment as Popperian falsification\. EvalXRL sits in this lineage but inverts the human/machine roles: the LLM coder, not the human, drives the closed loop, which lets the comparison run at scale and be scored end\-to\-end on a downstream repair task\.

#### LLM coding agents with tools\.

A parallel thread asks whether giving an LLM coding agent a debugging or ML\-engineering tool improves repair performance\.[73](https://arxiv.org/html/2608.17524#bib.bib23)’s Debug\-Gym lifts SWE\-bench Lite scores by exposingpdb, and[40](https://arxiv.org/html/2608.17524#bib.bib24)let an LLM autonomously drivepdb/gdb/lldbto fix student Python bugs\.[28](https://arxiv.org/html/2608.17524#bib.bib25)provide a counterpoint: naive injection of full execution traces helps in only 2 of 6 dataset\-model configurations\.[15](https://arxiv.org/html/2608.17524#bib.bib54)’s MLE\-bench and[68](https://arxiv.org/html/2608.17524#bib.bib55)’s RE\-Bench drop LLM agents into sandboxed ML engineering tasks scored by execution; EvalXRL inherits their harness pattern but re\-targets it at XRL\-method\-assisted repair of malfunctioning RL agents\.[29](https://arxiv.org/html/2608.17524#bib.bib59)’s Breakpoint takes the closest methodological step: it auto\-generates code\-repair tasks by corrupting functions in real repositories, using fault\-injection rather than naturally\-occurring bugs\. EvalXRL’s malfunction\-injection design follows the same logic, applied to RL\.

#### LLM surrogates and auditing precedents\.

[17](https://arxiv.org/html/2608.17524#bib.bib13)showed that LLMs can replicate human conclusions on XAI explanation evaluation tasks, while[23](https://arxiv.org/html/2608.17524#bib.bib14)caution that LLMs do not always replicate human behavior distributions\. In XRL specifically,[11](https://arxiv.org/html/2608.17524#bib.bib32)used LLMs both to generate explanations and to judge explanation quality\.[50](https://arxiv.org/html/2608.17524#bib.bib60)’s ALMANACS uses an LLM as an automated predictor and finds that no explanation method beats its no\-explanation baseline, a sobering precedent for why we include a strong no\-method floor\. Closer to EvalXRL’s design,[46](https://arxiv.org/html/2608.17524#bib.bib50)introduced an auditing game in which research teams tried to uncover a deliberately\-trained model’s hidden objective, and[14](https://arxiv.org/html/2608.17524#bib.bib58)extended this to autonomous investigator agents\.[59](https://arxiv.org/html/2608.17524#bib.bib57)’s AuditBench scales the format and reports atool\-to\-agent gap: tools that score well in standalone evaluations often fail to help an agentic investigator\.[74](https://arxiv.org/html/2608.17524#bib.bib61)’s Pando reaches a similar empirical floor in alignment auditing: most white\-box methods fail to outperform a budget\-matched black\-box prompting baseline\. We borrow the auditing\-game format and apply it to XRL by making an LLM coding agent the active surrogate user\. The tool\-to\-agent gap is a direct warning about EvalXRL’s methodology, which is part of why we include a strong no\-method baseline\.

Gap\.The combination we plan to fill: a head\-to\-head, application\-grounded benchmark for XRL methods, scored by repair\-task completion of deliberately broken RL agents, with an LLM coding agent as the scalable surrogate user\.

## 3Proposed Framework

### 3\.1Terminology

We use a small, deliberate vocabulary throughout\. Thecoderis the agent acting as the surrogate user; in our default configuration it is an LLM coding agent, but the framework is agnostic and a human could in principle play the same role\. TheRL agentis the reinforcement learning agent the coder is trying to repair, often calledbrokenormalfunctioningwhen it carries a deliberate fault\. Amalfunctionis that fault, injected as a unified diff to the training code or environment\. Each XRLmethodis packaged as adossiercontaining a bundle of papers, code, documentation, and analysis tools that are added to the coder’s context and command list\. Acellis one \(malfunction, method\) pair, the unit of replication in the experimental design\. Therepair scoreis the post\-repair RL agent’s normalized task performance, in\[0,1\]\[0,1\], computed by the harness, continuous \(never binary\), and averaged acrossNNreplications per cell\. Theharnessis the sandboxed Docker setup that runs each replication: two containers on an internal network with no internet access, one for the coder’s workspace, one for the immutable scoring environment\.

### 3\.2Architecture

The harness is a two\-container Docker setup, both containers cut off from the internet\. The first container holds the coder’s workspace: the environment source code, the malfunctioning RL agent’s parameters, the active XRL method, and an agent server that exposes the policy over the network\. The coder has standard coding\-agent tools \(shell, Python, file editing\) plus scoring and submission tools, and any additional tools the active method declares\. The second container holds an immutable copy of the environment, drives episodes against the agent server, and computes the repair score\. The coder cannot access or modify the second container, so scoring is tamper\-proof\. With no internet access, the coder also cannot look up solutions\.

### 3\.3Environments

We plan to evaluate on a mix of environments where repairing the RL agent has a clear real\-world analogue, all implemented in JAX[13](https://arxiv.org/html/2608.17524#bib.bib30)with fully JIT\-compiled training\. Our current candidate set:

- •Treasure Grid \(TG\): a6×66\\times 6gridworld with heterogeneous reward tiles \(large treasure\+10\+10, small treasure\+1\+1, pits−5\-5\)\.
- •Datacenter Cooling \(DC\): a simplified thermal model with server racks, cooling units, and an external weather model\. Continuous high\-dimensional observation, continuous action, multi\-component reward \(energy cost, thermal safety, equipment wear\)\. The repair scenario maps to a technician confronting a malfunctioning cooling agent and trying to restore acceptable performance on energy\-and\-temperature metrics\.
- •Traffic Light Control \(TLC\): a multi\-intersection signalized traffic network\. Discrete\-action signal phase selection, per\-intersection reward derived from standard transportation metrics \(vehicle delay, throughput, queue length\)\. The repair scenario maps to a transit engineer fixing an underperforming controller deployment\.

We are also considering CartPole as a small control\-task baseline, and applied alternatives such as energy grid balancing, building HVAC, inventory management, and packet routing\. We welcome the community’s suggestions\.

### 3\.4Malfunctions

Each malfunction is injected via a unified diff applied to the training code or environment\. The design draws on real\-fault taxonomies for DRL[53](https://arxiv.org/html/2608.17524#bib.bib35)and the RLMutation operator set[62](https://arxiv.org/html/2608.17524#bib.bib53), extended to XRL\-relevant failure modes such as reward hacking and non\-stationary control\.[21](https://arxiv.org/html/2608.17524#bib.bib52)showed that subtle code\-level details in PPO drive larger performance swings than the choice of algorithm itself, which validates that small ”boring” bugs are the right target\.

#### Candidate malfunctions\.

Diagnosing these requires reasoning about behavioral consequences, not just code inspection\. Several are instances of goal misgeneralization[38](https://arxiv.org/html/2608.17524#bib.bib56), where the agent retains capability out of distribution yet competently pursues an unintended proxy\. This is where XRL is supposed to help\.

- •Reward clippingon Treasure Grid: a widely\-used technique[51](https://arxiv.org/html/2608.17524#bib.bib29)that destroys the distinction between large and small treasures\.
- •Reward hackingon Treasure Grid: a step penalty that incentivizes hovering near small treasures rather than traversing the grid \(a pattern of the type studied by[56](https://arxiv.org/html/2608.17524#bib.bib36)and formalized by[60](https://arxiv.org/html/2608.17524#bib.bib37)\)\.
- •Mild myopiaon Treasure Grid: a discount factor of0\.30\.3instead of0\.990\.99\. The agent still cares about the future but undershoots distant rewards, settling for nearby small treasures\.
- •Reward imbalanceon Treasure Grid: the large\-treasure reward is cut from1010to1\.51\.5, making it locally rational to camp on small treasures rather than traverse to a large one\. Distinct from reward clipping \(which truncates during training\) and reward hacking \(which adds a perverse incentive\); this is a pure value\-tuning bug in the environment constants\.
- •Distributional shifton Datacenter Cooling: an agent trained on summer weather but deployed in winter\. No code bug\.
- •Sensor drifton Datacenter Cooling: a temperature sensor reading55°C higher, injected into the observation function\.
- •Reward hackingon Datacenter Cooling: oscillating setpoints that minimize an instantaneous\-energy term while violating multi\-step thermal constraints \(a reward\-hacking pattern of the type catalogued by[56](https://arxiv.org/html/2608.17524#bib.bib36)\)\.
- •Distributional shifton Traffic Light Control: an agent trained at off\-peak demand levels deployed at rush\-hour demand\. No code bug\.
- •Reward hackingon Traffic Light Control: oscillating signal phases that inflate per\-second throughput while increasing total queue length\.

### 3\.5Methods and Controls

The framework is agnostic to which XRL methods are packaged\. Every cell of the benchmark is reported alongside twocontrolsthat bracket the comparison from below and above and together implement the “fair fight” framing of[47](https://arxiv.org/html/2608.17524#bib.bib4): specify the affordances each method receives, then require it to beat the best non\-explanation baseline given the same affordances\.

- •No\-method baseline: the coder is told that no XRL method is available and must debug from source code, parameters, and the environment alone\. This is the lower control against which every XRL method is measured; H3 in particular is an explicit comparison against it\.
- •Cheat oracle: the coder is given a natural\-language description of the malfunction\. A soft upper bound on what any XRL method could provide; the symmetric counterpart to the no\-method baseline\.

The substantive XRL methods we plan to package between these two controls are:

- •Reward decomposition[34](https://arxiv.org/html/2608.17524#bib.bib19): classifies each reward by its source component, showing what the agent is optimizing for and what it is neglecting\.
- •Counterfactual analysis[65](https://arxiv.org/html/2608.17524#bib.bib42);[67](https://arxiv.org/html/2608.17524#bib.bib33): along representative trajectories, contrasts the agent’s actions or the resulting rewards against alternatives, surfacing what the agent is sensitive to\.
- •Action explanations[25](https://arxiv.org/html/2608.17524#bib.bib34): an LLM agent interrogates a counterfactual simulator and produces natural\-language rationales \(AXIS\)\.
- •Saliency / attribution maps[24](https://arxiv.org/html/2608.17524#bib.bib41);[44](https://arxiv.org/html/2608.17524#bib.bib40);[10](https://arxiv.org/html/2608.17524#bib.bib20): per\-feature importance scores, local for an individual decision or aggregated across the policy\.
- •Decision\-tree extraction[9](https://arxiv.org/html/2608.17524#bib.bib21): distills the policy into a readable tree\.
- •Programmatic policies[66](https://arxiv.org/html/2608.17524#bib.bib43): extracts a programmatic representation of the policy\.
- •Behavior summarization[3](https://arxiv.org/html/2608.17524#bib.bib44): surfaces the most informative trajectory snippets \(HIGHLIGHTS\)\.

We plan to start with the first three or four in the initial round and add others as resources allow\. We welcome the community’s suggestions for additional methods to package\.

### 3\.6Experimental Design

The independent variable is which method the coder receives\. The primary dependent variable is therepair score: the post\-repair agent’s task performance, normalized to\[0,1\]\[0,1\]viascore=clip\[0,1\]​\(Ppost−PbrokenPclean−Pbroken\)\\mathrm\{score\}=\\mathrm\{clip\}\_\{\[0,1\]\}\\\!\\left\(\\frac\{P\_\{\\text\{post\}\}\-P\_\{\\text\{broken\}\}\}\{P\_\{\\text\{clean\}\}\-P\_\{\\text\{broken\}\}\}\\right\), wherePpostP\_\{\\text\{post\}\}is the post\-repair agent’s environment\-specific performance,PbrokenP\_\{\\text\{broken\}\}is the unrepaired malfunctioning agent’s performance, andPcleanP\_\{\\text\{clean\}\}is the unmalfunctioned agent’s performance \(the natural reference, kept independent from the cheat oracle so the oracle remains a falsifiable upper bound rather than a definitional ceiling\)\. The score is continuous, never binary\. Throughout, a method’s headline result is reported as its*lift*over the no\-method baseline on the same cell \(its repair score minus the no\-method coder’s repair score\), so that the coder’s intrinsic debugging ability is differenced out and the benchmark credits a method only for the marginal value its explanations add\. Because the lower clip would conflate harm with no\-effect, we additionally report the un\-clipped valuescore~=\(Ppost−Pbroken\)/\(Pclean−Pbroken\)\\widetilde\{\\mathrm\{score\}\}=\(P\_\{\\text\{post\}\}\-P\_\{\\text\{broken\}\}\)/\(P\_\{\\text\{clean\}\}\-P\_\{\\text\{broken\}\}\)as a secondary outcome, and use it as the primary outcome for H3 \(where the direction of interest is below baseline\)\. A secondary dependent variable is the number of tool calls before submission, as a coarse efficiency proxy\. We considered a third dependent variable based on transcript analysis \(”did the coder correctly identify the root cause”\) but expect such judgment to be subjective and brittle, so we keep it only as a qualitative annotation rather than a primary measure\. Each cell is replicatedN≥6N\\geq 6times; we will set the per\-cellNNfrom a pilot variance estimate before the main round\. Few\-run RL evaluation is notoriously unstable[32](https://arxiv.org/html/2608.17524#bib.bib51), so we will report uncertainty intervals around per\-cell means rather than point estimates only, following the recommendations of[2](https://arxiv.org/html/2608.17524#bib.bib68)\.

## 4Hypotheses

We list three hypotheses about what the benchmark will measure\. We deliberately do*not*pre\-specify which methods will help with which malfunctions; our hypotheses are about properties of the \(method×\\timesmalfunction\) interaction structure\.

#### H1 \(interaction structure\)\.

The matrix of mean repair scores across \(method×\\timesmalfunction\) cells will exhibit non\-trivial structure: methods will cluster into groups with similar success profiles across malfunctions, and malfunctions will cluster into groups for which similar methods help\. We do not specify the clusters in advance: the test is whether the structure exists, not what shape it takes\. Operationally, the \(method×\\timesmalfunction\) interaction sum\-of\-squares from a two\-way decomposition of per\-cell repair scores will significantly exceed its null distribution under a permutation test that randomly permutes method labels within each malfunction \(we will report effect\-size estimates with stratified\-bootstrap confidence intervals over replications, following[2](https://arxiv.org/html/2608.17524#bib.bib68)\)\. If true, this would mean XRL methods cannot be ranked on a single axis, and the field should adopt multi\-condition evaluation suites rather than ranking on hand\-picked benchmarks\. The two ways H1 could fail are equally informative: a near\-uniform matrix would indicate no method is useful across the board; a rank\-1 matrix would indicate methods differ only in overall strength, not in what they expose\.

#### H2 \(oracle ceiling\)\.

The cheat oracle will set a high empirical bound but will not always reach a perfect score, because implementing a fix can be harder than diagnosing it\. We treat the oracle as an empirical, not a theoretical, ceiling: a high\-quality XRL trace could surface fix\-relevant information more directly than a natural\-language bug description does, and an XRL method that exceeds the oracle on some cell would itself be an interesting finding rather than a contradiction\. The size of the gap between the cheat oracle and an optimal repair will quantify how much of the difficulty is diagnosis versus engineering, which is itself a useful measurement\.

#### H3 \(methods can hurt\)\.

For at least one \(method, malfunction\) cell, the XRL method will produce a mean repair score significantly below the no\-method baseline, under a one\-tailed paired comparison appropriate for the bounded outcome \(e\.g\., Wilcoxon signed\-rank or bootstrap difference\-in\-means\)\. H3 is an existence\-of\-discovery claim: we therefore use Benjamini–Hochberg false\-discovery\-rate control atq=0\.05q=0\.05across all cells, rather than family\-wise \(Holm\) correction, which would be unattainably strict for a grid of this size\. We do not pre\-specify which cells; the prediction is that the harm\-effect exists somewhere\. We expect this pattern to occur when a method confidently reports a true\-but\-not\-causal symptom that misdirects the coder away from the actual failure mode\. This mirrors thetool\-to\-agent gapreported by[59](https://arxiv.org/html/2608.17524#bib.bib57)in alignment auditing, and is consistent with[36](https://arxiv.org/html/2608.17524#bib.bib67), who showed empirically that misleading explanations can manipulate user trust in tabular classification settings\. It is the most surprising prediction we make: a sound interpretability tool can have negative downstream value when its output is well\-formed but cause\-irrelevant for the malfunction at hand\. Falsifying H3 \(no cell satisfies the threshold above\) would be informative in the other direction, suggesting that XRL methods are at worst inert rather than misleading\.

## 5Open Design Questions

This section collects open questions in the EvalXRL design: some are limitations we acknowledge, some are extensions we plan to explore, and many are both\. Each entry states the issue and our current thinking, and is offered as a place where community input would shape the eventual benchmark\.

#### LLM\-vs\-human validity gap\.

EvalXRL measures whether an explanation contains actionable information for an LLM coder, not for a human one\. LLMs read structured text differently than humans read visualizations, and an XRL method that scores well with one need not score well with the other\. The LLM coder is a fairly literal reader, so a method that fails the LLM test is unlikely to help a human practitioner, but the converse is not guaranteed\. The natural calibration is a small \(N∼12\{\\sim\}12\) semi\-structured interview study comparing the coder’s verdict on each cell against SWE\-savvy human readers via inter\-rater agreement \(e\.g\. Cohen’sκ\\kappa\), using the objective debugging measures of[26](https://arxiv.org/html/2608.17524#bib.bib1)rather than repair task performance \(the latter would confound explanation quality with the user’s implementation skill\)\. Thematic coding of interview transcripts would also surface qualitative failure modes the score alone would miss\. We treat EvalXRL as a filter, not a substitute, until that calibration is run\.

#### Closed\-loop use is not unconditionally helpful\.

Our design choice to allow closed\-loop invocation reflects a longstanding argument for dialogical explanation, but the empirical case for closed\-loop over one\-shot use is mixed\.[4](https://arxiv.org/html/2608.17524#bib.bib27)’s Study 2 found that participants using a non\-interactive HIGHLIGHTS baseline produced more correct initial hypotheses than ASQ\-IT participants did, with ASQ\-IT’s advantage emerging only after iterative verification\.[69](https://arxiv.org/html/2608.17524#bib.bib69)’s discussion reports that users often did not know which parameters to start with when iteratively editing inputs\.[39](https://arxiv.org/html/2608.17524#bib.bib72)report mixed accuracy effects from hypothesis\-driven decision support across user populations\. We adopt closed\-loop use as a design choice on conceptual grounds and treat its actual benefit as an empirical question\. A natural extension is to vary the per\-session invocation budget down to a single invocation in the limit, which would isolate how much of each method’s utility comes from closed\-loop use versus one\-shot consumption\.

#### Method packaging\.

The unit we actually evaluate is a dossier: a bundle of README, papers, source, and examples handed to the coder\. Two competent dossiers for the same underlying method can differ in prose, examples, and visualization helpers, and could yield meaningfully different repair scores\. A published EvalXRL number is therefore a property of \(method×\\timespackaging\), not of the method alone, so method authors could improve scores by improving packaging without improving the method\. We mitigate by publishing every dossier and inviting authors to submit their preferred packaging\. A direct probe of the confound would be a rich\-versus\-minimal dossier ablation pairing each method with two packaging variants of differing length and detail, which would quantify how much of the per\-method score is attributable to packaging quality\.

#### Scaffold choice\.

Three candidate scaffolds bracket the design space\. \(i\) A*minimalist scaffold*likemini\-swe\-agent[61](https://arxiv.org/html/2608.17524#bib.bib74)gives the LLM only a sandboxed shell, with no specialized tools beyond bash\. This is the cleanest measurement instrument: most variance attaches to the LLM and to the XRL method \(the variables we want to vary\), not to scaffold cleverness; the choice also aligns with the method\-minimalism principle of[52](https://arxiv.org/html/2608.17524#bib.bib5), who advocate trying simpler tools before fancier ones\. We currently treat this as the default\. \(ii\) An*engineered Agent\-Computer Interface*likeSWE\-agent[72](https://arxiv.org/html/2608.17524#bib.bib22)adds purpose\-built tools \(file viewer, structured editor, syntax linter\) and lifts SWE\-bench resolution by∼10\{\\sim\}10percentage points over a vanilla shell baseline; the cost is framework lock\-in to choices made for an earlier model era\. \(iii\) A*self\-evolving*scaffold likeLive\-SWE\-agent[70](https://arxiv.org/html/2608.17524#bib.bib73)lets the agent synthesize its own ad\-hoc tools per task; the empirical risk for EvalXRL is that the agent reinvents probing tools \(saliency dumps, trajectory inspectors, value plotters\) that overlap the function of the XRL method being evaluated, narrowing the measurable XRL contribution\. We see no clean answer in advance: the measurement\-cleanness vs\. scaffold\-capability tradeoff is exactly the kind of choice we expect community input to shape\.

#### Synthetic malfunctions\.

We suggest a hand\-designed catalog of malfunctions \(reward clipping, mild myopia, reward imbalance, distributional shift, sensor drift, reward hacking\) rather than a sample of bugs that actually appear in deployed RL systems\. The wild distribution would include data\-pipeline errors, version drift between training and serving, RNG seed leaks, and emergent reward\-hacking, none of which our synthetic catalog easily reproduces\. Strong performance on EvalXRL is therefore evidence about utility on*our*bug distribution, not the wild one\. Sampling bugs from real RL incident reports or from the issue trackers of major RL libraries is a possible direction, though such bugs are often confounded with project\-specific context that resists isolation\.

#### Access model\.

EvalXRL evaluates XRL methods under a white\-box regime: the coder receives full source, weights, and the retraining loop\. We treat this as the easier evaluation regime: a method that fails to help under full access is unlikely to help under the restricted black\-box access of proprietary agents\. A black\-box variant would require a separate intervention API and a smaller applicable\-method set \(saliency, programmatic\-policy extraction, and decision\-tree distillation all require model internals\)\.

#### Memorization and self\-evaluation\.

The sandbox blocks test\-time leakage but not pretraining memorization: a frontier LLM’s training corpus likely contains the bug archetypes, the XRL libraries, and the failure modes[21](https://arxiv.org/html/2608.17524#bib.bib52)catalog\.[27](https://arxiv.org/html/2608.17524#bib.bib62)confirm this concretely in the adjacent setting of mechanistic interpretability evaluation: Claude Opus 4\.1, asked cold, can recite the exact attention\-head indices and roles of the canonical Indirect Object Identification \(IOI\) circuit from memory, and the GPT\-5 judge shows similar recall\. Mitigations: \(i\) amemorization probe\(no\-method coder identifying the bug from a description alone, before environment access\); \(ii\) a held\-out class ofnovel composite malfunctionsin the spirit of[29](https://arxiv.org/html/2608.17524#bib.bib59); and \(iii\) reporting the no\-method baseline so memorization\-only “improvements” stay visible\. Using one LLM family to evaluate methods consumed by the same family is also circular, so we plan a robustness check across a panel of additional model families\.

#### One axis of explanation utility\.

EvalXRL measures one thing: how much an explanation helps fix a malfunctioning agent\. Real explanations serve other purposes too\. They calibrate user trust, support accountability, build the user’s mental model of the agent’s reasoning, and communicate decisions to stakeholders\. An explanation method could excel at any of these while contributing nothing to repair, and the converse also holds: a method that boosts our repair score might do so by short\-circuiting understanding rather than building it\. We do not claim repair\-contribution is the most important purpose of an explanation, only that it is one important purpose, easy to measure, and currently absent from the field’s standard evaluation toolbox\.

#### Joint use of multiple methods\.

The benchmark default gives the coder one method at a time\. Real users may consult several methods simultaneously, and complementarity may matter, as[5](https://arxiv.org/html/2608.17524#bib.bib26)observed: counterfactual outcomes combined with reward decomposition outperform either alone\. An extension equips the coder with multiple methods at once and measures each method’s Shapley contribution to the repair score, isolating which method actually pulls weight in a given malfunction context\.

#### Cross\-environment composition\.

Each environment in the current set is self\-contained\. A composed environment \(e\.g\., a datacenter cooling agent that also handles dynamic pricing\) would test how methods scale when a single agent is responsible for multiple coupled domains\. The resulting interaction effects might surface different XRL strengths than the single\-domain setup\.

#### Generalization beyond reinforcement learning\.

The same shape of evaluation transports to large language models\. The coder, the dossier\-vs\-baseline structure, the repair\-scored metric, and the two\-container harness all carry over directly; what changes is the substrate \(a deliberately malfunctioning open\-source LLM at the 7–9B parameter scale\) and the method space \(mech\-interp methods such as sparse autoencoder features, steering vectors, linear probes, activation and attribution patching, and the logit and tuned lenses, alongside matched\-affordance behavioral baselines such as prompting, prefill, and chain\-of\-thought reading, which the current LLM\-interpretability discourse[52](https://arxiv.org/html/2608.17524#bib.bib5);[47](https://arxiv.org/html/2608.17524#bib.bib4)treats as first\-class competitors\)\. The malfunction catalog also shifts: published model\-organism recipes \(emergent misalignment, sleeper\-agent backdoors, refusal\-direction ablation, synthetic\-document implanted beliefs, persona drift, eval\-awareness training\) produce controlled LoRA\-scale faults that play the same role here as our per\-environment injected diffs\. The empirical regularity that simpler behavioral methods often match or beat mech\-interp methods on adjacent tasks[50](https://arxiv.org/html/2608.17524#bib.bib60);[59](https://arxiv.org/html/2608.17524#bib.bib57)gives this variant sharper baseline\-controlled stakes than the RL setting allows; whether that pattern transfers to a repair\-scored benchmark is the natural pilot for a separate paper\.

## 6Conclusion

We propose EvalXRL, a benchmark that evaluates XRL methods by how well they help a coder diagnose and repair RL agents\. If our hypotheses hold, EvalXRL will give the XRL community a common\-test, low\-cost way to measure the downstream utility of explanations, complementing the more expensive but more ecologically valid human studies the field cannot run at scale\. As a workshop\-stage proposal, we would especially value the community’s feedback on \(i\) whether our malfunction taxonomy covers the failure modes XRL practitioners actually care about, \(ii\) whether the LLM\-as\-surrogate framing is a defensible first\-pass methodology or whether the validity gap is too large to take useful conclusions from, and \(iii\) which additional XRL methods the community would most want to see packaged in the first public release of the benchmark\.

## Acknowledgments

We thank our colleagues for helpful discussions and feedback: David Aha, Nitay Alon, Jérôme Botoko Ekila, Eli Bronstein, David Manheim, and Yonatan Nakar\.

## References

- Adebayoet al\.\(2020\)J\. Adebayo, M\. Muelly, I\. Liccardi, and B\. KimDebugging tests for model explanations\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Agarwalet al\.\(2021\)R\. Agarwal, M\. Schwarzer, P\. S\. Castro, A\. Courville, and M\. G\. BellemareDeep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.6](https://arxiv.org/html/2608.17524#S3.SS6.p1.1),[§4](https://arxiv.org/html/2608.17524#S4.SS0.SSS0.Px1.p1.1)\.
- Amir and Amir \(2018\)D\. Amir and O\. AmirHIGHLIGHTS: summarizing agent behavior to people\.InInternational Conference on Autonomous Agents and Multiagent Systems,Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p2.1),[7th item](https://arxiv.org/html/2608.17524#S3.I4.i7.p1.1)\.
- Amitaiet al\.\(2024a\)Y\. Amitai, O\. Amir, and G\. AvniASQ\-IT: interactive explanations for reinforcement\-learning agents\.Artificial Intelligence335,pp\. 104182\.Cited by:[item 2](https://arxiv.org/html/2608.17524#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px2.p1.1)\.
- Amitaiet al\.\(2024b\)Y\. Amitai, Y\. Septon, and O\. AmirExplaining reinforcement learning agents through counterfactual action outcomes\.InAAAI Conference on Artificial Intelligence,Vol\.38,pp\. 10003–10011\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i9.28863)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px9.p1.1)\.
- Andersonet al\.\(2019\)A\. Anderson, J\. Dodge, A\. Sadarangani, Z\. Juozapaitis, E\. Newman, J\. Irvine, S\. Chattopadhyay, A\. Fern, and M\. BurnettExplaining reinforcement learning to mere mortals: an empirical study\.InInternational Joint Conference on Artificial Intelligence,pp\. 1328–1334\.Cited by:[Table 1](https://arxiv.org/html/2608.17524#S1.T1.4.4.1.1.1)\.
- Arzate Cruz and Igarashi \(2021\)C\. Arzate Cruz and T\. IgarashiInteractive explanations: diagnosis and repair of reinforcement learning based agent behaviors\.InIEEE Conference on Games \(CoG\),pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/CoG52621.2021.9618999)Cited by:[item 2](https://arxiv.org/html/2608.17524#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Barezet al\.\(2026\)F\. Barezet al\.Automated interpretability\-driven model auditing and control: a research agenda\.Note:Oxford Martin AI Governance Initiative \(AIGI\)External Links:[Link](https://aigi.ox.ac.uk/publications/automated-interpretability-driven-model-auditing-and-control-a-research-agenda)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1)\.
- Bastaniet al\.\(2018\)O\. Bastani, Y\. Pu, and A\. Solar\-LezamaVerifiable reinforcement learning via policy extraction\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p2.1),[5th item](https://arxiv.org/html/2608.17524#S3.I4.i5.p1.1)\.
- Beecheyet al\.\(2023\)D\. Beechey, T\. M\. S\. Smith, and Ö\. ŞimşekExplaining reinforcement learning with shapley values\.InInternational Conference on Machine Learning,pp\. 2003–2014\.Cited by:[4th item](https://arxiv.org/html/2608.17524#S3.I4.i4.p1.1)\.
- Belouadahet al\.\(2025\)A\. Belouadah, M\. L\. Ruiz\-Rodríguez, S\. Kubler, and Y\. Le TraonEvaluating the effectiveness of LLMs for explainable deep reinforcement learning\.Machine Learning with Applications22,pp\. 100795\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.
- Bhattet al\.\(2020\)U\. Bhatt, A\. Xiang, S\. Sharma, A\. Weller, A\. Taly, Y\. Jia, J\. Ghosh, R\. Puri, J\. M\. F\. Moura, and P\. EckersleyExplainable machine learning in deployment\.InACM Conference on Fairness, Accountability, and Transparency,pp\. 648–657\.External Links:[Document](https://dx.doi.org/10.1145/3351095.3375624)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p8.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Bradburyet al\.\(2018\)J\. Bradbury, R\. Frostig, P\. Hawkins, M\. J\. Johnson, Y\. Katariya, C\. Leary, D\. Maclaurin, G\. Necula, A\. Paszke, J\. VanderPlas, S\. Wanderman\-Milne, and Q\. ZhangJAX: composable transformations of Python\+NumPy programs\.External Links:[Link](http://github.com/jax-ml/jax)Cited by:[§3\.3](https://arxiv.org/html/2608.17524#S3.SS3.p1.1)\.
- Brickenet al\.\(2025\)T\. Bricken, R\. Wang, S\. Bowman, E\. Ong, J\. Treutlein, J\. Wu, E\. Hubinger, and S\. MarksBuilding and evaluating alignment auditing agents\.Note:Anthropic Alignment Science BlogExternal Links:[Link](https://alignment.anthropic.com/2025/automated-auditing/)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.
- Chanet al\.\(2024\)J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan, L\. Weng, and A\. MadryMLE\-bench: evaluating machine learning agents on machine learning engineering\.arXiv preprint arXiv:2410\.07095\.Note:ICLR 2025Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1)\.
- Chenet al\.\(2026\)H\. Chen, S\. R\. Grimm, O\. Russakovsky, and T\. LombrozoMachine understanding\.Trends in Cognitive Sciences\.External Links:ISSN 1364\-6613,[Document](https://dx.doi.org/10.1016/j.tics.2026.04.003),[Link](https://www.sciencedirect.com/science/article/pii/S136466132600077X)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1)\.
- De Bonaet al\.\(2024\)F\. B\. De Bona, G\. Dominici, T\. Miller, M\. Langheinrich, and M\. GjoreskiEvaluating explanations through LLMs: beyond traditional user studies\.InNeurIPS Workshop on Generative AI for Health,Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.
- de Regt \(2017\)H\. W\. de RegtUnderstanding scientific understanding\.Oxford University Press\.External Links:[Document](https://dx.doi.org/10.1093/oso/9780190652913.001.0001)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p3.1)\.
- Dodgeet al\.\(2021\)J\. Dodge, R\. Khanna, J\. Irvine, K\. Lam, T\. Mai, Z\. Lin, N\. Kiddle, E\. Newman, A\. Anderson, S\. Raja, C\. Matthews, C\. Perdriau, M\. Burnett, and A\. FernAfter\-action review for AI \(AAR/AI\)\.ACM Transactions on Interactive Intelligent Systems\.Cited by:[item 2](https://arxiv.org/html/2608.17524#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1)\.
- Doshi\-Velez and Kim \(2017\)F\. Doshi\-Velez and B\. KimTowards a rigorous science of interpretable machine learning\.arXiv preprint arXiv:1702\.08608\.Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1)\.
- Engstromet al\.\(2020\)L\. Engstrom, A\. Ilyas, S\. Santurkar, D\. Tsipras, F\. Janoos, L\. Rudolph, and A\. MadryImplementation matters in deep policy gradients: a case study on PPO and TRPO\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§3\.4](https://arxiv.org/html/2608.17524#S3.SS4.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px7.p1.1)\.
- Frostet al\.\(2021\)J\. Frost, O\. Watkins, E\. Weiner, P\. Abbeel, T\. Darrell, B\. Plummer, and K\. SaenkoExplaining reinforcement learning policies through counterfactual trajectories\.InICML Workshop on Human in the Loop Learning \(HILL\),Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2025\)Y\. Gao, D\. Lee, G\. Burtch, and S\. FazelpourTake caution in using LLMs as human surrogates\.Proceedings of the National Academy of Sciences122\(24\),pp\. e2501660122\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2501660122)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.
- Greydanuset al\.\(2018\)S\. Greydanus, A\. Koul, J\. Dodge, and A\. FernVisualizing and understanding Atari agents\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p2.1),[4th item](https://arxiv.org/html/2608.17524#S3.I4.i4.p1.1)\.
- Gyevnaret al\.\(2026\)B\. Gyevnar, C\. G\. Lucas, S\. V\. Albrecht, and S\. B\. CohenIntegrating counterfactual simulations with language models for explaining multi\-agent behaviour\.InInternational Conference on Autonomous Agents and Multiagent Systems,Cited by:[3rd item](https://arxiv.org/html/2608.17524#S3.I4.i3.p1.1)\.
- Gyevnar and Towers \(2025\)B\. Gyevnar and M\. TowersObjective metrics for human\-subjects evaluation in explainable reinforcement learning\.InMulti\-disciplinary Conference on Reinforcement Learning and Decision Making \(RLDM\),Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1),[§2](https://arxiv.org/html/2608.17524#S2.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px1.p1.1)\.
- Haklayet al\.\(2026\)T\. Haklay, N\. Prakash, S\. Pandey, A\. Torralba, A\. Mueller, J\. Andreas, T\. Rott Shaham, and Y\. BelinkovPitfalls in evaluating interpretability agents\.arXiv preprint arXiv:2603\.20101\.Cited by:[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px7.p1.1)\.
- Haqueet al\.\(2025\)M\. Haque, P\. Babkin, F\. Farmahinifarahani, and M\. VelosoTowards effectively leveraging execution traces for program repair with code LLMs\.InProceedings of the 4th International Workshop on Knowledge\-Augmented Methods for Natural Language Processing \(KnowledgeNLP\),pp\. 160–179\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1)\.
- Hariharanet al\.\(2025\)K\. Hariharan, U\. Girit, A\. Wang, and J\. AndreasBreakpoint: scalable evaluation of system\-level reasoning in LLM code agents\.arXiv preprint arXiv:2506\.00172\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px7.p1.1)\.
- Hase and Bansal \(2020\)P\. Hase and M\. BansalEvaluating explainable AI: which algorithmic explanations help users predict model behavior?\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 5540–5552\.Cited by:[Table 1](https://arxiv.org/html/2608.17524#S1.T1.4.4.1.1.1)\.
- Hayes and Shah \(2017\)B\. Hayes and J\. A\. ShahImproving robot controller transparency through autonomous policy explanation\.InACM/IEEE International Conference on Human\-Robot Interaction \(HRI\),pp\. 303–312\.External Links:[Document](https://dx.doi.org/10.1145/2909824.3020233)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InAAAI Conference on Artificial Intelligence,Cited by:[§3\.6](https://arxiv.org/html/2608.17524#S3.SS6.p1.1)\.
- Hoffmanet al\.\(2018\)R\. R\. Hoffman, S\. T\. Mueller, G\. Klein, and J\. LitmanMetrics for explainable AI: challenges and prospects\.arXiv preprint arXiv:1812\.04608\.Cited by:[Table 1](https://arxiv.org/html/2608.17524#S1.T1.4.2.1.1.1)\.
- Juozapaitiset al\.\(2019\)Z\. Juozapaitis, A\. Koul, A\. Fern, M\. Erwig, and F\. Doshi\-VelezExplainable reinforcement learning via reward decomposition\.InIJCAI Workshop on Explainable Artificial Intelligence \(XAI\),Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p2.1),[1st item](https://arxiv.org/html/2608.17524#S3.I4.i1.p1.1)\.
- Kulibabaet al\.\(2025\)S\. Kulibaba, A\. Dzhalilov, R\. Pakhomov, O\. Svidchenko, A\. Gasnikov, and A\. ShpilmanKompeteAI: Accelerated Autonomous Multi\-Agent System for End\-to\-End Pipeline Generation for Machine Learning Problems\.arXiv\.External Links:2508\.10177,[Document](https://dx.doi.org/10.48550/arXiv.2508.10177)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p8.1)\.
- Lakkaraju and Bastani \(2020\)H\. Lakkaraju and O\. Bastani“How do I fool you?”: manipulating user trust via misleading black box explanations\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,pp\. 79–85\.Cited by:[§4](https://arxiv.org/html/2608.17524#S4.SS0.SSS0.Px3.p1.1)\.
- Lakkarajuet al\.\(2022\)H\. Lakkaraju, D\. Slack, Y\. Chen, C\. Tan, and S\. SinghRethinking explainability as a dialogue: a practitioner’s perspective\.arXiv preprint arXiv:2202\.01875\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1)\.
- Langoscoet al\.\(2022\)L\. Langosco, J\. Koch, L\. D\. Sharkey, J\. Pfau, and D\. KruegerGoal misgeneralization in deep reinforcement learning\.InInternational Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.162,pp\. 12004–12019\.Cited by:[§3\.4](https://arxiv.org/html/2608.17524#S3.SS4.SSS0.Px1.p1.1)\.
- Leet al\.\(2024\)T\. Le, T\. Miller, L\. Sonenberg, R\. Singh, and H\. P\. SoyerFrom evidence to decision: exploring evaluative AI\.arXiv preprint arXiv:2402\.01292\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px2.p1.1)\.
- Levinet al\.\(2025\)K\. H\. Levin, N\. van Kempen, E\. D\. Berger, and S\. N\. FreundChatDBG: augmenting debugging with large language models\.Proceedings of the ACM on Software Engineering2\(FSE\)\.External Links:[Document](https://dx.doi.org/10.1145/3729355)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1)\.
- Lipton \(2009\)P\. LiptonUnderstanding without explanation\.InScientific Understanding: Philosophical Perspectives,H\. W\. de Regt, S\. Leonelli, and K\. Eigner \(Eds\.\),pp\. 43–63\.Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p3.1)\.
- Liuet al\.\(2025\)Z\. Liu, Y\. Cai, X\. Zhu, Y\. Zheng, R\. Chen, Y\. Wen, Y\. Wang, W\. E, and S\. ChenML\-Master: Towards AI\-for\-AI via Integration of Exploration and Reasoning\.arXiv\.External Links:2506\.16499,[Document](https://dx.doi.org/10.48550/arXiv.2506.16499)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p8.1)\.
- Lombrozo \(2006\)T\. LombrozoThe structure and function of explanations\.Trends in Cognitive Sciences10\(10\),pp\. 464–470\.External Links:[Document](https://dx.doi.org/10.1016/j.tics.2006.08.004)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1)\.
- Lundberg and Lee \(2017\)S\. M\. Lundberg and S\. LeeA unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems,Cited by:[4th item](https://arxiv.org/html/2608.17524#S3.I4.i4.p1.1)\.
- Madumalet al\.\(2018\)P\. Madumal, T\. Miller, F\. Vetere, and L\. SonenbergTowards a grounded dialog model for explainable artificial intelligence\.arXiv preprint arXiv:1806\.08055\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1)\.
- Markset al\.\(2025\)S\. Marks, J\. Treutlein, T\. Bricken, J\. Lindsey, J\. Marcus, S\. Mishra\-Sharma, D\. Ziegler,et al\.Auditing language models for hidden objectives\.arXiv preprint arXiv:2503\.10965\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.
- Marks \(2025\)S\. MarksDownstream applications as validation of interpretability progress\.Note:LessWrong / Alignment ForumExternal Links:[Link](https://www.lesswrong.com/posts/wGRnzCFcowRCrpX4Y/downstream-applications-as-validation-of-interpretability)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1),[§3\.5](https://arxiv.org/html/2608.17524#S3.SS5.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px11.p1.1)\.
- Miller \(2019\)T\. MillerExplanation in artificial intelligence: insights from the social sciences\.Artificial Intelligence267,pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p1.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1)\.
- Miller \(2023\)T\. MillerExplainable AI is dead, long live explainable AI\! hypothesis\-driven decision support using evaluative AI\.InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency \(FAccT\),Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1)\.
- Millset al\.\(2023\)E\. Mills, S\. Su, S\. Russell, and S\. EmmonsALMANACS: a simulatability benchmark for language model explainability\.arXiv preprint arXiv:2312\.12747\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px11.p1.1)\.
- Mnihet al\.\(2015\)V\. Mnih, K\. Kavukcuoglu, D\. Silver, A\. A\. Rusu, J\. Veness, M\. G\. Bellemare, A\. Graves, M\. Riedmiller, A\. K\. Fidjeland, G\. Ostrovski,et al\.Human\-level control through deep reinforcement learning\.Nature518\(7540\),pp\. 529–533\.Cited by:[1st item](https://arxiv.org/html/2608.17524#S3.I2.i1.p1.1)\.
- Nandaet al\.\(2025\)N\. Nanda, J\. Engels, A\. Conmy, S\. Rajamanoharan, B\. Chughtai, C\. McDougall, J\. Kramár, and L\. SmithA pragmatic vision for interpretability\.Note:LessWrong / Alignment ForumExternal Links:[Link](https://www.lesswrong.com/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability)Cited by:[§1](https://arxiv.org/html/2608.17524#S1.p4.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px11.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px4.p1.1)\.
- Nikanjamet al\.\(2022\)A\. Nikanjam, M\. M\. Morovati, F\. Khomh, and H\. Ben BraiekFaults in deep reinforcement learning programs: a taxonomy and a detection approach\.Automated Software Engineering29\(8\)\.Cited by:[§3\.4](https://arxiv.org/html/2608.17524#S3.SS4.p1.1)\.
- Olsonet al\.\(2021\)M\. L\. Olson, R\. Khanna, L\. Neal, F\. Li, and W\. WongCounterfactual state explanations for reinforcement learning agents via generative deep learning\.Artificial Intelligence295,pp\. 103455\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Palejaet al\.\(2021\)R\. Paleja, M\. Ghuy, N\. R\. Arachchige, R\. Jensen, and M\. GombolayThe utility of explainable AI in ad hoc human\-machine teaming\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Panet al\.\(2022\)A\. Pan, K\. Bhatia, and J\. SteinhardtThe effects of reward misspecification: mapping and mitigating misaligned models\.InInternational Conference on Learning Representations,Cited by:[2nd item](https://arxiv.org/html/2608.17524#S3.I2.i2.p1.1),[7th item](https://arxiv.org/html/2608.17524#S3.I2.i7.p1.1)\.
- Septonet al\.\(2023\)Y\. Septon, T\. Huber, E\. André, and O\. AmirIntegrating policy summaries with reward decomposition for explaining reinforcement learning agents\.InInternational Conference on Practical Applications of Agents and Multi\-Agent Systems \(PAAMS\),pp\. 320–332\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-37616-0%5F27)Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px1.p1.1)\.
- Sequeira and Gervasio \(2020\)P\. Sequeira and M\. GervasioInterestingness elements for explainable reinforcement learning: understanding agents’ capabilities and limitations\.Artificial Intelligence288,pp\. 103367\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Sheshadriet al\.\(2026\)A\. Sheshadri, A\. Ewart, K\. Fronsdal, I\. Gupta, S\. R\. Bowman, S\. Price, S\. Marks, and R\. WangAuditBench: evaluating alignment auditing techniques on models with hidden behaviors\.arXiv preprint arXiv:2602\.22755\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1),[§4](https://arxiv.org/html/2608.17524#S4.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px11.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward hacking\.InAdvances in Neural Information Processing Systems,External Links:2209\.13085Cited by:[2nd item](https://arxiv.org/html/2608.17524#S3.I2.i2.p1.1)\.
- SWE\-agent Team \(2025\)SWE\-agent TeamMini\-swe\-agent\.Note:[https://github\.com/SWE\-agent/mini\-swe\-agent](https://github.com/SWE-agent/mini-swe-agent)Software repositoryCited by:[item 3](https://arxiv.org/html/2608.17524#S1.I1.i3.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px4.p1.1)\.
- Tambonet al\.\(2023\)F\. Tambon, V\. Majdinasab, A\. Nikanjam, F\. Khomh, and G\. AntoniolMutation testing of deep reinforcement learning based on real faults\.InIEEE International Conference on Software Testing, Verification and Validation \(ICST\),Cited by:[§3\.4](https://arxiv.org/html/2608.17524#S3.SS4.p1.1)\.
- Tappleret al\.\(2025\)M\. Tappler, I\. D\. Lopez\-Miguel, S\. Tschiatschek, and E\. BartocciRule\-guided reinforcement learning policy evaluation and improvement\.InInternational Joint Conference on Artificial Intelligence,Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px2.p1.1)\.
- Towerset al\.\(2025\)M\. Towers, Y\. Du, C\. Freeman, and T\. J\. NormanA comparative user evaluation of XRL explanations using goal identification\.arXiv preprint arXiv:2510\.16956\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px1.p1.1)\.
- van der Waaet al\.\(2018\)J\. van der Waa, J\. van Diggelen, K\. van den Bosch, and M\. NeerincxContrastive explanations for reinforcement learning in terms of expected consequences\.InIJCAI Workshop on Explainable Artificial Intelligence \(XAI\),Cited by:[2nd item](https://arxiv.org/html/2608.17524#S3.I4.i2.p1.1)\.
- Vermaet al\.\(2018\)A\. Verma, V\. Murali, R\. Singh, P\. Kohli, and S\. ChaudhuriProgrammatically interpretable reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[6th item](https://arxiv.org/html/2608.17524#S3.I4.i6.p1.1)\.
- Wehneret al\.\(2024\)J\. Wehner, F\. A\. Oliehoek, and L\. Cavalcante SiebertExplaining learned reward functions with counterfactual trajectories\.InECAI Workshop on Implementing AI Ethics through a Behavioural Lens \(AIEB\),Cited by:[2nd item](https://arxiv.org/html/2608.17524#S3.I4.i2.p1.1)\.
- Wijket al\.\(2024\)H\. Wijk, T\. Lin, J\. Becker, S\. Jawhar, N\. Parikh, T\. Broadley, L\. Chan, M\. Chen, J\. Clymer, J\. Dhyani,et al\.RE\-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts\.arXiv preprint arXiv:2411\.15114\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1)\.
- Wuet al\.\(2022\)X\. Wu, Z\. Yan, C\. Zhang, and T\. WuDecisions that explain themselves: a user\-centric deep reinforcement learning explanation system\.arXiv preprint arXiv:2212\.00888\.Cited by:[item 2](https://arxiv.org/html/2608.17524#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px2.p1.1)\.
- Xiaet al\.\(2025\)C\. S\. Xiaet al\.Live\-SWE\-agent: can software engineering agents self\-evolve on the fly?\.arXiv preprint\.Cited by:[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px4.p1.1)\.
- Xionget al\.\(2024\)Y\. Xiong, Z\. Hu, Y\. Huang, R\. Wu, K\. Guan, X\. Fang, J\. Jiang, T\. Zhou, Y\. Hu, H\. Liu, T\. Lyu, and C\. FanXRL\-Bench: a benchmark for evaluating and comparing explainable reinforcement learning techniques\.Note:arXiv preprint arXiv:2402\.12685Cited by:[Table 1](https://arxiv.org/html/2608.17524#S1.T1.4.3.1.1.1)\.
- Yanget al\.\(2024\)J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. PressSWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2405\.15793Cited by:[§5](https://arxiv.org/html/2608.17524#S5.SS0.SSS0.Px4.p1.1)\.
- Yuanet al\.\(2025\)X\. Yuan, M\. M\. Moss, C\. El Feghali, C\. Singh, D\. Moldavskaya, D\. MacPhee, L\. Caccia, M\. Pereira, M\. Kim, A\. Sordoni, and M\. Côtédebug\-gym: a text\-based environment for interactive debugging\.arXiv preprint arXiv:2503\.21557\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px4.p1.1)\.
- Zhonget al\.\(2026\)Z\. Zhong, A\. Muhamed, M\. T\. Diab, V\. Smith, and A\. RaghunathanPando: do interpretability methods work when models won’t explain themselves?\.arXiv preprint arXiv:2604\.11061\.Cited by:[§2](https://arxiv.org/html/2608.17524#S2.SS0.SSS0.Px5.p1.1)\.