Explaining AI Agents Through Execution Traces

# Explaining AI Agents Through Execution Traces
Source: [https://arxiv.org/html/2609.06063](https://arxiv.org/html/2609.06063)
Vittoria Vineis††thanks:Corresponding author: vineis@diag\.uniroma1\.it\. In this preprint version, the source code and trace dataset are are available upon request\.Lorenzo AntonelliClaudia Di CarloMatteo SilvestriGabriele Tolomei

###### Abstract

AI Agents are increasingly deployed in real\-world settings, where they interact with external tools and make sequential decisions with limited human oversight\. This creates a pressing need for reliable and auditable explanations of what an agent did and why\. However, traditional Explainable AI \(XAI\) methods fall short of providing the process\-level transparency required for such interactive, multi\-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents\. To address this gap, we present a post\-hoc XAI framework that transforms a lengthy agent’s execution trace into a structured report and a faithful natural\-language explanation explicitly grounded in its observable behavior\. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks\. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high\-quality, trace\-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM\-generated explanations\.

1Sapienza University of Rome, Rome, Italy

## 1Introduction

Recent advances in Agentic AI are reshaping the capabilities of AI systems by enabling autonomous goal pursuit, adaptive decision\-making, and complex task execution with minimal human oversight\([Acharya et al\. 2025](https://arxiv.org/html/2609.06063#bib.bib1)\)\. Unlike traditional AI models, AI Agents111In this paper, we use the terms*AI agents*and*agentic AI systems*interchangeably to refer primarily to LLM\-based agents capable of autonomous planning and tool use\. While some of the challenges and solutions discussed may also apply to other types of autonomous AI systems, such as agents based on traditional reinforcement learning, our focus is on tool\-using LLM agents, which currently constitute the dominant paradigm of Agentic AI\.interact with external tools, maintain internal state, coordinate multiple actions, and adapt their behavior over extended execution horizons\. While these capabilities unlock significant opportunities across critical domains, they also introduce new challenges for transparency, accountability, and governance\([Zhu et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib2);[Shah et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib9)\), making surface\-level transparency no longer sufficient to support user trust or regulatory compliance\([Khalid et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib6)\)\. Recent studies further highlight novel failure modes, including error cascades, responsibility gaps, flawed execution monitoring, and multi\-step error propagation that obscures the origins of system behavior and outcomes\([Zhu et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib2);[Shah et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib9)\), highlighting the need for a rethinking of current XAI practices\.

Existing explainability approaches, in fact, were primarily designed for static, single\-step predictive models and are increasingly inadequate for explaining the behavior of autonomous agents operating in dynamic environments\([Chakrabarty 2025](https://arxiv.org/html/2609.06063#bib.bib7);[Raza et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib4);[Chaduvula et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib5)\)\. This challenge is further amplified by the interaction paradigm of Agentic AI, which introduces a complex triadic relationship among users, autonomous agents, and external digital resources, requiring explanation mechanisms that account for multi\-layered interactions across interconnected systems\([Ehsan et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib8)\)\. For these reasons, researchers have called for a shift from traditional feature\-level explanations toward system\-level interpretability capable of capturing agent trajectories, reasoning processes, tool interactions, and execution histories\([Zhu et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib2);[Raza et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib4);[Collaco et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib3)\)\.

Emerging work advocates for explainability frameworks that treat transparency as a continuous lifecycle property and capable of answering not only what decisions were made, but also why, how, when, and by whom they were influenced\([Raza et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib4)\)\. Similarly, other studies warn against the illusion of explainability that Chain\-of\-Thought disclosures may create, highlighting the need for human\-centered approaches that support meaningful understanding rather than merely exposing internal reasoning traces\([Ehsan et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib8)\)\. However, despite the growing recognition of these challenges, research at the intersection of XAI and Agentic AI remains relatively narrow in scope and is largely industry\-driven, with a primary focus on tasks such as failure detection and debugging\. As a result, existing approaches often fall short of addressing the broader interpretability and accountability debt incurred by these systems across diverse stakeholder groups, including regulators, deployers, and end users\. For these reasons, we introduce a post\-hoc, log\-grounded framework for generating faithful and auditable structured and natural\-language explanations of AI agents’ decisions\. Our approach leverages execution traces, which are often too lengthy and complex to be readily inspected or interpreted, particularly by non\-technical users, to reconstruct and expose the specific sequences of actions and environmental observations that shaped an agent’s behavioral trajectory and ultimately led to a given outcome\. Because the framework operates exclusively on the logged interaction trace, it can be seamlessly applied across different underlying agent architectures, tool ecosystems, and task domains, enabling broad applicability to real\-world AI systems\. Overall, our work makes the following contributions:

1. 1\.A framework for end\-user explainability of AI agents\.We introduce a framework specifically designed to make AI agent behavior transparent and understandable to end users\. To the best of our knowledge, existing approaches primarily target developer\- and system\-oriented objectives, such as debugging, monitoring, and failure analysis, rather than treating user\-facing explanations as a primary objective\. Our framework addresses this gap by defining a process for transforming agent execution traces into faithful and human\-readable explanations\.
2. 2\.Two complementary trace\-grounded XAI artifacts\.We introduce two complementary artifacts for explaining agent behavior: \(i\) a structured report that provides a systematic, step\-by\-step representation of the agent’s behavioral trajectory, and \(ii\) a concise natural\-language narrative that communicates the relevant behavior to end users while remaining grounded in the execution trace\. The structured report serves as an intermediate, auditable layer connecting the trace to the final explanation and enables precise identification of mismatches between the agent’s available knowledge and its observed behavior\.
3. 3\.A taxonomy of agent behavior grounded in knowledge alignment\.We develop a taxonomy that characterizes agent behavior according to the relationship between the agent’s available knowledge and information acquired through interaction with the environment\. The taxonomy captures aligned and mismatched knowledge states, providing a principled basis for identifying and interpreting different forms of agent behavior from execution traces\.
4. 4\.An empirical evaluation of explanation quality\.We evaluate the framework across two established agentic benchmarks, three agent architectures, and two baseline approaches to generating XAI artifacts\. We assess explanations along four dimensions \(faithfulness, exhaustiveness, linguistic quality, and overall satisfaction\) using both human evaluation and LLM\-as\-a\-judge\. Our results show that the proposed method produces structured reports that are substantially more faithful to the execution trace than those obtained by directly prompting an LLM to generate them\. Furthermore, for both human and LLM evaluators, narratives grounded in our reports are judged to be more faithful and exhaustive, and receive higher overall ratings, than narratives generated by prompting an LLM to explain the execution trace\.

## 2Related Work

While a growing body of work explores how agentic systems can themselves support explainability tasks\([He and Martens 2026](https://arxiv.org/html/2609.06063#bib.bib10);[Mehta 2026](https://arxiv.org/html/2609.06063#bib.bib13);[Vineis et al\. 2026](https://arxiv.org/html/2609.06063#bib.bib11)\), to the best of our knowledge no prior work has proposed a framework for making the decision process of Agentic AI systems transparent to end users\. Instead, existing work has primarily focused on trace\-based analysis for debugging, evaluation and failure diagnosis\. Nevertheless, this line of research provides valuable insights that can inform execution trace\-grounded explainability methods\. In this context,[Mazhar et al\. \(2026\)](https://arxiv.org/html/2609.06063#bib.bib12)show that outcome\-only evaluation systematically overlooks contamination\-induced failures that are only visible through trace inspection\. Similarly,[Deshpande et al\. \(2025\)](https://arxiv.org/html/2609.06063#bib.bib14), while introducing a formal taxonomy of agent failures and human\-annotated traces \(TRAIL benchmark\), show that state\-of\-the\-art LLMs remain unreliable at automatically identifying and diagnosing such failures\. Addressing procedural compliance in enterprise settings,[Sharma et al\. \(2026\)](https://arxiv.org/html/2609.06063#bib.bib16)propose a method \(AgentPex\) which employs an LLM\-as\-a\-judge to verify whether agent executions satisfy safety and formatting requirements at scale\. Likewise,[Mulian et al\. \(2026\)](https://arxiv.org/html/2609.06063#bib.bib17)introduce AgentFixer, which analyzes execution logs to detect parsing errors, schema violations, and reasoning inconsistencies before generating corrective actions\. Moving beyond LLM\-centric approaches,[Wang \(2026\)](https://arxiv.org/html/2609.06063#bib.bib18)reconstructing causal graphs from execution logs to localize failure root causes\. Although these methods leverage execution traces as we do, they are primarily designed as developer\-facing analysis tools rather than explanation systems\. Their objective is to identify why an agent failed, not to communicate how it reached a decision in a faithful and user\-understandable manner\. By contrast, our framework treats execution traces as the basis for generating trace\-grounded explanations that support end\-user understanding while simultaneously providing a structured basis for inspecting and auditing agent behavior\. This perspective is further supported by[Hu et al\. \(2026\)](https://arxiv.org/html/2609.06063#bib.bib15), who argue that explicit execution provenance, rather than increasingly sophisticated benchmarks, is a key missing infrastructure for responsibility attribution in Agentic AI\. However, their approach is validated only on a proof\-of\-concept system and does not provide a general framework for generating user\-facing explanations, a gap that our work aims to address\.

RawExecutionTraceProcesses1\. Normalize→τ\\rightarrow\\tau2\. Decompose→ℛ\\rightarrow\\mathcal\{R\}3\. Extract→β,ϵint,ϵext\\rightarrow\\beta,\\epsilon^\{\\text\{int\}\},\\epsilon^\{\\text\{ext\}\}4\. Validate5\. Reconcile→δ,γ\\rightarrow\\delta,\\gammaStructured report\(first XAI artifact\)X⁡\(τ\)=\{𝒢,n,\(at,Ot,Σt,δt,Γt\)\}t=0TX\(\\tau\)=\\\{\\mathcal\{G\},n,\(a\_\{t\},O\_\{t\},\\Sigma\_\{t\},\\delta\_\{t\},\\Gamma\_\{t\}\)\\\}\_\{t=0\}^\{T\}Narratorψ\\psiNatural\-language explanation\(second XAI artifact\)E1,…,ET,E¯E\_\{1\},\\dots,E\_\{T\},\\bar\{E\}

Figure 1:Framework overview\. The raw AI agent execution trace is processed through a sequential operational pipeline that constructs the structured report𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)\. Trace normalization maps heterogeneous execution logs to a canonical step sequence, while requirement decomposition constructs the requirement registryℛ\\mathcal\{R\}, to decompose user’s goal\. At each step, an LLM extracts candidate behavioral commitments and evidence with explicit pointers to the original trace; deterministic validation filters unsupported or ill\-typed updates and enforces the framework’s grounding constraints\. Deterministic reconciliation then computes the per\-step state transitionsδt\\delta\_\{t\}and requirement\-level grounding reportsΓt\\Gamma\_\{t\}, yielding the structured tuples\(Σt,δt,Γt\)\(\\Sigma\_\{t\},\\delta\_\{t\},\\Gamma\_\{t\}\)\. Finally, the constrained narratorψ\\psirenders𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)into per\-step explanationsE1,…,ETE\_\{1\},\\ldots,E\_\{T\}and a concise overall summaryE¯\\bar\{E\}\.
## 3Trace\-Grounded XAI Methodology

In this section, we formalize the core components of our methodology and their relationships \(Section[3\.1](https://arxiv.org/html/2609.06063#S3.SS1)\); then we describe their concrete operationalization in a sequential pipeline \(Section[3\.2](https://arxiv.org/html/2609.06063#S3.SS2)\)\.

### 3\.1Main Components

As visually summarize in Figure[1](https://arxiv.org/html/2609.06063#S2.F1), the proposed framework transforms a heterogeneous agent execution trace into a structured, trace\-grounded representation of the agent’s behavioral trajectory, which we refer to as the*structured report*\(our first XAI artifact\)\. The report is organized around the goal and its requirements, the actions and observations recorded during execution, and requirement\-level states that capture both the agent’s behavioral commitments and the evidence available to support them\. These components distinguish what the agent appears to commit to from what can be established from the execution trace, while preserving the temporal structure needed to reconstruct how requirements evolve throughout the trajectory\. The resulting representation provides the basis for both*natural\-language explanations*\(our second XAI artifact\) and grounding diagnostics\.

#### Goal and Requirements\.

Consider a tool\-using LLM agent tasked with achieving a goalGG, specified by a user’s query\. We assume thatGGcan be decomposed into a set of atomic, independently verifiable requirementsℛ=r1,r2,…,rN,\\mathcal\{R\}=\{r\_\{1\},r\_\{2\},\\ldots,r\_\{N\}\},where each requirementrir\_\{i\}represents a minimal criterion that must be satisfied for the goal to be considered complete\. The agent pursuesGGthrough iterative interactions with external tools and the environment\.

#### Objective\.

Our objective is to generate a*post\-hoc*,*trace\-faithful*explanation that makes the agent’s reconstructed behavioral trajectory transparent and inspectable, including to non\-technical users\. Specifically, we produce two complementary XAI artifacts: \(i\) a structured report that organizes the trajectory into a sequence of epistemically relevant actions and observations, enabling the characterization of transitions in the agent’s knowledge–action state and step\-by\-step inspection of its progress toward the goal; and \(ii\) a concise natural\-language explanation that summarizes this trajectory in human\-readable form\.

Crucially, the framework does not seek to assess the objective correctness of the agent’s final answer, but rather to make transparent the process that led to it\. Failure detection is likewise not its primary objective\. Although the resulting artifacts can help localize potential errors and characterize inconsistencies exhibited by the agent, these signals concern mismatches between the agent’s behavior and the evidence available in the trace\. They do not, by themselves, establish failures in the underlying tools or the incorrectness of the agent’s knowledge\.

#### Event Trace\.

The fundamental input to our framework is an agent\-specific execution trace, generated through the agent’s interaction with its environment\. This raw trace records agent actions, tool invocations and outputs, together with heterogeneous execution metadata, including timestamps, call identifiers, intermediate states, and framework\-specific information\. While the structure and granularity of such traces vary across agent implementations, we assume that the recorded information is sufficient to reconstruct the underlying behavioral trajectory\. We denote the resulting epistemically meaningful*event trace*byτ=\(e1,e2,…,eT\),\\tau=\(e\_\{1\},e\_\{2\},\\ldots,e\_\{T\}\),where each eventete\_\{t\}captures an action, observation, or other interaction that may alter the information available to the agent at that point in the trajectory\. By convention,TTdenotes the terminal step at which the agent produces its final response to the user; all preceding stepst<Tt<Tcorrespond to intermediate actions and observations along the trajectory\. This representation abstracts away implementation\-specific logging details while preserving the information required to characterize how the agent progresses from the initial query toward its goal\.

#### Knowledge–Action State Transitions\.

The event traceτ\\tauprovides the basis for constructing a structured representation of the agent’s behavioral trajectory\. We denote byata\_\{t\}the identifier of the agent action at stepttand byot,ko\_\{t,k\}the identifier of thekk\-th observation associated with that step\. At each steptt, we reconstruct a requirement\-indexed knowledge–action state

Σt=\(ℛt,\{βt​\(r\),ϵt​\(r\)\}r∈ℛt\),\\Sigma\_\{t\}=\\left\(\\mathcal\{R\}\_\{t\},\\left\\\{\\beta\_\{t\}\(r\),\\epsilon\_\{t\}\(r\)\\right\\\}\_\{r\\in\\mathcal\{R\}\_\{t\}\}\\right\),whereℛt⊆ℛ\\mathcal\{R\}\_\{t\}\\subseteq\\mathcal\{R\}is the set of requirements introduced up to steptt,βt​\(r\)\\beta\_\{t\}\(r\)records the agent’s behavioral commitment with respect to requirementrr, andϵt​\(r\)\\epsilon\_\{t\}\(r\)records the evidence available for that requirement in the execution trace\. For eachr∈ℛtr\\in\\mathcal\{R\}\_\{t\},βt​\(r\)\\beta\_\{t\}\(r\)ansϵt​\(r\)\\epsilon\_\{t\}\(r\)take values in\{open,satisfied\}\\\{\\texttt\{open\},\\texttt\{satisfied\}\\\}\.

The behavioral componentβt\\beta\_\{t\}is a trace\-grounded reconstruction of the commitment expressed by the agent’s subsequent behavior\. In particular, its value is inferred from the next relevant agent action:βt​\(r\)=fβ​\(at\+1,r\),\\beta\_\{t\}\(r\)=f\_\{\\beta\}\(a\_\{t\+1\},r\),wherefβf\_\{\\beta\}determines whether the action selected after stepttindicates that the agent treats requirementrras resolved\.

The evidence component is instead decomposed into two provenance\-specific channels:

ϵt​\(r\)=\(ϵtint​\(r\),ϵtext​\(r\)\),\\epsilon\_\{t\}\(r\)=\\bigl\(\\epsilon\_\{t\}^\{\\mathrm\{int\}\}\(r\),\\epsilon\_\{t\}^\{\\mathrm\{ext\}\}\(r\)\\bigr\),whose statuses may independently beopenorsatisfiedand may therefore disagree for the same requirementrrat a given steptt\. The internal component is supported by observations corresponding to the agent’s own surfaced information, such as reflections or tool inputs, whereas the external component is supported by observations returned by the environment, such as tool outputs and execution feedback\.

On both behavioral and evidence axes, thesatisfiedstatus requires an explicit supporting reference in the trace\. Formally,∀r∈ℛt∗\\forall\\,r\\in\\mathcal\{R\}\_\{t\}^\{\*\}and∀x∈\{β\}∪\{ϵc:c∈\{int,ext\}\}\\forall x\\in\\\{\\beta\\\}\\cup\\\{\\epsilon^\{c\}:c\\in\\\{\\mathrm\{int\},\\mathrm\{ext\}\\\}\\\},

xt​\(r\)=satisfied⟹suppx⁡\(r\)≠∅\.x\_\{t\}\(r\)=\\texttt\{satisfied\}\\Longrightarrow\\operatorname\{supp\}\_\{x\}\(r\)\\neq\\varnothing\.
withs​u​p​pβ​\(r\)supp\_\{\\beta\}\(r\)pointing to the actionat\+1a\_\{t\+1\}\(or the goalGG\) whose content expresses the commitment, ands​u​p​pϵc​\(r\)supp\_\{\\epsilon^\{c\}\}\(r\)pointing to at least one observationot,ko\_\{t,k\}of the corresponding provenance whose content entailsrr\. When no admissible reference exists, the status remainsopen\. This invariant makes every positive assertion inΣt\\Sigma\_\{t\}pointer\-cited and directly auditable against the raw trace\.

As the agent proceeds through the observed action–observation sequence, itsΣt\\Sigma\_\{t\}states evolve to reflect changes in requirements, behavioral commitments, and available evidence\. We summarize each consecutive state change with a transition operator

δt=Δ⁡\(Σt,Σt\+1\),\\delta\_\{t\}\\;=\\;\\Delta\(\\Sigma\_\{t\},\\Sigma\_\{t\+1\}\),which identifies changes in requirement status acrossβ\\betaandϵ\\epsilon, including requirements becoming satisfied or reopening, as well as the introduction of new requirements\.

#### Behaviour\-Evidence Discrepancies\.

Given the status ofβ\\betaandϵ\\epsilonfor each requirementr∈ℛtr\\in\\mathcal\{R\}\_\{t\}, we deterministically assign a grounding label based on the joint configuration of the agent’s behavioral commitment and the available internal and external evidence:

γt​\(r\)=f⁡\(βt​\(r\),ϵtint​\(r\),ϵtext​\(r\)\)\.\\gamma\_\{t\}\(r\)\\;=\\;f\\\!\\left\(\\beta\_\{t\}\(r\),\\,\\epsilon\_\{t\}^\{\\mathrm\{int\}\}\(r\),\\,\\epsilon\_\{t\}^\{\\mathrm\{ext\}\}\(r\)\\right\)\.
The label characterizes whether the agent’s behavioral commitment is consistent with the evidence available in the trace and, when it is not, identifies the nature of the discrepancy\. The complete232^\{3\}configuration space is reported in Table[1](https://arxiv.org/html/2609.06063#S3.T1)\. We organize these configurations into two groups\.*Concordant*configurations are those in which the behavioral commitment is consistent with the available evidence\. This group includes both cases in which a requirement is supported by internal and/or external evidence and the*genuine gap*, in which the requirement remains unresolved and no supporting evidence is available\.*Mismatches*are configurations in which the behavioral commitment diverges from the available evidence\. The four mismatch categories constitute the core diagnostic output ofγ\\gamma\. Aggregatingγt\\gamma\_\{t\}over all requirements yields the per\-step grounding report

Γt=\{\(r,γt\(r\)\):r∈ℛt,γt\(r\)∈ℒ\},\\Gamma\_\{t\}\\;=\\;\\bigl\\\{\\,\\bigl\(r,\\gamma\_\{t\}\(r\)\\bigr\)\\,:\\,r\\in\\mathcal\{R\}\_\{t\},\\;\\gamma\_\{t\}\(r\)\\in\\mathcal\{L\}\\,\\bigr\\\},whereℒ\\mathcal\{L\}is the label set of Table[1](https://arxiv.org/html/2609.06063#S3.T1)\. By constructionΓt=∅\\Gamma\_\{t\}=\\varnothingwhen every requirement is warranted, soΓt\\Gamma\_\{t\}acts as the step\-level record of the discrepancies between behavioral commitment and trace evidence\.

β\\betaϵi​n​t\\epsilon^\{int\}ϵe​x​t\\epsilon^\{ext\}LabelInterpretation*Concordant configurations \(bahaviour matches available evidence\)*111Fully GroundedResolved with both internal and external evidence\.110Self GroundedResolved from internal evidence only\.101Externally GroundedResolved from external evidence only\.000Genuine GapUnresolved and no supporting evidence available\.*Discordant configurations \(behaviour and evidence disagree\)*100UngroundedResolved without trace\-grounded evidence\.010Missed Internal EvidenceInternal evidence is available but not reflected in behaviour\.001Missed External EvidenceExternal evidence is available but not reflected in behaviour\.011Missed RecognitionBoth evidence sources are available but not reflected in behaviour\.Table 1:Diagnostic configurations derived from behaviour–evidence reconciliation\. Here,β\\beta,ϵi​n​t\\epsilon^\{int\}, andϵe​x​t\\epsilon^\{ext\}denote behavioral committment, internal evidence, and external evidence, respectively, measured with respect to a specific requirementrrat steptt; each equals11when the corresponding state issatisfied\.
#### Structured report\.

Aggregating across the trajectory, the framework’s first XAI artifact is the*structured report*

𝒳⁡\(τ\)=\(G,\{\(at,𝒪t,Σt,δt,Γt\)\}t=0T\),\\mathcal\{X\}\(\\tau\)\\;=\\;\\Bigl\(\\,G,\\;\\bigl\\\{\\bigl\(a\_\{t\},\\,\\mathcal\{O\}\_\{t\},\\,\\Sigma\_\{t\},\\,\\delta\_\{t\},\\,\\Gamma\_\{t\}\\bigr\)\\bigr\\\}\_\{t=0\}^\{T\}\\,\\Bigr\),where𝒪t=\{ot,k\}k=1Kt\\mathcal\{O\}\_\{t\}=\\\{o\_\{t,k\}\\\}\_\{k=1\}^\{K\_\{t\}\}denotes the set of observations associated with steptt\.

The structured report records, for every step, the action taken, the observations received, the reconstructed knowledge–action state, the transition summary, and the grounding report \(withδ0\\delta\_\{0\}omitted andδT\\delta\_\{T\}defined against the terminal state\)\.𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)is directly inspectable and provides the audited input to the natural\-language explanation described next\.

#### Final Narrative\.

The second XAI artifact is then produced by a constrained narrator

ψ:𝒳⁡\(τ\)⟶\(E1,E2,…,ET,E¯\),\\psi:\\;\\mathcal\{X\}\(\\tau\)\\;\\longrightarrow\\;\\bigl\(E\_\{1\},\\,E\_\{2\},\\,\\ldots,\\,E\_\{T\},\\,\\bar\{E\}\\bigr\),where each per\-step explanation

Et=ψstep​\(at,𝒪t,Σt,δt,Γt\)E\_\{t\}=\\psi\_\{\\mathrm\{step\}\}\(a\_\{t\},\\mathcal\{O\}\_\{t\},\\Sigma\_\{t\},\\delta\_\{t\},\\Gamma\_\{t\}\)verbalizes the epistemic change at stepttand any divergence recorded inΓt\\Gamma\_\{t\}, andE¯\\bar\{E\}compresses the trajectory into an episode\-level summary\.

### 3\.2Operational Pipeline

Below, we describe the concrete instantiation of the framework components introduced above and their organization into a sequential operational pipeline\.

#### Trace Normalization\.

The raw execution trace is mapped to the canonical step sequence required by the framework\. Each step preserves the action, its observations, their semantic types, and the terminal condition\. The representation uses a fixed observation vocabulary—user query, tool input, tool output, reflection, error, and final response—and separates tool inputs from tool outputs so that agent\-emitted information remains distinguishable from environment\-returned information\. The resulting canonical sequence is the common interface to all subsequent stages, decoupling the analysis from the underlying agent architecture and enabling the same pipeline to be applied across heterogeneous paradigms\.

#### Requirement Decomposition\.

The normalized goal is decomposed into the requirement registryℛ\\mathcal\{R\}using a summarize\-then\-decompose strategy inspired from[Cho et al\. \(2026\)](https://arxiv.org/html/2609.06063#bib.bib19)\. The decomposition targets atomicity at the level required for independent status assessment: requirements concerning different entities are kept distinct whenever their satisfaction can be established independently\. Each requirement is assigned a stable identifier and classified as either a*content requirement*\(information that must appear in the answer\) or a*constraint requirement*\(formatting or structural condition\)\. Depending on the application, requirements may be generated upfront, incrementally, or through a hybrid strategy that admits newly discovered subgoals during execution\.

#### Local Behaviour–Evidence Extraction\.

At each step, an LLM produces local updates to the requirement\-level state\. Given the information available at steptttogether with the subsequent actionat\+1a\_\{t\+1\}, the extractor proposes behavioral commitments and evidence items, each accompanied by pointers to the specific trace elements that support them\. We recall that behavioural commitment is inferred fromat\+1a\_\{t\+1\}; evidence proposals are assigned to the internal or external channel according to the provenance of their cited observations\.

#### Validation and Grounding\.

Candidate updates are subjected to deterministic validation\. Behavioral supports must correspond to the action designated as the behavioral basis, and evidence supports must reference observations present in the trace and admissible for their assigned channel; ill\-typed or unsupported references are discarded\. When entailment verification is enabled, an evidence channel is markedsatisfiedonly if the cited content additionally entails the corresponding requirement\. Validated updates are then incorporated into the state subject to the framework’s consistency and monotonicity constraints\.

#### Deterministic Reconciliation\.

The validated state sequence is processed deterministically to obtain the per\-step summaries\. Consecutive states yield the transitionδt\\delta\_\{t\}, and the joint status of behavioral commitment and the two evidence channels determines the labelγt​\(r\)\\gamma\_\{t\}\(r\)for every requirement, which aggregate into the step\-level reportΓt\\Gamma\_\{t\}\. The resulting\(Σt,δt,Γt\)\(\\Sigma\_\{t\},\\delta\_\{t\},\\Gamma\_\{t\}\)tuples complete the structured report𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)\.

#### Narrative Generation\.

The narratorψ\\psiis instantiated as a constrained LLM call over𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)\. Because every belief, evidence item, and diagnostic has already been grounded in the trace, narrative generation reduces to verbalizing an audited execution history rather than reconstructing reasoning from raw logs\. The narrator is constrained to reference only elements of its input—requirement identifiers, action identifiersata\_\{t\}, observation identifiersot,ko\_\{t,k\}, and their statuses—so that every generated statement remains trace\-grounded and no content is introduced beyond𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)\. The resulting explanation reports what the agent attempted, what information it acquired, which requirements became satisfied, and where potential failures or inconsistencies occurred\.

## 4Experiments

We evaluate the framework through a controlled study to examine the reliability and trace\-grounded fidelity of the structured report and the quality of the natural\-language explanations it supports\. The following sections describe the experimental setup \(Section[4\.1](https://arxiv.org/html/2609.06063#S4.SS1)\) and evaluation protocols \(Section[4\.2](https://arxiv.org/html/2609.06063#S4.SS2)\)\.

Table 2:Evaluation corpus and pipeline reliability\. For each of the five \(benchmark×\\timesagent\) subsets we report the number of traces processed and the number of*failed*generations per pipeline configuration \(percentages relative to the row total\)\. TheStructured reportblock covers the intermediate representation \(𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)\), produced either by our deterministic pipeline \(Our\) or by a single\-shot LLM prompt with the same schema \(LLM\)\. TheNarrativeblock covers the final natural\-language explanation \(E¯\\bar\{E\}\), produced by: \(i\)Our–Our, our narrator applied to our structured report \(the full pipeline described in Section[3\.2](https://arxiv.org/html/2609.06063#S3.SS2)\); \(ii\)LLM\-Our, our narrator applied to an directly LLM\-generated version of the structured report; and \(iii\)LLM\-only, an LLM prompted to summarise the raw trace directly in natural language\.### 4\.1Experimental Setup

We evaluate the proposed framework on execution traces from two question\-answering \(QA\) benchmarks and an external trace corpus\. The evaluation is designed along two complementary dimensions\. First, we assess the contribution of the proposed structured analysis by comparing our XAI artifacts with outputs generated directly by an LLM under matched input and model conditions\. Second, we evaluate the quality of the resulting artifacts using both human assessment and LLM\-as\-a\-judge evaluation\. This design allows us to disentangle the contribution of the structured, trace\-grounded analysis from that of the underlying language model used for generation\.

#### Trace Corpus\.

We focus our empirical evaluation on tool\-empowered QA tasks because they provide a controlled setting for assessing explanations of agent behavior\. QA benchmarks typically specify well\-defined user goals and information\-seeking objectives, making requirement satisfaction more amenable to systematic assessment and reducing reliance on subjective judgments of task success\. Moreover, tool\-mediated QA produces explicit action–observation sequences, enabling us to assess whether explanations faithfully account for observable agent behavior rather than merely whether the final answer is correct\. This makes QA well suited to evaluating transparency and auditability while enabling larger\-scale evaluation at lower annotation cost than tasks requiring extensive expert assessment\. The choice of QA is specific to our empirical evaluation, not a limitation of the framework, which can be applied to other agentic tasks given suitable execution traces\.

Among existing QA resources, TRAIL\([Deshpande et al\. 2025](https://arxiv.org/html/2609.06063#bib.bib14)\)is, to our knowledge, the only dataset providing publicly available execution traces for a QA benchmark\. We therefore use TRAIL as our external trace source and build our evaluation around its underlying GAIA\([Mialon et al\. 2024](https://arxiv.org/html/2609.06063#bib.bib20)\)question set\. The original TRAIL agent and its execution environment are not available to us, and the released trajectories do not expose explicit external tool calls, primarily capturing the agent’s internal reasoning\. We therefore complement TRAIL with independently generated trajectories exhibiting richer and more heterogeneous tool use\. Specifically, we implement two additional agent configurations and execute them on the same GAIA questions, enabling controlled comparison across different agent architectures and execution structures while preserving a common task set\.

The first agent is a*ReAct*agent\([Yao et al\. 2022](https://arxiv.org/html/2609.06063#bib.bib22)\), which interleaves reasoning and tool use within a single execution loop\. The second is a*plan\-and\-act*agent that follows an iterative plan–act–reflect loop with an explicit step for routing between actions\. Both agents usegemini\-3\.1\-flash\-lite222https://deepmind\.google/models/model\-cards/gemini\-3\-1\-flash\-lite/as the underlying LLM and share a heterogeneous tool set comprising Wikipedia, Tavily web search, calculator, Python code execution, PDF reader, and text editor\. This tool set covers information retrieval, web browsing, numerical computation, programmatic analysis, and document processing, yielding diverse tool interactions representative of practical agentic QA systems\.

For direct comparison with the external trace source, we use the116116GAIA questions for which TRAIL provides executions and run both independently implemented agents on the same questions\. We additionally evaluate the two agents on100100HotpotQA\([Yang et al\. 2018](https://arxiv.org/html/2609.06063#bib.bib21)\)questions sampled from the distractor validation split\. This complementary multi\-hop QA setting allows us to assess whether the framework generalizes beyond the GAIA/TRAIL task distribution\.

The resulting evaluation corpus comprises five benchmark–agent subsets: GAIA–plan\-and\-act \(116116traces\), GAIA–ReAct \(116116\), GAIA–TRAIL \(116116\), HotpotQA–plan\-and\-act \(100100\), and HotpotQA–ReAct \(100100\), for a total of548548execution traces\.

#### XAI Artifact Generation\.

Each trace is processed independently using the pipeline described in Section[3\.2](https://arxiv.org/html/2609.06063#S3.SS2)\. The LLM\-based extraction and validation stage usesgemini\-3\.1\-flash\-lite, while state updates, reconciliation, and report construction follow the deterministic procedures defined by the framework\. Requirements are generated in upfront mode, such that the complete requirement registry is constructed from the user goal att=0t=0\. We evaluate both artifacts produced by the framework: the structured report𝒳⁡\(τ\)\\mathcal\{X\}\(\\tau\)and its natural\-language rendering\. Across the548548execution traces, we obtain1,0961\{,\}096structured reports from the two structured\-report configurations and1,6441\{,\}644natural\-language explanations from the three narrative configurations\.

#### Baselines and Ablations\.

We consider two complementary comparisons to isolate the contributions of the structured representation and its trace\-grounded construction\. To distinguish the source of each artifact, we use paired notation\(S,N\)\(S,N\), whereSSdenotes the source of the structured report andNNthe source of the narrative\.

TheOur–Ourconfiguration denotes the complete proposed pipeline: the structured report is constructed by our trace\-grounded pipeline and the resulting natural language explanation is rendered by our constrained narratorψ\\psi\.

TheLLM–Ourconfiguration isolates the contribution of the structured\-report construction procedure\. The LLM receives the same canonical trace representation, requirement vocabulary, and target schema used by our framework, but is prompted to generate the complete structured report autonomously\. This preserves the conceptual representation introduced by our method while bypassing its multi\-stage extraction, validation, state\-update, and deterministic reconciliation procedures\. The resulting report is then passed unchanged to the same constrained narratorψ\\psi\. This comparison tests whether the benefits of our structured representation can be obtained simply by prompting an LLM to populate the proposed schema, or instead depend on the trace\-grounded construction process\.

Finally, we use theLLM\-onlynotation to denote direct narrative generation without an intermediate structured representation, which correspond to use the LLM directly as the explainer\. In this case the original trace is provided to a single LLM call, which is prompted to generate the natural\-language explanation directly\. The prompt is designed based on a review of existing approaches to LLM\-based generation and evaluation of XAI narratives\([Silvestri et al\. 2025](https://arxiv.org/html/2609.06063#bib.bib23)\), with the goal of providing a competitive baseline informed by the state of the art\. This configuration therefore provides an end\-to\-end comparison in which the model must derive and verbalize the explanation directly from the execution trace\.

All configurations usegemini\-3\.1\-flash\-litefor LLM\-based generation, including the construction of intermediate structured reports and final narratives\. Where comparisons are available, configurations operate on the same input trace and use the same underlying model, varying only the representation and generation procedure\. This controls for differences in model capacity and focuses the comparison on the contribution of the proposed structured and trace\-grounded pipeline\.

#### Generation Failures\.

Original execution traces can be long and structurally complex, making single\-shot generation particularly demanding when the model must process the complete trajectory and produce the XAI artifacts\. We therefore allow up to2020generation attempts for each requested artifact\. A generation is considered*failed*if the requested artifact has not been successfully produced after these2020attempts333ForLLM\-onlyconfiguration, narrative generation may fail for excessively long inputs, as the trace and system prompt alone can exceed 250k tokens, thereby reaching the context\-window limit of the free Gemini plan used for generating XAI outputs across all configurations\.\. Failed generations are excluded from downstream artifact\-quality evaluation but are retained in the reliability analysis reported in Table[2](https://arxiv.org/html/2609.06063#S4.T2)\. As shown in the table, failures occur primarily for the single\-shot LLM configurations and are more frequent for the longer and more complex traces \(TRAIL has shorter execution trace than the one obtained from our agentic configurations\)\. Our proposed pipeline records no failures, owing to its design choices described in Section[3](https://arxiv.org/html/2609.06063#S3), which make the framework robust to long and structurally complex execution traces\.

### 4\.2Evaluation

We evaluate the framework at two levels: the fidelity of the intermediate structured representation and the quality of the generated narratives\. The evaluation uses two complementary protocols\. First, on a stratified subset of the execution corpus, human annotators independently reconstruct the structured representation directly from the original execution traces\. These human\-annotated representations provide a trace\-grounded reference for evaluating the structured reports produced by our framework and by the LLM baseline\. Second, we evaluate the generated narratives using both human assessment and an LLM\-as\-a\-judge protocol\([Zheng et al\. 2023](https://arxiv.org/html/2609.06063#bib.bib24)\),using*Muse\-Spark\-1\.2\-Contributor*as evaluation model\.

#### Human evaluation subset and reference annotation\.

We conduct human evaluation on a stratified subset of the execution corpus, comprising3232traces from1212queries across three agent systems \(ReAct, plan\-and\-act, and TRAIL agent\)\. The ReAct and plan\-and\-act agents contribute1212traces each, covering99GAIA and33HotpotQA questions, while TRAIL contributes88GAIA traces444One sampled GAIA question has no corresponding TRAIL execution and is therefore excluded from the TRAIL subset\.\. GAIA questions are selected through stratified random sampling over difficulty and number of tool interactions, following the characterization provided by the original GAIA benchmark paper\. For HotpotQA, whose original difficulty characterization is based on query length and is not meaningful to our setting, we assign each query a difficulty level and tool\-use stratum using the same criteria adopted for GAIA\.

For each selected trace, a human annotator independently constructs a structured representation directly from the original execution trace, following the representation schema defined in Section[3](https://arxiv.org/html/2609.06063#S3)\. This human\-annotated representation serves as the reference for evaluating the two structured\-report variants: the report produced by our full pipeline and the report obtained by prompting the LLM to instantiate the same schema\. Because the reference is constructed directly from the trace and independently of both generated reports, it provides an independent basis for assessing the fidelity of the intermediate representation\.

Considering the two structured\-output methods \(OurandLLM\) and the three narrative\-generation settings \(Our\-Our,LLM\-Our, andLLM\-only\), and excluding cases in which artifact generation failed \(see Table[2](https://arxiv.org/html/2609.06063#S4.T2)\), the human evaluation covers6363structured reports and8989narratives\.

Table 3:Narrative evaluation criteria and evaluation configurations\.
#### Structured Report Evaluation

On the human\-annotated subset, we assess the faithfulness of the two structured\-report variants\. Because this comparison requires reliable assessment of the correspondence between the structured representation and the underlying execution, and LLM\-as\-a\-judge methods cannot be assumed to provide faithful judgments, particularly for lengthy and complex execution traces, we restrict this evaluation to the human\-annotated subset\. This provides a direct assessment of whether the intermediate representation accurately captures the behavior exhibited in the trace\.

We assess the fidelity of each generated structured report to the human\-annotated reference obtain by the original execution trace using an ad hoc metric we called the*Normalized Faithfulness of the Intermediate representation*\(NFI\)\. For a generated report, letRtG​TR\_\{t\}^\{GT\}denote the set of ground\-truth requirements associated with stepttandRtg​e​nR\_\{t\}^\{gen\}the corresponding requirements in the predicted report\. We define

NFI=∑tmatched⁡\(Rtg​e​n,RtG​T\)∑t\|RtG​T\|,\\mathrm\{NFI\}=\\frac\{\\sum\_\{t\}\\mathrm\{matched\}\\\!\\left\(R\_\{t\}^\{gen\},R\_\{t\}^\{GT\}\\right\)\}\{\\sum\_\{t\}\|R\_\{t\}^\{GT\}\|\},\(1\)
wherematched⁡\(⋅,⋅\)\\mathrm\{matched\}\(\\cdot,\\cdot\)counts the ground\-truth requirements correctly represented by the generated report\. The denominator is therefore the total number of ground\-truth requirements across all steps, yielding a score in\[0,1\]\[0,1\]\. NFI directly measures how much of the trace\-grounded behavior captured by the human reference is recovered by each structured representation\. To adopt a maximally strict criterion, we count as an error any discrepancy at a given step, including the omission of a ground\-truth requirement, a missedδt\\delta\_\{t\}orγt\\gamma\_\{t\}, or an incorrect classification ofβt\\beta\_\{t\}orϵt\\epsilon\_\{t\}, including an incorrect supporting evidence assignment\.

Table 4:NFI: mean and standard deviation by method and agent, overall and by number\-of\-steps group\.
#### Narrative Evaluation\.

Following consolidated practices identified in the narrative XAI literature\([Silvestri et al\. 2025](https://arxiv.org/html/2609.06063#bib.bib23)\), we evaluate natural language explanations along four dimensions, summarized in Table[3](https://arxiv.org/html/2609.06063#S4.T3)\.

*Faithfulness*is assessed at two main levels: narrative\-to\-report faithfulness \(M1\-a\) and trace\-level faithfulness \(M1\-b\)\. The former measures whether the narrative accurately reflects the behavior represented in the structured report from which it was generated, without introducing unsupported claims, omitting relevant information, or altering the meaning of the represented behavior\. The narrative\-to\-source\-report assessment applies to theOur\-OurandLLM\-Ournarratives, for which the structured report is available\. Trace\-level faithfulness measures whether the narrative faithfully reflects the behavior observed in the original execution trace\. This assessment requires comparison against the human\-annotated ground truth and is therefore performed only through human evaluation on the human\-evaluated subset, providing a direct assessment of trace\-level faithfulness\. As a further specification of narrative\-to\-report faithfulness, in the LLM\-as\-a\-judge evaluation we also assess each narrative against the structured report produced by our framework \(M1\-a\_our\)\. This configuration is used for cross\-validation: we compare the resulting LLM\-judge scores with the corresponding human assessments of the narrative’s trace\-level faithfulness to the original execution trace\. Agreement between the two provides evidence that the structured representation produced by our framework preserves the behaviour relevant to narrative faithfulness and can therefore serve as a suitable proxy for trace\-level evaluation\. This comparison also provides evidence that LLM\-as\-a\-judge evaluation can capture, at least partially, trace\-level faithfulness when the judge has access to the structured representation rather than the original execution trace\.

*Exhaustiveness*\(M2\) measures whether the narrative covers all the essential information required to explain the agent’s behavior\.*Linguistic quality*\(M3\) assesses clarity, logical organization, conciseness, and the absence of unnecessary or unsupported verbosity\.*Overall quality*\(M4\) captures the evaluator’s holistic assessment of the narrative as an explanation of the agent’s execution\.

To ensure consistent application of the evaluation criteria, we operationalize each metric through an explicit evaluation rubric, which specifies the criteria and corresponding rating anchors for both human and LLM\-judge assessment\. Faithfulness is evaluated on a three\-point ordinal scale \(“Totally agree,” “Partially agree,” “Disagree”\), whereas M2–M4 are evaluated on a four\-point Likert scale \(“Strongly agree,” “Agree,” “Disagree,” “Strongly disagree”\)\.

#### Agreement analysis\.

Before using LLM\-as\-a\-judge for scalable evaluation, we assess its alignment with human judgments, on the common scored subset\. For each criterion rated by both evaluators, we report raw percentage agreement after collapsing the ordinal scale into positive and negative judgments, together with quadratic\-weighted Cohen’sκ\\kappa\(κw\\kappa\_\{w\}\), which preserves the ordinal structure of the ratings and penalizes larger disagreements more strongly than near\-misses\.

Both measures are interpreted in conjunction with the marginal distribution of the ratings\. In particular, when narratives from stronger configurations receive positive judgments almost uniformly, the negative class becomes small and chance\-corrected agreement can be attenuated: high raw agreement may therefore coexist with a lowκw\\kappa\_\{w\}when little disagreement remains for chance correction to capture\. We therefore additionally report, for each criterion, the proportion of positive ratings assigned by each evaluator and the judge’s*negative recall*, defined as the proportion of cases rated negatively by humans that are also rated negatively by the judge\. Negative recall is particularly relevant for scalable evaluation, as it measures whether the judge identifies the failures that human evaluators would flag, independently of a shared tendency to assign positive ratings\.

Statistics are pooled across the three narrative configuration\. They are computed over the paired ratings available for each criterion; consequently, the effective sample size varies across criteria because of missing or inapplicable annotations\.

Table 5:Agreement between the human annotator and the LLM judge, pooled over conditions \(bold\) and split by condition\. H/J = percentage of ratings on the agree side for the human and judge, respectively; Agr\. = raw agreement on the agree/disagree collapse;κw\\kappa\_\{w\}= quadratic\-weighted Cohen’sκ\\kappa; Neg\. rec\. = share of human\-negative ratings also rated negative by the judge, with counts in parentheses\.†marks rows with fewer than three human\-negative ratings, for whichκw\\kappa\_\{w\}is not informative; raw agreement and negative recall should be preferred\. M1\-a\_our vs M1\-b is a cross\-level comparison: the judge scores the narrative against our structured report, whereas the human scores it against the original trace\.Table 6:Hypothesis tests on the LLM\-as\-judge corpus and comparison with human\-evaluation effects\. Inferential statistics are reported for the LLM\-as\-judge corpus, while human\-evaluation effects are reported for descriptive comparison only\.Table 7:H3 direction consistency for the LLM\-as\-judge corpus\. Cell = mean paired difference within that agent architecture\.
#### Hypotheses and analysis protocol\.

We formulate four directional hypotheses prior to analysis, corresponding to the principal claims tested in the evaluation:

- •H0 – Structured\-representation fidelity\.Our structured report is more faithful and complete as a step\-by\-step representation of the execution trace than the freely LLM\-generated report, as measured by the normalized faithfulness index \(NFI\) against human\-annotated ground truth \(M0\)\. H0 provides a further basis for using our structured report as the reference for evaluating narrative faithfulness in M1\-a\_our\.
- •H1 – End\-to\-end superiority\.Narratives produced by the full pipeline,Our\-Our, receive higher ratings than narratives generated by directly prompting an LLM to explain the execution trace,LLM\-only, across all narrative\-quality criteria: trace\-level faithfulness \(M1\-b\), exhaustiveness \(M2\), linguistic quality \(M3\), and overall satisfaction \(M4\)\.
- •H2 – Value of the structured representation\.Holding the narrator fixed, narratives produced from our structured report,Our\-Our, receive higher ratings than those produced from an LLM\-generated structured report,LLM\-Our, on structured\-report faithfulness \(M1\-a\), trace\-level faithfulness \(M1\-b\), exhaustiveness \(M2\), linguistic quality \(M3\), and overall satisfaction \(M4\)\.
- •H3 – Invariance across agent architectures\.The direction of the H1 and H2 effects is preserved across the three source agent architectures, such that the observed advantages ofOur\-OuroverLLM\-OnlyandOur\-OuroverLLM\-Ourdo not depend on the architecture generating the execution trace\.

All comparisons are paired on the same execution traces\. Because the human\-evaluated subset comprises 12 queries, with each query executed by up to three agents, ratings associated with the same query are not treated as independent observations\. For each directional comparison, we therefore obtainpp\-values from a one\-sided paired permutation test in which the sign of the paired difference is flipped at the query level and applied jointly to all agents associated with that query\. With 12 queries, the resulting null distribution contains2122^\{12\}possible sign assignments and can be enumerated exhaustively, yielding exactpp\-values\. We control multiplicity using the Benjamini–Hochberg procedure across criteria within each hypothesis and designate M4 \(overall satisfaction\) as the primary confirmatory outcome, with the remaining criteria treated as secondary\. Each comparison is reported together with the mean paired difference, a cluster bootstrap interval over queries, and the corresponding win, tie, and loss counts\. For analysis, ordinal categories are mapped to consecutive integers in ascending order \(11–33for faithfulness,11–44for M2–M4\), andΔ\\Deltadenotes the mean paired difference on that coding; it should be read as an average movement in rating categories rather than as an interval\-scale quantity\. The tests themselves do not depend on this choice: under the null the sign\-flip distribution is exact for any statistic computed from the paired differences, and the reported win, tie and loss counts are invariant under any order\-preserving recoding of the categories\. Confidence intervals forΔ\\Deltaare obtained by a nonparametric cluster bootstrap withB=5,000B=5\{,\}000replicates and a fixed seed\. The resampling unit is the query: each replicate draws with replacement as many clusters as the contrast contains and takes each drawn cluster entire, with all of its \(query, agent\) blocks\. The number of clusters per replicate is therefore fixed atkk, whereas the number of blocks is random with expectation equal to the observednn, since a query contributes between one and three blocks; the statistic is the unweighted mean over the blocks of a replicate, matching the point estimate\. Reported intervals are the2\.52\.5th and97\.597\.5th percentiles of theBBreplicate means\. The same procedure, with the sameBBand the same resampling unit, produces the intervals accompanying the correlations\. Atk≈12k\\approx 12the percentile bootstrap is known to under\-cover, so on the human subset these intervals are indicative and no inference rests on them\.

H3 is evaluated through consistency of effect direction across agent architectures; in the current study, this analysis is limited to the human\-evaluated subset, while LLM\-as\-a\-judge\-based assessment of the remaining architectures is left for future work\.

## 5Results

In the main text, we report the corresponding LLM\-as\-a\-judge analysis, conducted on a larger evaluation corpus and therefore providing greater statistical power\. Where relevant, we compare these results with the directional findings observed in the human evaluation\. Detailed hypothesis tests and further analyses of the human\-evaluation results are reported in Appendix[A](https://arxiv.org/html/2609.06063#A1)\.

#### Structured\-representation fidelity \(H0\)\.

Table[4](https://arxiv.org/html/2609.06063#S4.T4)reports NFI for the two structured\-report variants against the human\-annotated reference\. Our structured reports substantially outperform those generated by directly prompting the LLM: across 63 reports, our method achievesNFI=0\.816\\mathrm\{NFI\}=0\.816\(SD=0\.219\\mathrm\{SD\}=0\.219\), compared with0\.0610\.061\(SD=0\.124\\mathrm\{SD\}=0\.124\) for the LLM baseline\. The advantage is consistent across all three agent architectures and shows no evident degradation with trace length, whereas the LLM\-generated reports decline as the number of steps increases: under the strict matching criterion74%74\\%of them score zero, versus none of ours\. These results support H0 and the use of our structured report as the reference for evaluating narrative faithfulness\.

#### LLM\-as\-judge validation\.

Before applying the LLM\-as\-judge evaluation at scale, we assess its agreement with human annotations on the subset evaluated by both methods \(Table[5](https://arxiv.org/html/2609.06063#S4.T5)\)\. Raw agreement is highest forOur\-Our\(9494–97%97\\%\), intermediate forLLM\-Our\(6161–90%90\\%\), and lower forLLM\-only\(up to70%70\\%\)\. These values must be read together with the marginal distributions: where ratings are highly skewed—Our\-Ourhas fewer than three human\-negative ratings per criterion—high raw agreement carries little information andκw\\kappa\_\{w\}is correspondingly attenuated\.

For criteria with sufficient variation, agreement is substantial: pooledκw\\kappa\_\{w\}is0\.630\.63for cross\-level faithfulness,0\.520\.52for linguistic quality, and0\.410\.41for overall quality\. The main discrepancy sits in the calibration, since the judge is systematically more positive than humans, most visibly on the weaker configurations \(e\.g\., for M4 theLLM\-onlypositive rate is22%22\\%for humans but83%83\\%for the judge\), which compresses the differences between conditions, particularly for exhaustiveness\. Consequently the judge recovers only9/259/25human\-negative M2 cases and11/3311/33M4 cases, so M2\-based judge scores in particular cannot replace human assessment\. This may reflect differing weightings of surface fluency, apparent completeness, and trace\-grounded evidence between human and model\-based evaluation\.

The cross\-level faithfulness comparison \(M1\-a\_our vs\. M1\-b\) provides an additional test of the structured representation\. Without access to the ground\-truth trace, the judge nonetheless identifies14/1614/16narratives that humans judge unfaithful to the trace\. Together with the NFI analysis, this supports using the structured report as a practical proxy for trace\-level faithfulness assessment when direct trace\-based evaluation is costly\.

#### End\-to\-End Superiority \(H1\)

We first evaluate H1, which tests whether the completeOur\-Ourpipeline yields higher scores than theLLM\-onlyconfiguration\. As shown in Table[6](https://arxiv.org/html/2609.06063#S4.T6), all three metrics exhibit positive effects on the LLM\-as\-judge corpus\. The effect is smallest for M2 \(Δ=0\.07\\Delta=0\.07\), but remains statistically significant\. The effects are substantially larger for M3 \(Δ=1\.18\\Delta=1\.18\) and M4 \(Δ=0\.62\\Delta=0\.62\), both withp<0\.0001p<0\.0001\. The corresponding human\-evaluation estimates follow the same ordering for M3 and M4, although their magnitudes are larger\. Overall, these results support H1: replacing the LLM\-only pipeline with the complete proposed pipeline improves performance across all evaluated criteria, with the strongest advantage observed for M3\.

#### Value of the Structured Representation \(H2\)

H2 isolates the contribution of the structured representation by comparing configurations that differ in whether this representation is incorporated into the generation process\. The results in Table[6](https://arxiv.org/html/2609.06063#S4.T6)reveal a differentiated pattern across metrics\. M2 shows a positive effect \(Δ=0\.49\\Delta=0\.49\), as does M4, with a substantially larger effect \(Δ=0\.82\\Delta=0\.82\); both effects are statistically significant\. In contrast, M3 shows essentially no difference between the two configurations \(Δ≈0\\Delta\\approx 0,95%95\\%CI\[−0\.03,0\.03\]\[\-0\.03,0\.03\],p=0\.6418p=0\.6418\)\. This pattern is also reflected in the human\-evaluation estimates, although the effects are generally larger in magnitude \(see Appendix[A](https://arxiv.org/html/2609.06063#A1)\)\. This is expected given that M3 evaluates linguistic realization, which depends on the narrator, and the two configurations compared under H2 share the same narrator\.

#### Invariance Across Agent Architectures \(H3\)

H3 examines whether the effects observed under H1 and H2 are consistent across agent architectures\. Table[7](https://arxiv.org/html/2609.06063#S4.T7)reports the mean paired differences separately for ReAct, Plan\-and\-act, and TRAIL agent\. Under H1, all three measures show a positive effect across all architectures, yielding consistent directions\. The same pattern holds under H2 for M2 and M4, with positive effects across agent configurations\. In contrast, M3 remains effectively null under H2 across all three architectures \(Δ=−0\.01\\Delta=\-0\.01,−0\.01\-0\.01, and\+0\.03\+0\.03, respectively\)\. This behavior is expected given that the two configurations compared under H2 share the same narrator across architectures, while M3 evaluates linguistic realization produced by that narrator\. Thus, changing the underlying agent architecture does not introduce a systematic M3 difference between the two configurations\.

## 6Conclusion

We presented a post\-hoc framework for explaining AI agents through their execution traces\. By transforming heterogeneous traces into an auditable structured representation that explicitly separates behavioral commitments from available internal and external evidence, the framework provides both step\-level diagnostics and trace\-grounded natural\-language explanations\. Across three agent architectures and two QA benchmarks, the structured reports were substantially more faithful to human references than direct LLM generation, while narratives produced from these reports achieved higher faithfulness, exhaustiveness, and overall quality than direct trace\-to\-text explanations\. The results further show that the structured representation can serve as a practical proxy for trace\-level faithfulness when direct trace inspection is costly\.

Current limitations include the evaluation’s focus on tool\-mediated QA tasks and a relatively small human\-evaluated subset\. We stress that, by construction, the framework can explain only behavior and evidence recorded in the execution trace; incomplete or insufficient logging necessarily limits what can be reconstructed\. Future work will evaluate the framework on a wider range of agentic tasks and increasingly complex execution scenarios\.

## References

- Acharyaet al\.\(2025\)D\. B\. Acharya, K\. Kuppan, and B\. DivyaAgentic ai: autonomous intelligence for complex goals—a comprehensive survey\.IEEe Access13,pp\. 18912–18936\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p1.1)\.
- Chaduvulaet al\.\(2026\)S\. Chaduvula, J\. Ho, K\. Kim, A\. Narayanan, M\. Alinoori, M\. Garg, D\. Ramachandram, and S\. RazaFrom features to actions: explainability in traditional and agentic ai systems\.arXiv preprint arXiv:2602\.06841\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p2.1)\.
- Chakrabarty \(2025\)P\. K\. ChakrabartyCausal inference in agentic ai: bridging explainability and dynamic decision making\.International Journal of Science and Research \(IJSR\)14\(4\),pp\. 10–21275\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p2.1)\.
- Choet al\.\(2026\)S\. Cho, K\. Chawla, P\. Cai, Z\. Liu, C\. Zhu, S\. Zhang, and S\. SahuAsk, don’t judge: binary questions for interpretable LLM evaluation and self\-improvement\.In2nd Workshop on Compositional Learning: Safety, Interpretability, and Agents,External Links:[Link](https://openreview.net/forum?id=tJytD1JIkR)Cited by:[§3\.2](https://arxiv.org/html/2609.06063#S3.SS2.SSS0.Px2.p1.1)\.
- Collacoet al\.\(2026\)B\. G\. Collaco, S\. Prabha, C\. A\. Gomez\-Cabello, S\. A\. Haider, A\. Genovese, N\. G\. Wood, N\. Gopala, R\. Raman, E\. O\. Hester, and A\. J\. ForteExplainable agentic artificial intelligence in healthcare: a scoping review\.Bioengineering13\(5\),pp\. 513\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p2.1)\.
- Deshpandeet al\.\(2025\)D\. Deshpande, V\. Gangal, H\. Mehta, J\. Krishnan, A\. Kannappan, and R\. QianTrail: trace reasoning and agentic issue localization\.arXiv preprint arXiv:2505\.08638\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.06063#S4.SS1.SSS0.Px1.p2.1)\.
- Ehsanet al\.\(2026\)U\. Ehsan, A\. Alabdulkarim, K\. Holstein, M\. K\. Lee, A\. Riener, and J\. D\. WeiszHuman\-centered explainable ai \(hcxai\): re\-examining xai in the era of agentic ai\.InProceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems,pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p2.1),[§1](https://arxiv.org/html/2609.06063#S1.p3.1)\.
- He and Martens \(2026\)Y\. He and D\. MartensAn agentic approach to generating xai\-narratives\.arXiv preprint arXiv:2603\.20003\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Huet al\.\(2026\)J\. Hu, X\. Huang, Q\. He, Y\. Sun, Y\. Dong, and X\. HuangResponsible agentic ai requires explicit provenance\.arXiv preprint arXiv:2605\.17169\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Khalidet al\.\(2026\)O\. Khalid, A\. U\. H\. Farooqi, and M\. BilalAgentic ai: a review, applications and open research challenges\.Computing108\(6\),pp\. 80\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p1.1)\.
- Mazharet al\.\(2026\)A\. Mazhar, H\. Suri, and S\. GalhotraTrace\-level analysis of information contamination in multi\-agent systems\.InProceedings of the ACM Conference on AI and Agentic Systems,pp\. 483–496\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Mehta \(2026\)S\. MehtaMatrag: multi\-agent transparent retrieval\-augmented generation for explainable recommendations\.arXiv preprint arXiv:2604\.20848\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Mialonet al\.\(2024\)G\. Mialon, C\. Fourrier, T\. Wolf, Y\. LeCun, and T\. ScialomGaia: a benchmark for general ai assistants\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9025–9049\.Cited by:[§4\.1](https://arxiv.org/html/2609.06063#S4.SS1.SSS0.Px1.p2.1)\.
- Mulianet al\.\(2026\)H\. Mulian, S\. Zeltyn, I\. Levy, L\. Galanti, A\. Yaeli, and S\. ShlomovAgentFixer: from failure detection to fix recommendations in agentic systems\.InProceedings of the 2026 International Workshop on Agentic Engineering,pp\. 96–103\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Razaet al\.\(2026\)S\. Raza, A\. Radwan, S\. Chaduvula, M\. Alinoori, and C\. EmmanouilidisTransparency in agentic ai: a survey of interpretability, explainability, and governance\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p2.1),[§1](https://arxiv.org/html/2609.06063#S1.p3.1)\.
- Shahet al\.\(2026\)M\. B\. Shah, M\. M\. Morovati, M\. M\. Rahman, and F\. KhomhCharacterizing faults in agentic ai: a taxonomy of types, symptoms, and root causes\.arXiv preprint arXiv:2603\.06847\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p1.1)\.
- Sharmaet al\.\(2026\)R\. K\. Sharma, S\. Barke, and B\. ZornWillful disobedience: automatically detecting failures in agentic traces\.InProceedings of the ACM Conference on AI and Agentic Systems,pp\. 584–596\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Silvestriet al\.\(2025\)M\. Silvestri, V\. Vineis, E\. Gabrielli, F\. Giorgi, F\. Veglianti, F\. Silvestri, and G\. TolomeiA survey on explainable ai narratives based on large language models\.Cited by:[§4\.1](https://arxiv.org/html/2609.06063#S4.SS1.SSS0.Px3.p4.1),[§4\.2](https://arxiv.org/html/2609.06063#S4.SS2.SSS0.Px3.p1.1)\.
- Vineiset al\.\(2026\)V\. Vineis, M\. Silvestri, L\. Antonelli, F\. Betello, and G\. TolomeiPONTE: personalized orchestration for natural language trustworthy explanations\.arXiv preprint arXiv:2603\.06485\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Wang \(2026\)Z\. G\. WangAgentTrace: causal graph tracing for root cause analysis in deployed multi\-agent systems\.arXiv preprint arXiv:2603\.14688\.Cited by:[§2](https://arxiv.org/html/2609.06063#S2.p1.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§4\.1](https://arxiv.org/html/2609.06063#S4.SS1.SSS0.Px1.p4.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, I\. Shafran, K\. R\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.InNeurIPS 2022 Foundation Models for Decision Making Workshop,External Links:[Link](https://openreview.net/forum?id=tvI4u1ylcqs)Cited by:[§4\.1](https://arxiv.org/html/2609.06063#S4.SS1.SSS0.Px1.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[§4\.2](https://arxiv.org/html/2609.06063#S4.SS2.p1.1)\.
- Zhuet al\.\(2026\)J\. Zhu, D\. Gandhi, H\. Joshi, A\. R\. Mianroodi, S\. A\. Kocak, and D\. RamachandranInterpreting agentic systems: beyond model explanations to system\-level accountability\.arXiv preprint arXiv:2601\.17168\.Cited by:[§1](https://arxiv.org/html/2609.06063#S1.p1.1),[§1](https://arxiv.org/html/2609.06063#S1.p2.1)\.

## Appendix ADetailed Human\-Evaluation Results

#### End\-to\-end superiority \(H1\)\.

H1 compares narratives produced by the full pipeline \(Our\-Our\) with those generated by directly prompting an LLM to explain the execution trace \(LLM\-only\)\. The full pipeline significantly outperforms the direct explainer on all four criteria, withpBH=0\.0019p\_\{\\mathrm\{BH\}\}=0\.0019throughout \(Table[9](https://arxiv.org/html/2609.06063#A1.T9)\)\. The mean paired differences areΔ=1\.35\\Delta=1\.35for overall quality,1\.111\.11for trace\-level faithfulness,1\.041\.04for linguistic quality, and0\.850\.85for exhaustiveness\. Because Across the four criteria and 26 paired traces,Our\-Ourreceives higher ratings in 80 of 104 comparisons, ties in 23, and loses only once\.

The distribution of ratings \(Figure[3](https://arxiv.org/html/2609.06063#A1.F3)\) further clarifies the difference\. For exhaustiveness,LLM\-onlyreceives positive ratings in73%73\\%of cases but only8%8\\%reach the highest category\. This apparent strength does not extend to trace\-level faithfulness:46%46\\%ofLLM\-onlynarratives receive an outright*Disagree*\. Overall quality follows the faithfulness pattern: only31%31\\%ofLLM\-onlynarratives are rated positively, versus97%97\\%forOur\-Our\. Thus direct prompting can produce narratives perceived as reasonably complete and fluent, but these qualities do not translate into faithful explanations of the trace, consistent with the benefit of grounding generation in the structured, step\-level representation\.

#### Value of the Structured Representation \(H2\)\.

H2 holds the narrator fixed and varies only the structured report, isolating the contribution of the structuring stage\.Our\-Oursignificantly outperformsLLM\-Ouron four of five criteria \(Table[9](https://arxiv.org/html/2609.06063#A1.T9)\): exhaustiveness \(Δ=1\.03\\Delta=1\.03\), overall quality \(Δ=1\.00\\Delta=1\.00\), faithfulness to the source report \(Δ=0\.55\\Delta=0\.55\), and trace\-level faithfulness \(Δ=0\.52\\Delta=0\.52\)\. Linguistic quality does not significantly differ \(Δ=0\.39\\Delta=0\.39\), consistent with both configurations sharing the same narrator\. The effect is smaller and more frequently tied than in the end\-to\-end comparison, with 75 wins, 70 ties, and 10 losses across 155 paired comparisons\.

The distribution of ratings \(Figure[3](https://arxiv.org/html/2609.06063#A1.F3)\) shows how the deficit in the LLM\-generated report propagates: narratives built on it remain largely faithful to the report itself \(90%90\\%positive M1\-a\) but exhibit substantially lower content quality \(exhaustiveness positive in48%48\\%vs\.94%94\\%forOur\-Our; overall quality55%55\\%vs\.97%97\\%\), while linguistic quality stays similar \(87%87\\%vs\.94%94\\%\)\. The primary effect of the structuring stage is thus on the content available to the narrator rather than on linguistic realization\. For M1\-a,1717of3131paired comparisons are ties whileOur\-Ourwins the remaining1414and loses none: when the LLM report is adequate both configurations produce faithful narratives, and our pipeline’s advantage emerges primarily when the intermediate representation becomes limiting\.

#### Invariance Across Agent Architectures \(H3\)\.

Table[9](https://arxiv.org/html/2609.06063#A1.T9)reports the H1 and H2 contrasts separately for each agent architecture\. All subgroup effects are positive: none of the 27 architecture\-specific effects reverses direction or is zero, so H3 is supported in terms of effect\-direction consistency\. The subgroups contain only 8–12 queries, insufficient for a reliable condition\-by\-architecture interaction test, so we report directions and magnitudes withoutpp\-values\. The confirmatory outcome, overall quality, is particularly stable \(H1:\+1\.33\+1\.33,\+1\.33\+1\.33,\+1\.38\+1\.38; H2:\+1\.00\+1\.00,\+1\.08\+1\.08,\+0\.88\+0\.88across ReAct, Plan\-and\-act, and TRAIL\)\. Effect magnitudes vary for individual criteria—H2 faithfulness to the source report ranges from\+0\.27\+0\.27on ReAct to\+1\.12\+1\.12on TRAIL, and linguistic quality shows the smallest effects \(\+0\.64\+0\.64,\+0\.33\+0\.33,\+0\.12\+0\.12\)\. At the judge level, the same contrasts stay direction\-consistent across architectures \(Table[7](https://arxiv.org/html/2609.06063#S4.T7)\), the one exception being H2 on M3, which is≈0\\approx 0everywhere as expected under a fixed narrator\.

#### Relations among criteria\.

Figure[3](https://arxiv.org/html/2609.06063#A1.F3)shows strong associations between overall quality \(M4\) and trace\-level faithfulness \(ρ=0\.85\\rho=0\.85\), exhaustiveness \(ρ=0\.84\\rho=0\.84\), and delivery \(ρ=0\.81\\rho=0\.81\), suggesting that human holistic evaluation reflects substantive content and faithfulness rather than fluency alone\. Faithfulness to the structured report \(M1\-a\) is comparatively weakly associated with the narrative criteria \(ρ=0\.23\\rho=0\.23–0\.400\.40\): a narrative may faithfully reflect an imperfect report, so M1\-a alone is insufficient to characterize explanation quality\. NFI shows moderate associations \(ρ=0\.33\\rho=0\.33–0\.480\.48\), indicating that the structured representation constrains narrative quality without fully determining it\.

Table 8:Hypothesis tests on the human\-evaluated subset \(12 queries×\\times3 agents\)\.nn= paired blocks; W, T and L = wins, ties and losses for the first condition;Δ\\Delta= mean paired difference with a cluster bootstrap CI over queries \(indicative at 12 clusters\);pp= exact one\-sided query\-level permutation;pBHp\_\{\\mathrm\{BH\}\}= Benjamini–Hochberg corrected within hypothesis\. Bold = pre\-designated confirmatory outcome\. H1: Our–Our\>\>LLM\-only; H2: Our–Our\>\>LLM–Our; H0: our structured report\>\>LLM structured report\.
Table 9:H3 direction consistency, according to human evaluations\. Entries are mean paired differences within each agent architecture\. H3 is supported when the effect retains its sign across all three architectures\.

![Refer to caption](https://arxiv.org/html/2609.06063v1/human_all_metrics.png)Figure 2:Distribution of Likert ratings provided by human evaluators across the five evaluation dimensions\. Bars show the proportion of responses in each category for Our–Our, LLM–Our, and LLM\-only\. By design, dimension M1\-a does not include an LLM\-only configuration\.
![Refer to caption](https://arxiv.org/html/2609.06063v1/corr_human.png)Figure 3:Spearman correlation across evaluation dimensions\. Coefficients are computed over pairwise\-complete ratings:n=63n=63when at least one dimension in the pair evaluates only structured outputs, andn=89n=89when both dimensions evaluate narratives\.