The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations

# The Failures of Marginal Influence-Based Attribution Methods for Global Time Series Explanations
Source: [https://arxiv.org/html/2607.16236](https://arxiv.org/html/2607.16236)
###### Abstract

Explainability methods for time series models predominantly produce flat attribution scores: they quantify the direct influence of a feature at a timestamp by a scalar\. We prove that the dominant failure mode of such methods is not the scalar format itself but a fundamental computational mismatch: existing methods compute scores via marginal conditioning or off\-manifold gradients, both of which conflate direct temporal dependencies with mediated ones under autocorrelation\. We also define DAG\-faithfulness: an explanation is DAG\-faithful if the temporal dependency graph it encodes is Markov\-equivalent to the temporal directed acyclic graph \(DAG\) implicitly learned by the model\. Particularly, we observe that standard attribution methods, specifically SHAP, are not DAG\-faithful in general, and that recent time\-series\-aware extensions inherit the same computational limitation\.

## 1Introduction

The deployment of deep learning on time series data in domains ranging from medical monitoring to financial forecasting to industrial fault detection has created pressing demand for explanations that are both human\-comprehensible and technically faithful to the model’s reasoning process\. The field of explainable AI \(XAI\) has produced a rich toolkit of attribution methods, yet the dominant paradigm remains*importance scoring*: each input feature at each timestep receives a scalar value indicating its "direct" contribution to a given prediction\.

We contend that this paradigm is structurally inadequate for time series models\. The reasoning of a temporal model, whether an RNN, a Transformer, or a temporal convolutional network, is inherently relational: what matters is not merely*that*a feature took a particular value, but*how*its value at one time influenced variables at another time, conditioned on the broader temporal context\. Most existing importance\-scoring methods do not express this; they ignore a high\-dimensional relational structure for a one\-dimensional summary, potentially discarding the conditional dependency information that constitutes the model’s actual reasoning\.

This paper makes the following contributions:

1. 1\.We formalize the computational gap between existing flat attribution scoring methods and the conditional dependency structure of temporal models and on\-manifold dependency \(Section[3](https://arxiv.org/html/2607.16236#S3)\)\.
2. 2\.We define*DAG\-faithfulness*as the appropriate faithfulness criterion, together with admissibility conditions that constrain candidate attribution scoring methods \(Section[4](https://arxiv.org/html/2607.16236#S4)\)\.
3. 3\.We show that standard attribution methods fail DAG\-faithfulness by construction, that time\-series\-aware extensions inherit the same failure mode\.
4. 4\.We outline an evaluation protocol and identify open research challenges, centrally the construction of admissible and DAG\-faithful instantiations \(Sections[4\.3](https://arxiv.org/html/2607.16236#S4.SS3)and[6](https://arxiv.org/html/2607.16236#S6)\)\.

## 2Background and Related Work

### 2\.1XAI for Time Series

Existing XAI methods for time series fall into four broad families\.*Gradient\-based methods*Simonyanet al\.\([2014](https://arxiv.org/html/2607.16236#bib.bib1)\); Sundararajanet al\.\([2017](https://arxiv.org/html/2607.16236#bib.bib3)\)compute the sensitivity of the output with respect to each input dimension, yielding a saliency map of shapeT×DT\\times D\.*Perturbation\-based methods*Ribeiroet al\.\([2016](https://arxiv.org/html/2607.16236#bib.bib4)\); Lundberg and Lee \([2017](https://arxiv.org/html/2607.16236#bib.bib5)\)estimate feature importance by measuring output changes under input occlusion, again collapsing to scalar attributions per feature–time pair\.*Attention\-based methods*Vaswaniet al\.\([2017](https://arxiv.org/html/2607.16236#bib.bib6)\)expose the model’s internal attention weights, which form a matrix over timesteps; whether these weights constitute faithful explanations remains contestedJain and Wallace \([2019](https://arxiv.org/html/2607.16236#bib.bib7)\); Wiegreffe and Pinter \([2019](https://arxiv.org/html/2607.16236#bib.bib8)\); Abnar and Zuidema \([2020](https://arxiv.org/html/2607.16236#bib.bib9)\)\.

A fourth, more recent, family of*temporally\-aware attribution methods*explicitly accounts for temporal dependence in the computation of importance\. FITTonekaboniet al\.\([2020](https://arxiv.org/html/2607.16236#bib.bib21)\)quantifies importance via KL\-divergence of the predictive distribution under feature occlusion\. DynamaskCrabbé and van der Schaar \([2021](https://arxiv.org/html/2607.16236#bib.bib22)\)learns near\-binary perturbation masks over the input sequence\. WinITLeunget al\.\([2023](https://arxiv.org/html/2607.16236#bib.bib23)\)extends feature\-removal scores over sliding windows to capture delayed effects\. TimeSHAPBentoet al\.\([2021](https://arxiv.org/html/2607.16236#bib.bib24)\)and ShapTimeZhanget al\.\([2023](https://arxiv.org/html/2607.16236#bib.bib2)\)adapt SHAP to sequence models with event\-level and feature\-level Shapley values\. These methods move beyond generic saliency by modeling the temporal structure of the input, but as we show in Section[3](https://arxiv.org/html/2607.16236#S3), some of these methods fail under auto\-correlation: they cannot tell between mediated effect onXj\(t\)X\_\{j\}^\{\(t\)\}ofXi\(t−k\)X\_\{i\}^\{\(t\-k\)\}*via*Xl\(t−k′\)X\_\{l\}^\{\(t\-k^\{\\prime\}\)\},0<k′≤k0<k^\{\\prime\}\\leq k,i,j,l∈\[D\]i,j,l\\in\[D\]and direct effect\.

### 2\.2Causal Structure in Time Series

Granger causalityGranger \([1969](https://arxiv.org/html/2607.16236#bib.bib10)\)operationalizes temporal influence as predictive improvement:XXGranger\-causesYYif past values ofXXimprove the forecast ofYYbeyond past values ofYYalone\. Structural causal modelsPeterset al\.\([2017](https://arxiv.org/html/2607.16236#bib.bib11)\)and dynamic Bayesian networksMurphy \([2002](https://arxiv.org/html/2607.16236#bib.bib12)\)provide richer frameworks for representing conditional independence in temporal data\. Modern causal discovery methods for time series, including constraint\-based approaches such as PCMCI\+Runge \([2020](https://arxiv.org/html/2607.16236#bib.bib13)\)and score\-based approaches such as DYNOTEARSPamfilet al\.\([2020](https://arxiv.org/html/2607.16236#bib.bib14)\)and RhinoGonget al\.\([2023](https://arxiv.org/html/2607.16236#bib.bib15)\), operate on the raw data to recover its underlying dependency structure\. Our framework connects XAI faithfulness to these causal structures but reverses the target: rather than discovering dependencies in the data, we seek to recover the dependencies that a trained model has learned, grounding explanation quality in the model’s learned probabilistic geometry rather than its output value alone\.

### 2\.3Faithfulness in XAI

Existing faithfulness criteria are predominantly*output\-centric*: an explanation is faithful if perturbing highly\-attributed features changes the output more than perturbing low\-attributed featuresSameket al\.\([2016](https://arxiv.org/html/2607.16236#bib.bib16)\); Alvarez\-Melis and Jaakkola \([2018](https://arxiv.org/html/2607.16236#bib.bib17)\)\. This criterion is necessary but insufficient; a method can pass it while misrepresenting the model’s internal reasoning structureKindermanset al\.\([2019](https://arxiv.org/html/2607.16236#bib.bib18)\); Jacovi and Goldberg \([2020](https://arxiv.org/html/2607.16236#bib.bib19)\)\. A parallel line of work on mechanistic and causal\-abstraction\-based faithfulnessGeigeret al\.\([2021](https://arxiv.org/html/2607.16236#bib.bib20)\)moves toward structure\-centric criteria by asking whether an explanation preserves the model’s internal causal structure under interventions\. We contribute to this shift by proposing a faithfulness criterion specific to temporal models: the explanation must recover the model’s conditional independence structure, not merely correlate with its outputs\.

## 3The Computational Gap

This section makes the central claim precise\. We first fix notation, then define the class of methods we target \(flat attribution\) and the structure they ought to recover \(the model dependency graph\)\. We then isolate a property that any faithful explanation \(DAG faithfulnes\) must have and prove that no flat method can hold it\. A proof argument is presented in the appendix\.

Denote\[D\]=\{1,…,D\}\[D\]=\\\{1,\\dots,D\\\}and\[T\]=\{1,…,T\}\[T\]=\\\{1,\\dots,T\\\}\. Let𝐗=\{Xi\(t\)\}i∈\[D\]​t∈\[T\]\\mathbf\{X\}=\\\{X\_\{i\}^\{\(t\)\}\\\}\_\{i\\in\[D\]\\;t\\in\[T\]\}denote a multivariate time series withDDvariables andTTtimesteps\. Letf:ℝT×D→ℝW×Df:\\mathbb\{R\}^\{T\\times D\}\\to\\mathbb\{R\}^\{W\\times D\}be a trained time series model withWWthe forecast window size, that takes such a window as input and produces aW×DW\\times D\-dimensional forecast at the nextWWtime steps\. Throughout,p​\(𝐱\)p\(\\mathbf\{x\}\)denotes the*reference distribution*over inputs against which the explanation is defined: a probability measure onℝT×D\\mathbb\{R\}^\{T\\times D\}with respect to which all expectations, conditional independences, and support statements are taken\. In practice,ppis the deployment distribution offf, approximated empirically by the evaluation set on which the global explanation is computed\.

The methods we critique share one feature: each score is a function of a single input coordinate considered in isolation, whether through marginal conditioning or through a gradient\. We name this class precisely\.

###### Definition 1\(Flat Attribution\)\.

A flat attribution method produces a mapϕ:ℝT×D→ℝT×D\\phi:\\mathbb\{R\}^\{T\\times D\}\\to\\mathbb\{R\}^\{T\\times D\}, assigning a scalar importance scoreϕi\(t\)​\(𝐱\)\\phi\_\{i\}^\{\(t\)\}\(\\mathbf\{x\}\)to each variable\-time pair\(Xi,t\)\(X\_\{i\},t\)for a given input𝐱\\mathbf\{x\}via marginal or gradient based methods\.

The label “flat” refers not to the scalar output but to the computation: the score for\(Xi,t\)\(X\_\{i\},t\)never conditions on the value of the other inputs that may mediate its effect\. Contrary to classic XAI or even XAI for time series classification that use flat attribution scoring, XAI for forecasting involves not only deriving feature relevance but also understanding it with respect to the model’s time\-dependent forecasting dynamics; i\.e\., an appropriate temporal\-aware XAI method should be able to properly portray the data dynamics as learned by the model in a faithful way\. The most interpretable way of portraying such a dynamic is via a Directed Acyclic Graph \(DAG\), namely a*Model Dependency Graph*that encodes forecasting dependencies via edges between vertices which we now define\.

###### Definition 2\(Model Dependency Graph \(MDDAG\)\)\.

Letf:ℝT×D→ℝW×Df:\\mathbb\{R\}^\{T\\times D\}\\to\\mathbb\{R\}^\{W\\times D\}be a trained time series model and letp​\(𝐱\)p\(\\mathbf\{x\}\)be the reference distribution\. The*model dependency graph \(MDDAG\)*𝒢f\\mathcal\{G\}\_\{f\}is a directed acyclic graph with node set

V=\{Xi\(t\):i∈\[D\],t∈\[T\+W\]\},V=\\bigl\\\{X\_\{i\}^\{\(t\)\}:i\\in\[D\],\\ t\\in\[T\+W\]\\bigr\\\},where theD⋅TD\\cdot Tnodes witht∈\[T\]t\\in\[T\]are input positions and theD⋅WD\\cdot Wnodes att∈\{T\+1,…,T\+W\}t\\in\\\{T\+1,\\ldots,T\+W\\\}are forecast targets \(sinks\)\. A directed edgeXi\(t1\)→Xj\(T\+w\)X\_\{i\}^\{\(t\_\{1\}\)\}\\to X\_\{j\}^\{\(T\+w\)\},j∈\[D\]j\\in\[D\]witht1∈\[T\]t\_\{1\}\\in\[T\]andw∈\[W\]w\\in\[W\]belongs to𝒢f\\mathcal\{G\}\_\{f\}if and only if

Xi\(t1\)⟂̸⟂f\(𝐗\)j,w\|𝐗∖\{Xi\(t1\)\},f\(𝐗\)−j,w,f\(𝐗\)⋅,≠wX\_\{i\}^\{\(t\_\{1\}\)\}\\not\\perp\\\!\\\!\\\!\\perp f\(\\mathbf\{X\}\)\_\{j,w\}\\;\\Big\|\\;\\mathbf\{X\}\\setminus\\\{X\_\{i\}^\{\(t\_\{1\}\)\}\\\},\\ f\(\\mathbf\{X\}\)\_\{\-j,w\},\\ f\(\\mathbf\{X\}\)\_\{\\cdot,\\neq w\}underp​\(𝐱\)p\(\\mathbf\{x\}\), wheref​\(𝐗\)j,wf\(\\mathbf\{X\}\)\_\{j,w\}denotes the model’s forecast for variablejjat horizonww,f​\(𝐗\)−j,wf\(\\mathbf\{X\}\)\_\{\-j,w\}denotes the forecasts of all other variables at the same horizonww, andf​\(𝐗\)⋅,≠wf\(\\mathbf\{X\}\)\_\{\\cdot,\\neq w\}denotes the full joint forecast at all other horizonsw′≠ww^\{\\prime\}\\neq w\. Acyclicity holds by construction: edges run only from input positionst∈\[T\]t\\in\[T\]to forecast targetst∈\{T\+1,…,T\+W\}t\\in\\\{T\+1,\\ldots,T\+W\\\}\. An illustration is presented in Figure[1](https://arxiv.org/html/2607.16236#S3.F1)\.

![Refer to caption](https://arxiv.org/html/2607.16236v1/x2.png)Figure 1:The model dependency graphGfG\_\{f\}\.A trained modelf:ℝT×D→ℝW×Df:\\mathbb\{R\}^\{T\\times D\}\\\!\\to\\\!\\mathbb\{R\}^\{W\\times D\}maps an input \(training\) window ofTTtimesteps overDDvariables to aW×DW\\times Dforecast horizon\. An edgeXi\(t1\)→Xj\(T\+w\)X\_\{i\}^\{\(t\_\{1\}\)\}\\\!\\to\\\!X\_\{j\}^\{\(T\+w\)\}is present iff input celliiat timet1t\_\{1\}has a*direct*influence on forecast celljjthat is not mediated by any other input\. Here the model has learned the dependenciesc→1c\\\!\\to\\\!1,c→2c\\\!\\to\\\!2,e→1e\\\!\\to\\\!1,e→3e\\\!\\to\\\!3,f→4f\\\!\\to\\\!4; the remaining inputs \(a,b,da,b,d\) are edge\-free, exerting no direct influence on the forecast\.Faithfulness now becomes the guarantee that the XAI method properly learns this model dependency graph\. We introduce a novel metric,*DAG\-faithfulness*, to quantify this guarantee, using notions from graph theory and causal inference\. DAG\-faithfulness examines if the dependencies learned by the XAI match the edges in the MDDAG\.

###### Definition 3\(DAG\-Faithfulness\)\.

An explanation method is*DAG\-faithful*if the dependency graph𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}it induces encodes the exact same conditional independencies \(Markov Equivalence\) as the MDDAG𝒢f\\mathcal\{G\}\_\{f\}\. That is, for all input–forecast pairs\(Xi\(t1\),Xj\(T\+w\)\)\(X\_\{i\}^\{\(t\_\{1\}\)\},X\_\{j\}^\{\(T\+w\)\}\)witht1∈\[T\]t\_\{1\}\\in\[T\],w∈\[W\]w\\in\[W\]:

Xi\(t1\)⟂⟂𝒢𝒯Xj\(T\+w\)∣rest⇔Xi\(t1\)⟂⟂𝒢fXj\(T\+w\)∣restX\_\{i\}^\{\(t\_\{1\}\)\}\\perp\\\!\\\!\\\!\\perp\_\{\\mathcal\{G\}\_\{\\mathcal\{T\}\}\}X\_\{j\}^\{\(T\+w\)\}\\mid\\mathrm\{rest\}\\iff X\_\{i\}^\{\(t\_\{1\}\)\}\\perp\\\!\\\!\\\!\\perp\_\{\\mathcal\{G\}\_\{f\}\}X\_\{j\}^\{\(T\+w\)\}\\mid\\mathrm\{rest\}where⟂⟂𝒢\\perp\\\!\\\!\\\!\\perp\_\{\\mathcal\{G\}\}denotes independence in graph𝒢\\mathcal\{G\}\.

![Refer to caption](https://arxiv.org/html/2607.16236v1/x3.png)Figure 2:DAG\-faithfulness\.A global explanation is DAG\-faithful when the graphG𝒯G\_\{\\mathcal\{T\}\}induced by its transition tensor is Markov\-equivalent to the model dependency graphGfG\_\{f\}\. Here the two coincide: every source–target dependency the model has learned \(c→1c\\\!\\to\\\!1,c→2c\\\!\\to\\\!2,e→1e\\\!\\to\\\!1,e→3e\\\!\\to\\\!3,f→4f\\\!\\to\\\!4\) is recovered, and the edge\-free inputs are correctly left unconnected, so the explanation neither hallucinates nor misses an edge\.Requiring Markov equivalence is a rather strict constraint111Requiring Markov equivalence rather than exact graph recovery is deliberate and principled: it is the strongest structural claim that can be made from observational data alone, as multiple DAGs with different edge orientations may be indistinguishable without interventional dataPearl \([2009](https://arxiv.org/html/2607.16236#bib.bib25)\)\.; however, DAG faithfulness can be reduced to two fundamental admissibility conditions

- •\(C1\) On\-manifold evaluation\.The attribution functional depends onffonly through its behavior on the data distribution supportsupp​\(p\)\\mathrm\{supp\}\(p\)\. Two models that agree onsupp​\(p\)\\mathrm\{supp\}\(p\)must receive identical attributions; in particular, the attribution must not depend on off\-manifold reparametrizations offf\.
- •\(C2\) Conditional separation\.The attribution to inputXi\(t1\)X\_\{i\}^\{\(t\_\{1\}\)\}vanishes wheneverXi\(t1\)X\_\{i\}^\{\(t\_\{1\}\)\}is conditionally independent off​\(𝐗\)f\(\\mathbf\{X\}\)given the remaining inputs underp​\(𝐱\)p\(\\mathbf\{x\}\)\.

In what follows, we show that no flat attribution method, as defined, can hold either C1 or C2, or both\. First, we make the following claim

##### Claim \(thresholding commits the explanation to a graph\)\.

Any flat attribution methodϕ\\phiassigns a single global scoreΦi,t1f\\Phi^\{f\}\_\{i,\\,t\_\{1\}\}to each input variable\-time pair\(Xi,t1\)\(X\_\{i\},t\_\{1\}\)witht1∈\[T\]t\_\{1\}\\in\[T\]\. Thresholding this score atτ≥0\\tau\\geq 0induces a graph𝒢ϕτ\\mathcal\{G\}\_\{\\phi\}^\{\\tau\}in which\(Xi,t1\)\(X\_\{i\},t\_\{1\}\)is connected to the forecast if and only ifΦi,t1f\>τ\\Phi^\{f\}\_\{i,\\,t\_\{1\}\}\>\\tau\. A faithful explanation, therefore, requires:

Φi,t1f\>τ⇔∃j∈\[D\]:Xi\(t1\)→Xj\(T\+1\)∈𝒢f,\\Phi^\{f\}\_\{i,\\,t\_\{1\}\}\>\\tau\\iff\\exists\\,j\\in\[D\]:X\_\{i\}^\{\(t\_\{1\}\)\}\\to X\_\{j\}^\{\(T\+1\)\}\\in\\mathcal\{G\}\_\{f\},i\.e\., the attribution exceeds threshold iff the input has at least one direct edge to some forecast target \(the specificjjis retrievable by applying the attribution method per output coordinate\)\. In the single\-variable setting \(D=1D=1\), the existential collapses toX\(t1\)→X\(T\+1\)∈𝒢fX^\{\(t\_\{1\}\)\}\\to X^\{\(T\+1\)\}\\in\\mathcal\{G\}\_\{f\}, and we writeΦt1f\\Phi^\{f\}\_\{t\_\{1\}\}for the scalar attribution \(formally defined in \([5](https://arxiv.org/html/2607.16236#A1.E5)\) below\)\.

### 3\.1Case Studies

##### Scenario I:

LetD=1D=1\(single variable, so we drop theiiindex\) andT=2T=2, with input window\{X\(1\),X\(2\)\}\\\{X^\{\(1\)\},X^\{\(2\)\}\\\}and forecast target at positionT\+1=3T\+1=3\. Suppose the data\-generating process satisfies

X\(2\)=δX\(1\)\+ε,ε∼𝒩\(0,1\),ε⟂⟂X\(1\),δ≠0,X^\{\(2\)\}=\\delta\\,X^\{\(1\)\}\+\\varepsilon,\\quad\\varepsilon\\sim\\mathcal\{N\}\(0,1\),\\;\\varepsilon\\perp\\\!\\\!\\\!\\perp X^\{\(1\)\},\\;\\delta\\neq 0,\(1\)inducing temporal autocorrelationCov​\(X\(2\),X\(1\)\)≠0\\mathrm\{Cov\}\(X^\{\(2\)\},X^\{\(1\)\}\)\\neq 0\. Define

f1​\(𝐱\)\\displaystyle f\_\{1\}\(\\mathbf\{x\}\)=β​x\(2\),\\displaystyle=\\beta\\,x^\{\(2\)\},\(2\)f2​\(𝐱\)\\displaystyle f\_\{2\}\(\\mathbf\{x\}\)=β​x\(2\)\+γ​x\(1\),\\displaystyle=\\beta\\,x^\{\(2\)\}\+\\gamma\\,x^\{\(1\)\},\(3\)withβ,γ≠0\\beta,\\gamma\\neq 0\. Applying Definition[2](https://arxiv.org/html/2607.16236#Thmdefinition2)directly: conditioning onX\(2\)X^\{\(2\)\},f1​\(𝐗\)=β​X\(2\)f\_\{1\}\(\\mathbf\{X\}\)=\\beta X^\{\(2\)\}is deterministic inX\(2\)X^\{\(2\)\}alone, soX\(1\)⟂⟂f1\(𝐗\)∣X\(2\)X^\{\(1\)\}\\perp\\\!\\\!\\\!\\perp f\_\{1\}\(\\mathbf\{X\}\)\\mid X^\{\(2\)\}andX\(1\)→X\(3\)∉𝒢f1X^\{\(1\)\}\\to X^\{\(3\)\}\\notin\\mathcal\{G\}\_\{f\_\{1\}\}\. Forf2f\_\{2\}, the same conditioning leaves aγ​X\(1\)\\gamma X^\{\(1\)\}term, soX\(1\)→X\(3\)∈𝒢f2X^\{\(1\)\}\\to X^\{\(3\)\}\\in\\mathcal\{G\}\_\{f\_\{2\}\}\. Hence𝒢f1≇𝒢f2\\mathcal\{G\}\_\{f\_\{1\}\}\\not\\cong\\mathcal\{G\}\_\{f\_\{2\}\}, and \(C2\) requires zero attribution toX\(1\)X^\{\(1\)\}underf1f\_\{1\}and nonzero underf2f\_\{2\}\.

![Refer to caption](https://arxiv.org/html/2607.16236v1/x4.png)Figure 3:SCENARIO I: data\-generating structure and the two models\.Under the processX\(2\)=δ​X\(1\)\+εX^\{\(2\)\}=\\delta X^\{\(1\)\}\+\\varepsilonwithε⟂⟂X\(1\)\\varepsilon\\\!\\perp\\\!\\\!\\\!\\perp\\\!X^\{\(1\)\}, the inputX\(1\)X^\{\(1\)\}influences the forecastX\(3\)X^\{\(3\)\}only through the mediatorX\(2\)X^\{\(2\)\}\(autocorrelation\)\.\(a\)Gf1G\_\{f\_\{1\}\}forf1=β​X\(2\)f\_\{1\}=\\beta X^\{\(2\)\}: the only edge into the forecast isX\(2\)→X\(3\)X^\{\(2\)\}\\\!\\to\\\!X^\{\(3\)\}, soX\(1\)X^\{\(1\)\}has no direct effect\.\(b\)Gf2G\_\{f\_\{2\}\}forf2=β​X\(2\)\+γ​X\(1\)f\_\{2\}=\\beta X^\{\(2\)\}\+\\gamma X^\{\(1\)\}: a genuine direct edgeX\(1\)→X\(3\)X^\{\(1\)\}\\\!\\to\\\!X^\{\(3\)\}is present\. The two graphs differ in exactly one edge, yet a flat marginal score assignsX\(1\)X^\{\(1\)\}the same value under both, unable to separate them\.
##### Scenario II:

LetD=1D=1,T=2T=2with input window\{X\(1\),X\(2\)\}\\\{X^\{\(1\)\},X^\{\(2\)\}\\\}and forecast targetX\(3\)X^\{\(3\)\}, and assume the deterministic data\-generating process

X\(2\)=δ​X\(1\),δ≠0,X^\{\(2\)\}=\\delta\\,X^\{\(1\)\},\\qquad\\delta\\neq 0,\(4\)sosupp​\(p\)\\mathrm\{supp\}\(p\)is the line\{\(x\(1\),x\(2\)\):x\(2\)=δ​x\(1\)\}\\\{\(x^\{\(1\)\},x^\{\(2\)\}\):x^\{\(2\)\}=\\delta x^\{\(1\)\}\\\}inℝ2\\mathbb\{R\}^\{2\}\. Define

f1​\(𝐱\)\\displaystyle f\_\{1\}\(\\mathbf\{x\}\)=β​x\(2\),\\displaystyle=\\beta\\,x^\{\(2\)\},f3​\(𝐱\)\\displaystyle f\_\{3\}\(\\mathbf\{x\}\)=β​x\(2\)\+γ​\(x\(2\)−δ​x\(1\)\)\.\\displaystyle=\\beta\\,x^\{\(2\)\}\+\\gamma\\,\\bigl\(x^\{\(2\)\}\-\\delta\\,x^\{\(1\)\}\\bigr\)\.Onsupp​\(p\)\\mathrm\{supp\}\(p\),x\(2\)−δ​x\(1\)=0x^\{\(2\)\}\-\\delta x^\{\(1\)\}=0identically, sof3=f1f\_\{3\}=f\_\{1\}pointwise onsupp​\(p\)\\mathrm\{supp\}\(p\)\. Hence𝒢f3=𝒢f1\\mathcal\{G\}\_\{f\_\{3\}\}=\\mathcal\{G\}\_\{f\_\{1\}\}, and the edgeX\(1\)→X\(3\)X^\{\(1\)\}\\to X^\{\(3\)\}is absent from both\. \(C2\) is satisfied: a method conditioning on the actual conditional distribution would assign zero toX\(1\)X^\{\(1\)\}under both models\.

![Refer to caption](https://arxiv.org/html/2607.16236v1/x5.png)Figure 4:SCENARIO II\.\(c\)The true dependency structure shared byf1f\_\{1\}andf3f\_\{3\}: the chainX\(1\)→X\(2\)→X\(3\)X^\{\(1\)\}\\\!\\to\\\!X^\{\(2\)\}\\\!\\to\\\!X^\{\(3\)\}, with no direct edgeX\(1\)→X\(3\)X^\{\(1\)\}\\\!\\to\\\!X^\{\(3\)\}\.\(d\)On the deterministic manifoldX\(2\)=δ​X\(1\)X^\{\(2\)\}=\\delta X^\{\(1\)\}, the modelf3f\_\{3\}agrees withf1f\_\{1\}everywhere onsupp​\(p\)\\mathrm\{supp\}\(p\), so the direct edgeX\(1\)→X\(3\)X^\{\(1\)\}\\\!\\to\\\!X^\{\(3\)\}is*absent onsupp​\(p\)\\mathrm\{supp\}\(p\)*\(crossed\)\. A gradient method nonetheless reports∂f3/∂x\(1\)=−γ​δ≠0\\partial f\_\{3\}/\\partial x^\{\(1\)\}=\-\\gamma\\delta\\neq 0, registering off\-manifold sensitivity and thus violating on\-manifold evaluation \(C1\)\.
#### 3\.1\.1Case Study A: Marginal Influence Based Methods

For our investigation, we consider the classic marginal influence\-based methods KernelSHAP \(KS\), TimeSHAP \(TS\), ShapTime \(ST\), and TS\-MULE \(TM\)\. We proceed in two ways, analytically and empirically\. For the analytical proof, see Appendix\. We make the following conjecture,

###### Proposition 1\.

These marginal influence\-based methods falter in scenario I and/or II, thus failing C1 and/or C2\.

The interpretation of this proposition:

- •Conditioning on a single variable fails to capture autocorrelation,
- •Baseline methods like SHAP are likely to lay mass on off\-manifold data\.

To prove our proposition, we present empirical evidence from the experiment\.

Table 1:Attribution toX\(1\)X^\{\(1\)\}in Scenario I and II\.Each method’s global scoreΦ1f\\Phi^\{f\}\_\{1\}for inputX\(1\)X^\{\(1\)\}under the three modelsf1,f2,f3f\_\{1\},f\_\{2\},f\_\{3\}\. The correct verdict isΦ1=0\\Phi\_\{1\}=0underf1f\_\{1\}andf3f\_\{3\}\(whereX\(1\)X^\{\(1\)\}has*no*direct edge to the forecast\) andΦ1\>0\\Phi\_\{1\}\>0underf2f\_\{2\}\(where it does\); ✓/×\\timesmarks whether the method agrees\. KS \(KernelSHAP\) and TS \(TimeSHAP\) handlef1f\_\{1\}andf2f\_\{2\}correctly but assign a spurious nonzero score underf3f\_\{3\}, wheref3≡f1f\_\{3\}\\\!\\equiv\\\!f\_\{1\}onsupp​\(p\)\\mathrm\{supp\}\(p\), a violation of on\-manifold evaluation \(C1\)\. ST \(ShapTime\) and TM \(TS\-MULE\) fail already onf1f\_\{1\}, attributing influence to the fully\-mediatedX\(1\)X^\{\(1\)\}, a violation of conditional separation \(C2\)\. No method is correct across all three models\.The results in Table[1](https://arxiv.org/html/2607.16236#S3.T1)prove our proposition\. We observe that even when the SHAP\-based methods TS and KS correctly identify spurious relationships caused by autocorrelation, they fail under on\-manifold constraints caused by faulty baselines used in their marginalization strategies\. And ShapTime and TS\-MULE fail in all cases where autocorrelation is present\.

### 3\.2Case Study: Gradient\-Based Methods

We consider in this case Saliency \(SA\), Integrated Gradients \(IG\), its extension Temporal Integrated Gradients \(TIG\), and SmoothGrad \(SG\)\.

###### Proposition 2\.

Gradient\-based methods fail in scenario II, thus failing C1\.

The interpretation:

- •Gradient computation causes a perturbation that makes explanations learn on off\-manifold data\.

Table 2:Gradient\-based attribution\.Global scoreΦ1f\\Phi^\{f\}\_\{1\}for inputX\(1\)X^\{\(1\)\}under the three models; the correct verdict isΦ1=0\\Phi\_\{1\}=0underf1,f3f\_\{1\},f\_\{3\}\(no direct edgeX\(1\)→X\(3\)X^\{\(1\)\}\\\!\\to\\\!X^\{\(3\)\}\) andΦ1\>0\\Phi\_\{1\}\>0underf2f\_\{2\}\(direct edge present\)\. SA \(Saliency\), IG \(Integrated Gradients\), TIG \(Temporal IG\), and SG \(SmoothGrad\) all separatef1f\_\{1\}fromf2f\_\{2\}correctly, but every method assigns a spurious nonzero score underf3f\_\{3\}: sincef3≡f1f\_\{3\}\\\!\\equiv\\\!f\_\{1\}onsupp​\(p\)\\mathrm\{supp\}\(p\)yet∂f3/∂x\(1\)=−γ​δ≠0\\partial f\_\{3\}/\\partial x^\{\(1\)\}=\-\\gamma\\delta\\neq 0, the gradient registers off\-manifold sensitivity, violating on\-manifold evaluation \(C1\)\. The uniform failure onf3f\_\{3\}is the gradient\-family signature, complementing the marginal methods’ \(C2\) failures in Table[1](https://arxiv.org/html/2607.16236#S3.T1)\.Table[2](https://arxiv.org/html/2607.16236#S3.T2)proves Proposition[2](https://arxiv.org/html/2607.16236#Thmproposition2)\. Indeed, in scenario II, we expect variableX1X\_\{1\}to be attributed0relevance towards prediction inf3f\_\{3\}ons​u​p​p​\(p\)supp\(p\), but for all gradient\-based methods, we consistently get non\-zero attributions forX1X\_\{1\}underf3f\_\{3\}, indicating that they break on\-manifold restrictions\.

## 4Admissible Instantiations of Temporal\-aware Explanations

### 4\.1Existing Methods

Among existing methods, the temporally\-aware perturbation approaches, namely FIT, WinIT and Dynamask are the closest to admissibility, i\.e\., their underlying structures fulfill C1 and C2:

##### FIT

scores each observation by its contribution to the shift in the model’s predictive distribution over time, measured as a KL divergence between the prediction given all observed features and the prediction under a counterfactual in which the target feature is treated as unobserved, that feature being replaced by a draw from a learned conditional generatorp​\(xunobs∣xobs\)p\(x\_\{\\text\{unobs\}\}\\mid x\_\{\\text\{obs\}\}\)rather than a fixed value\.

##### WinIT

extends FIT in two ways: it measures a feature’s importance on later predictions by aggregating over a temporal window \(capturing delayed effects\), and it accounts for dependencies between features across time\. Like FIT, it removes a feature by sampling an in\-distribution replacement from a conditional generator and scores the resulting predictive\-distribution change over the window\.

##### Dynamask

learns a near\-binary maskm∈\[0,1\]T×Dm\\in\[0,1\]^\{T\\times D\}by optimizing a perturbation objective with a sparsity/area regularizer; masked entries are replaced not by a constant baseline but by a temporally coherent perturbation operator \(e\.g\., a moving\-average / blurred version of the series\), so the perturbed input stays close to the local temporal structure\.

Each already incorporates, to a degree, the two ingredients our conditions require, namely conditional, in\-distribution replacement of removed features \(via a learned conditional generator for FIT and WinIT \(WI\), or a temporally coherent perturbation operator for Dynamask \(DM\)\)\. Empirically they remain only partially reliable, their edge\-recovery accuracy still falls short of what DAG\-faithfulness demands, but their structure is the right one, and improving the fidelity of the conditional sampler and the choice of conditioning set is a promising route toward an admissible standard for explainability in time\-series forecasting\.

Table 3:Temporally\-aware perturbation methods\.Global scoreΦ1f\\Phi^\{f\}\_\{1\}forX\(1\)X^\{\(1\)\}under the three models; correct isΦ1=0\\Phi\_\{1\}=0forf1,f3f\_\{1\},f\_\{3\}andΦ1\>0\\Phi\_\{1\}\>0forf2f\_\{2\}\. Unlike the marginal and gradient families \(Tables[1](https://arxiv.org/html/2607.16236#S3.T1)–[2](https://arxiv.org/html/2607.16236#S3.T2)\), WI \(WinIT\), DM \(Dynamask\) and FIT do not hallucinate the mediated edge underf1f\_\{1\}or the off\-manifold edge underf3f\_\{3\}: by replacing removed features with conditional, in\-distribution samples they partially respect both admissibility conditions\. Their weakness is the opposite one, low sensitivity to the*genuine*direct edge: WI and FIT score0underf2f\_\{2\}\(a missed true edge, a false negative\), and DM detects it only weakly \(0\.010\.01\)\. These methods thus err toward under\-attribution rather than the spurious over\-attribution of the other families, which is why we regard them as the most promising starting point for an admissible instantiation\.

### 4\.2The Way Forward

The failure identified in Section[3](https://arxiv.org/html/2607.16236#S3)is, at its root,*representational*: a single scalar per feature cannot record*which*other variables a dependency is conditioned on\. The minimal object that can is a tensor indexed by source variable, target variable, and lag\.

###### Definition 4\(Transition Tensor\)\.

A*transition tensor*for a modelffat input𝐱\\mathbf\{x\}is a three\-dimensional array

𝐓loc​\(𝐱\)∈ℝD×D×T,\\mathbf\{T\}^\{\\mathrm\{loc\}\}\(\\mathbf\{x\}\)\\in\\mathbb\{R\}^\{D\\times D\\times T\},whose entry𝐓i​jloc\(𝐱\)\(t1\)\\mathbf\{T\}^\{\\mathrm\{loc\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\)\}\(\\mathbf\{x\}\)quantifies the influence ofXi\(t1\)X\_\{i\}^\{\(t\_\{1\}\)\}onf​\(𝐱\)jf\(\\mathbf\{x\}\)\_\{j\}at the forecast target, for input positiont1∈\[T\]t\_\{1\}\\in\[T\]\. The forecast time is fixed atT\+1T\+1, so a single time axis \(the source positiont1t\_\{1\}\) suffices\. WhenW\>1W\>1, we haveWWsuch containers, that is,

𝐓loc​\(𝐱\)∈ℝD×D×T×W,\\mathbf\{T\}^\{\\mathrm\{loc\}\}\(\\mathbf\{x\}\)\\in\\mathbb\{R\}^\{D\\times D\\times T\\times W\},and entry𝐓i​jloc\(𝐱\)\(t1\),w\\mathbf\{T\}^\{\\mathrm\{loc\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\),w\}\(\\mathbf\{x\}\)quantifies the influence ofXi\(t1\)X\_\{i\}^\{\(t\_\{1\}\)\}on forecast atf​\(𝐱\)jf\(\\mathbf\{x\}\)\_\{j\}at forecast timeT\+wT\+w,w∈\[W\]w\\in\[W\]\. A*Global Transition Tensor*summarizes the model’s behavior across the reference distribution, so we aggregate the per\-instance tensors\.

𝐓glob=𝔼𝐱∼p​\(𝐱\)​\[𝐓loc​\(𝐱\)\]\\mathbf\{T\}^\{\\mathrm\{glob\}\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim p\(\\mathbf\{x\}\)\}\\left\[\\mathbf\{T\}^\{\\mathrm\{loc\}\}\(\\mathbf\{x\}\)\\right\]wherep​\(𝐱\)p\(\\mathbf\{x\}\)is the reference distribution\. In practice, this is estimated as

𝐓glob^=1N​∑n=1N𝐓loc​\(𝐱\(n\)\)\\hat\{\\mathbf\{T\}^\{\\mathrm\{glob\}\}\}=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{T\}^\{\\mathrm\{loc\}\}\(\\mathbf\{x\}^\{\(n\)\}\)over an evaluation set ofNNsamples drawn frompp\. The DAG structure of the global explanation is read from the*support*of𝐓glob\\mathbf\{T\}^\{\\mathrm\{glob\}\}, not its magnitude: an edgeXi\(t1\)→Xj\(T\+1\)X\_\{i\}^\{\(t\_\{1\}\)\}\\to X\_\{j\}^\{\(T\+1\)\}is present in the induced graph𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}if and only if\|𝐓i​jglob\|\(t1\)\>ϵ\|\\mathbf\{T\}^\{\\mathrm\{glob\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\)\}\|\>\\epsilonfor some thresholdϵ\>0\\epsilon\>0\. This cleanly separates*structural*information \(which dependencies exist, global and discrete\) from*strength*information \(how strong each dependency is, which may vary per instance\)\. Figure[5](https://arxiv.org/html/2607.16236#S4.F5)summarizes the pipeline from local tensors to the induced graph\.

The tensor is only a*structural container*: its shape provides the minimal capacity needed to encode pairwise temporal dependencies, but the choice of influence functional𝐓i​jloc\(⋅\)\(t1\)\\mathbf\{T\}^\{\\mathrm\{loc\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\)\}\(\\cdot\)determines whether a given instantiation is DAG\-faithful \(Definition[3](https://arxiv.org/html/2607.16236#Thmdefinition3)\)\.

###### Definition 5\(Admissible Instantiation\)\.

An influence functionalϕ\\phiis*admissible*if its resulting transition tensor satisfies two conditions:

1. 1\.On\-manifold evaluation:ϕ\\phidepends onffonly through its behavior onsupp​\(p\)\\mathrm\{supp\}\(p\), so that models agreeing onsupp​\(p\)\\mathrm\{supp\}\(p\)receive identical tensors\.
2. 2\.Conditional separation:the entry𝐓i​jglob\(t1\)\\mathbf\{T\}^\{\\mathrm\{glob\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\)\}vanishes wheneverXi\(t1\)⟂⟂f\(𝐗\)j∣𝐗∖\{Xi\(t1\)\}X\_\{i\}^\{\(t\_\{1\}\)\}\\perp\\\!\\\!\\\!\\perp f\(\\mathbf\{X\}\)\_\{j\}\\mid\\mathbf\{X\}\\setminus\\\{X\_\{i\}^\{\(t\_\{1\}\)\}\\\}underp​\(𝐱\)p\(\\mathbf\{x\}\)\.

![Refer to caption](https://arxiv.org/html/2607.16236v1/x6.png)Figure 5:From local influence to a global dependency graph\.Instance\-level transition tensors𝐓loc​\(𝐱\(n\)\)\\mathbf\{T\}^\{\\mathrm\{loc\}\}\(\\mathbf\{x\}^\{\(n\)\}\)\(Definition[4](https://arxiv.org/html/2607.16236#Thmdefinition4)\), indexed by source variableii, target variablejj, and source lagt1t\_\{1\}, are averaged over the reference distribution into the global tensor𝐓glob\\mathbf\{T\}^\{\\mathrm\{glob\}\}\. The induced graph𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}\(Definition[3](https://arxiv.org/html/2607.16236#Thmdefinition3)\) is read from the*support*of𝐓glob\\mathbf\{T\}^\{\\mathrm\{glob\}\}: an edge is present iff\|𝐓i​jglob\|\(t1\)\>ϵ\|\\mathbf\{T\}^\{\\mathrm\{glob\}\}\_\{ij\}\{\}^\{\(t\_\{1\}\)\}\|\>\\epsilon, separating which dependencies exist from how strong each one is\.This motivates the position we advocate:*a global explanation method for time\-series forecasting is acceptable only if it is admissible in the sense of Definition[5](https://arxiv.org/html/2607.16236#Thmdefinition5)and expresses its output as a transition tensor\.*The two conditions are not meant to be verified as external metrics after the fact, that is intractable in general on real world data, but to be built into the influence functional*by construction*\.

##### Recommendation\.

A concrete admissible functional must combine two ingredients, one per condition:

- •\(C1\) On\-manifold evaluation, approached through manifold\-aware Jacobian entries \(directional derivatives\), integrated gradients along the temporal axis, or attention\-flow measuresAbnar and Zuidema \([2020](https://arxiv.org/html/2607.16236#bib.bib9)\);
- •\(C2\) Conditional separation, approached through conditional\-distribution\-shift measures such as those underlying FIT and WinIT, but conditioning on the*full*set of remaining inputs rather than a single variable\.

The main obstacle lies in \(C2\): the conditioning set is of order𝒪​\(D​T\)\\mathcal\{O\}\(DT\)\. This can be reduced by replacing the full complement with an estimated Markov blanket of the source variable, by discretizing the conditioning variables into bins, or by running the tests in the state space of a lower\-order surrogate that approximatesff\.

### 4\.3Evaluation Protocol

So far DAG\-faithfulness has been an abstract criterion\. We now make it*observable*, a quantity that can be measured for an existing method or used to test a new one\. The key is that the global criterion decomposes into independent per\-edge tests\.

###### Proposition 3\(Edge\-local characterization of DAG\-faithfulness\)\.

DAG\-faithfulness \(Definition[3](https://arxiv.org/html/2607.16236#Thmdefinition3)\) admits an edge\-local characterization that does not require recovering𝒢f\\mathcal\{G\}\_\{f\}in advance\. For each candidate edge, the edge condition of Definition[2](https://arxiv.org/html/2607.16236#Thmdefinition2)reduces to a conditional\-independence test between the source input and the model’s forecast, which can be estimated from model queries and held\-out data via kernel\-based or neural conditional\-independence estimators\.

The consequence is practical: since each edge is an independent conditional\-independence test, an explanation’s induced graph𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}can be scored against𝒢f\\mathcal\{G\}\_\{f\}edge by edge, without ever materializing𝒢f\\mathcal\{G\}\_\{f\}in full\. We outline the resulting protocol in three settings of decreasing access to ground truth\.

##### Synthetic benchmarks\.

Generate data from a known dynamic Bayesian network with a fixed ground\-truth DAG𝒢∗\\mathcal\{G\}^\{\*\}, and train a temporal modelffon it\. Under our model\-centric framing the target of recovery is𝒢f\\mathcal\{G\}\_\{f\}\(what the model learned\), not𝒢∗\\mathcal\{G\}^\{\*\}\(what generated the data\)\. Whenffis analytically characterizable e\.g\. a linear VAR with known coefficients𝒢f\\mathcal\{G\}\_\{f\}is known by construction, and the*structural Hamming distance*between𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}and𝒢f\\mathcal\{G\}\_\{f\}is a direct scalar measure of faithfulness\. When𝒢f\\mathcal\{G\}\_\{f\}is not analytically accessible,𝒢∗\\mathcal\{G\}^\{\*\}serves as a proxy, with the caveat that recovery error then conflates model\-learning error with explanation error\.

##### Flat\-method baselines\.

On the same benchmark, any flat attribution method is evaluated by thresholding its scores into a binary edge mask and comparing that mask to the skeleton of𝒢f\\mathcal\{G\}\_\{f\}\. This places flat and tensor\-based methods on a common footing and makes the representational gap of Section[3](https://arxiv.org/html/2607.16236#S3)concrete and measurable\.

##### Real\-data evaluation\.

When𝒢∗\\mathcal\{G\}^\{\*\}is unknown, faithfulness is assessed indirectly through*interventional consistency*: if the explanation marks an edgeXi\(t1\)→Xj\(T\+1\)X\_\{i\}^\{\(t\_\{1\}\)\}\\to X\_\{j\}^\{\(T\+1\)\}as absent, then intervening onXi\(t1\)X\_\{i\}^\{\(t\_\{1\}\)\}should leaveff’s forecast forXjX\_\{j\}unchanged once the rest of the window is held fixed\. This links the criterion to model debugging and counterfactual evaluationGoyalet al\.\([2019](https://arxiv.org/html/2607.16236#bib.bib26)\)\.

## 5Analysis of Standard Methods

### 5\.1Standard Methods Fail DAG\-Faithfulness

###### Theorem 1\(SHAP is not DAG\-faithful in general\)\.

There exists a modelffand reference distributionp​\(𝐱\)p\(\\mathbf\{x\}\)such that the global SHAP explanation is not DAG\-faithful\.

###### Proof sketch\.

Consider a linear model withD=2D=2,T=2T=2implementing the chainX1\(1\)→X1\(2\)→Y\(3\)X\_\{1\}^\{\(1\)\}\\to X\_\{1\}^\{\(2\)\}\\to Y^\{\(3\)\}, whereY\(3\)=α​X1\(2\)Y^\{\(3\)\}=\\alpha X\_\{1\}^\{\(2\)\}and the data\-generating process satisfiesX1\(2\)=δ​X1\(1\)\+εX\_\{1\}^\{\(2\)\}=\\delta X\_\{1\}^\{\(1\)\}\+\\varepsilonwithε\\varepsilonindependent ofX1\(1\)X\_\{1\}^\{\(1\)\}\. The model dependency graph𝒢f\\mathcal\{G\}\_\{f\}contains the direct edgeX1\(2\)→Y\(3\)X\_\{1\}^\{\(2\)\}\\to Y^\{\(3\)\}but notX1\(1\)→Y\(3\)X\_\{1\}^\{\(1\)\}\\to Y^\{\(3\)\}: conditional onX1\(2\)X\_\{1\}^\{\(2\)\},X1\(1\)X\_\{1\}^\{\(1\)\}is independent ofY\(3\)Y^\{\(3\)\}underpp\.

SHAP values \(either marginal or conditional variant\) assign a nonzero Shapley value toX1\(1\)X\_\{1\}^\{\(1\)\}whenever its coalition contributions shift the conditional expectation offf, which they do via the autocorrelation structure above\. When the induced graph𝒢𝒯SHAP\\mathcal\{G\}\_\{\\mathcal\{T\}^\{\\mathrm\{SHAP\}\}\}is defined by thresholding Shapley values, it contains the spurious edgeX1\(1\)→Y\(3\)X\_\{1\}^\{\(1\)\}\\to Y^\{\(3\)\}that is absent from𝒢f\\mathcal\{G\}\_\{f\}\. Hence𝒢𝒯SHAP\\mathcal\{G\}\_\{\\mathcal\{T\}^\{\\mathrm\{SHAP\}\}\}and𝒢f\\mathcal\{G\}\_\{f\}are not Markov\-equivalent: they disagree on the d\-separationX1\(1\)⟂⟂Y\(3\)∣X1\(2\)X\_\{1\}^\{\(1\)\}\\perp\\\!\\\!\\\!\\perp Y^\{\(3\)\}\\mid X\_\{1\}^\{\(2\)\}, which holds in𝒢f\\mathcal\{G\}\_\{f\}but fails in𝒢𝒯SHAP\\mathcal\{G\}\_\{\\mathcal\{T\}^\{\\mathrm\{SHAP\}\}\}\. The same argument applies to TimeSHAPBentoet al\.\([2021](https://arxiv.org/html/2607.16236#bib.bib24)\), which inherits the coalition structure of SHAP on sequence models\. ∎

The same argument applies to gradient saliency \(which computes marginal partial derivatives, collapsing mediated paths\) and attention weights \(which are not formally tied to conditional independenceJain and Wallace \([2019](https://arxiv.org/html/2607.16236#bib.bib7)\); Wiegreffe and Pinter \([2019](https://arxiv.org/html/2607.16236#bib.bib8)\)\)\.

## 6Discussion and Open Challenges

##### Instantiating the transition tensor\.

Definition[4](https://arxiv.org/html/2607.16236#Thmdefinition4)specifies the structural container; Definition[5](https://arxiv.org/html/2607.16236#Thmdefinition5)specifies the two conditions any DAG\-faithful instantiation must satisfy\. Demonstrating a concrete admissible instantiation that achieves DAG\-faithfulness in general remains the central open problem\. The failure modes \(C1\) and \(C2\) identified rule out naive Jacobian\-based instantiations \(which violate on\-manifold evaluation\) and marginal perturbation\-based instantiations \(which violate conditional separation\)\. Candidate directions include local conditional\-independence estimators on the model’s predictive distribution and low\-rank factorizations that exploit temporal structure to reduce theO​\(D2​T\)O\(D^\{2\}T\)sample complexity of the full tensor\.

##### Joint Window Forecasting

One of the main pain points in XAI for time series forecasting, and an issue we stated but did not address in this text, was the potential joint predictive behavior ofW\>1W\>1forecasting models\. Indeed, when discussing the model dependency graph, we focused on singular past time steps to forecast step edges and omitted intra\-forecast\-window dependency edges, a point that is also very valuable in the context of temporal\-structure dependency\-based XAI methods\. Possible future directions include the addition of causal discovery methods for edge recovery in the forecast window, or methods leveraging the idea of Markov Chains, to understand joint predictive behavior\.

##### Robustness to distributional shift\.

DAG\-faithfulness is defined relative to a reference distributionpp\. Whether the induced graph𝒢𝒯\\mathcal\{G\}\_\{\\mathcal\{T\}\}is invariant under reasonable variations inppis an open question and a natural target for theoretical analysis\.

##### Identifiability\.

We require Markov equivalence rather than exact DAG recovery, since the latter is not identifiable from observational data in general\. Strengthening exact recovery requires additional assumptions \(e\.g\. linear\-Gaussian models, or access to interventional data\), which are worth exploring as restricted settings with sharper guarantees\.

##### Human interpretability\.

AD×D×TD\\times D\\times Ttensor is not directly human\-readable\. Visualization strategies \(lag\-aggregated heatmaps, interactive graph browsers, dimensionality\-reduced summaries\) are needed to make the structure accessible\. We view this as a complementary challenge to the theoretical framework: the tensor is the right internal representation for faithfulness, but user\-facing explanations will require principled projections of it\.

## 7Conclusion

We have argued that faithful global explanations for time series models cannot be achieved using conventional flat attribution scoring methods\. We have defined*DAG\-faithfulness*as the appropriate criterion for global time series explanations, formalized the admissibility conditions that candidate instantiations must satisfy, shown that standard methods and their time\-series\-aware extensions fail these conditions by construction, and connected the framework to Granger causality and dynamic Bayesian networks\. We call on the XAI community to adopt a structure\-centric faithfulness criterion and the transition tensor representation as the foundation for explainability in temporal domains, and we identify the construction of admissible, DAG\-faithful instantiations as the central open problem for future work\.

## References

- S\. Abnar and W\. Zuidema \(2020\)Quantifying attention flow in transformers\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2607.16236#S4.I2.i1.p1.1)\.
- D\. Alvarez\-Melis and T\. S\. Jaakkola \(2018\)On the robustness of interpretability methods\.InICML Workshop on Human Interpretability in Machine Learning,Cited by:[§2\.3](https://arxiv.org/html/2607.16236#S2.SS3.p1.1)\.
- J\. Bento, P\. Saleiro, A\. F\. Cruz, M\. A\. Figueiredo, and P\. Bizarro \(2021\)TimeSHAP: explaining recurrent models through sequence perturbations\.InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p2.5),[§5\.1](https://arxiv.org/html/2607.16236#S5.SS1.2.p2.10)\.
- J\. Crabbé and M\. van der Schaar \(2021\)Explaining time series predictions with dynamic masks\.InInternational Conference on Machine Learning,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p2.5)\.
- A\. Geiger, H\. Lu, T\. Icard, and C\. Potts \(2021\)Causal abstractions of neural networks\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§2\.3](https://arxiv.org/html/2607.16236#S2.SS3.p1.1)\.
- W\. Gong, J\. Jennings, C\. Zhang, and N\. Pawlowski \(2023\)Rhino: deep causal temporal relationship learning with history\-dependent noise\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- Y\. Goyal, Z\. Wu, J\. Ernst, D\. Batra, D\. Parikh, and S\. Lee \(2019\)Counterfactual visual explanations\.InInternational Conference on Machine Learning,pp\. 2376–2384\.Cited by:[§4\.3](https://arxiv.org/html/2607.16236#S4.SS3.SSS0.Px3.p1.5)\.
- C\. W\. Granger \(1969\)Investigating causal relations by econometric models and cross\-spectral methods\.Econometrica37\(3\),pp\. 424–438\.Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- A\. Jacovi and Y\. Goldberg \(2020\)Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2\.3](https://arxiv.org/html/2607.16236#S2.SS3.p1.1)\.
- S\. Jain and B\. C\. Wallace \(2019\)Attention is not explanation\.InProceedings of NAACL\-HLT,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16236#S5.SS1.p1.1)\.
- P\. Kindermans, S\. Hooker, J\. Adebayo, M\. Alber, K\. T\. Schütt, S\. Dähne, D\. Erhan, and B\. Kim \(2019\)The \(un\)reliability of saliency methods\.InExplainable AI: Interpreting, Explaining and Visualizing Deep Learning,pp\. 267–280\.Cited by:[§2\.3](https://arxiv.org/html/2607.16236#S2.SS3.p1.1)\.
- K\. K\. Leung, C\. Rooke, J\. Smith, S\. Zuberi, and M\. Volkovs \(2023\)Temporal dependencies in feature importance for time series prediction\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p2.5)\.
- S\. M\. Lundberg and S\. Lee \(2017\)A unified approach to interpreting model predictions\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1)\.
- K\. P\. Murphy \(2002\)Dynamic bayesian networks: representation, inference and learning\.Ph\.D\. Thesis,University of California, Berkeley\.Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- R\. Pamfil, N\. Sriwattanaworachai, S\. Desai, P\. Pilgerstorfer, K\. Georgatzis, P\. Beaumont, and B\. Aragam \(2020\)DYNOTEARS: structure learning from time\-series data\.InInternational Conference on Artificial Intelligence and Statistics,Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- J\. Pearl \(2009\)Causality: models, reasoning, and inference\.2nd edition,Cambridge University Press\.Cited by:[footnote 1](https://arxiv.org/html/2607.16236#footnote1)\.
- J\. Peters, D\. Janzing, and B\. Schölkopf \(2017\)Elements of causal inference: foundations and learning algorithms\.MIT Press\.Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- M\. T\. Ribeiro, S\. Singh, and C\. Guestrin \(2016\)“Why should i trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1135–1144\.Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1)\.
- J\. Runge \(2020\)Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets\.Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence\.Cited by:[§2\.2](https://arxiv.org/html/2607.16236#S2.SS2.p1.5)\.
- W\. Samek, A\. Binder, G\. Montavon, S\. Lapuschkin, and K\. Müller \(2016\)Evaluating the visualization of what a deep neural network has learned\.IEEE Transactions on Neural Networks and Learning Systems28,pp\. 2660–2673\.Cited by:[§2\.3](https://arxiv.org/html/2607.16236#S2.SS3.p1.1)\.
- K\. Simonyan, A\. Vedaldi, and A\. Zisserman \(2014\)Deep inside convolutional networks: visualising image classification models and saliency maps\.InInternational Conference on Learning Representations Workshop,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1)\.
- M\. Sundararajan, A\. Taly, and Q\. Yan \(2017\)Axiomatic attribution for deep networks\.InInternational Conference on Machine Learning,pp\. 3319–3328\.Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1)\.
- S\. Tonekaboni, S\. Joshi, K\. Campbell, D\. Duvenaud, and A\. Goldenberg \(2020\)What went wrong and when? instance\-wise feature importance for time\-series black\-box models\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p2.5)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1)\.
- S\. Wiegreffe and Y\. Pinter \(2019\)Attention is not not explanation\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.16236#S5.SS1.p1.1)\.
- Y\. Zhang, Q\. Sun, D\. Qi, J\. Liu, R\. Ma, and O\. L\. Petrosian \(2023\)ShapTime: a general xai approach for explainable time series forecasting\.InIntelligent Systems and Applications \- Proceedings of the 2023 Intelligent Systems Conference \(IntelliSys\), Amsterdam, The Netherlands, 7\-8 September, 2023, Volume 1,K\. Arai \(Ed\.\),Lecture Notes in Networks and Systems, Vol\.822,pp\. 659–673\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-47721-8%5F45),[Link](https://doi.org/10.1007/978-3-031-47721-8_45),ISBN 978\-3\-031\-47721\-8Cited by:[§2\.1](https://arxiv.org/html/2607.16236#S2.SS1.p2.5)\.

## Appendix AAppendix

###### Proposition 4\(Computational Insufficiency of Flat Attribution Methods\)\.

Flat attribution methods cannot satisfy both admissibility conditions \(C1\) and \(C2\) in general\. Specifically:

1. 1\.Any flat attribution method whose global score reduces to marginal conditioning over a single input \(i\.e\., conditioning only onX\(t1\)X^\{\(t\_\{1\}\)\}and not on the remaining inputs\) violates \(C2\): there exist modelsf1,f2f\_\{1\},f\_\{2\}with non\-isomorphic dependency graphs𝒢f1≇𝒢f2\\mathcal\{G\}\_\{f\_\{1\}\}\\not\\cong\\mathcal\{G\}\_\{f\_\{2\}\}that the method assigns identical global attributions so that no threshold can separate them\.
2. 2\.Any gradient\-based method that evaluates∂f/∂x\\partial f/\\partial xas a function onℝT×D\\mathbb\{R\}^\{T\\times D\}violates \(C1\): there exist modelsf1,f3f\_\{1\},f\_\{3\}with𝒢f1=𝒢f3\\mathcal\{G\}\_\{f\_\{1\}\}=\\mathcal\{G\}\_\{f\_\{3\}\}that the method assigns different attributions, since the gradient registers off\-manifold sensitivity that is not part of the model’s on\-manifold dependency structure\.

A DAG\-faithful explanation, therefore, requires an attribution functional satisfying both \(C1\) and \(C2\), properties that no existing flat attribution scoring method can assure\. The transition tensor \(Definition[4](https://arxiv.org/html/2607.16236#Thmdefinition4)\) provides the minimal representational container in which both conditions can be expressed\.

###### Proof\.

We prove the two parts via independent constructions\.

##### Part 1: violation of \(C2\) by marginal\-conditioning methods\.

Assume Scenario I

The expected marginal influence

Φt1f=𝔼p\[\|𝔼p\[f\(𝐗\)∣X\(t1\)\]−𝔼p\[f\(𝐗\)\]\|\]\\Phi^\{f\}\_\{t\_\{1\}\}=\\mathbb\{E\}\_\{p\}\\Bigl\[\\bigl\|\\,\\mathbb\{E\}\_\{p\}\\bigl\[f\(\\mathbf\{X\}\)\\mid X^\{\(t\_\{1\}\)\}\\bigr\]\-\\mathbb\{E\}\_\{p\}\\bigl\[f\(\\mathbf\{X\}\)\\bigr\]\\,\\bigr\|\\Bigr\]\(5\)is the natural global characterization of any method that conditions only onX\(t1\)X^\{\(t\_\{1\}\)\}\. ComputingΦ1f1\\Phi^\{f\_\{1\}\}\_\{1\}via \([1](https://arxiv.org/html/2607.16236#S3.E1)\):

𝔼p​\[f1​\(𝐗\)∣X\(1\)=x\]−𝔼p​\[f1​\(𝐗\)\]\\displaystyle\\mathbb\{E\}\_\{p\}\\bigl\[f\_\{1\}\(\\mathbf\{X\}\)\\mid X^\{\(1\)\}=x\\bigr\]\-\\mathbb\{E\}\_\{p\}\\bigl\[f\_\{1\}\(\\mathbf\{X\}\)\\bigr\]=β​𝔼p​\[X\(2\)∣X\(1\)=x\]−β​𝔼p​\[X\(2\)\]\\displaystyle\\quad=\\beta\\,\\mathbb\{E\}\_\{p\}\\bigl\[X^\{\(2\)\}\\mid X^\{\(1\)\}=x\\bigr\]\-\\beta\\,\\mathbb\{E\}\_\{p\}\\bigl\[X^\{\(2\)\}\\bigr\]=β​\(δ​x\+𝔼p​\[ε\]\)−β​𝔼p​\[δ​X\(1\)\+ε\]\\displaystyle\\quad=\\beta\\,\(\\delta x\+\\mathbb\{E\}\_\{p\}\[\\varepsilon\]\)\-\\beta\\,\\mathbb\{E\}\_\{p\}\\bigl\[\\delta X^\{\(1\)\}\+\\varepsilon\\bigr\]=β​δ​x−β​δ​𝔼p​\[X\(1\)\]\\displaystyle\\quad=\\beta\\delta\\,x\-\\beta\\delta\\,\\mathbb\{E\}\_\{p\}\\bigl\[X^\{\(1\)\}\\bigr\]=β​δ​\(x−𝔼p​\[X\(1\)\]\),\\displaystyle\\quad=\\beta\\delta\\,\\bigl\(x\-\\mathbb\{E\}\_\{p\}\[X^\{\(1\)\}\]\\bigr\),where we used𝔼p​\[X\(2\)∣X\(1\)=x\]=δ​x\\mathbb\{E\}\_\{p\}\[X^\{\(2\)\}\\mid X^\{\(1\)\}=x\]=\\delta xfrom \([1](https://arxiv.org/html/2607.16236#S3.E1)\) and𝔼​\[ε\]=0\\mathbb\{E\}\[\\varepsilon\]=0\. Taking the expectation overX\(1\)X^\{\(1\)\}:

Φ1f1\\displaystyle\\Phi^\{f\_\{1\}\}\_\{1\}=𝔼p​\[\|β​δ\|​\|X\(1\)−𝔼p​\[X\(1\)\]\|\]\\displaystyle=\\mathbb\{E\}\_\{p\}\\Bigl\[\|\\beta\\delta\|\\,\\bigl\|X^\{\(1\)\}\-\\mathbb\{E\}\_\{p\}\[X^\{\(1\)\}\]\\bigr\|\\Bigr\]=\|β​δ\|​MAD​\(X\(1\)\)\>0,\\displaystyle=\|\\beta\\delta\|\\,\\mathrm\{MAD\}\(X^\{\(1\)\}\)\>0,whereMAD\\mathrm\{MAD\}denotes the mean absolute deviation, which is strictly positive sinceVar​\(X\(1\)\)\>0\\mathrm\{Var\}\(X^\{\(1\)\}\)\>0by assumption, andβ​δ≠0\\beta\\delta\\neq 0\.Φ1f1\>0\\Phi^\{f\_\{1\}\}\_\{1\}\>0even though the edgeX\(1\)→X\(3\)X^\{\(1\)\}\\to X^\{\(3\)\}is absent from𝒢f1\\mathcal\{G\}\_\{f\_\{1\}\}\. The spurious attribution arises from temporal autocorrelation: marginal conditioning cannot separate the indirect pathX\(1\)→X\(2\)→f1X^\{\(1\)\}\\to X^\{\(2\)\}\\to f\_\{1\}from a direct effect, which is exactly a violation of \(C2\)\. Similarly, computingΦ1f2\\Phi^\{f\_\{2\}\}\_\{1\}:

Φ1f2=\|β​δ\+γ\|​MAD​\(X\(1\)\)\.\\Phi^\{f\_\{2\}\}\_\{1\}=\|\\beta\\delta\+\\gamma\|\\,\\mathrm\{MAD\}\(X^\{\(1\)\}\)\.Settingγ=−2​β​δ\\gamma=\-2\\beta\\deltayieldsΦ1f1=Φ1f2\\Phi^\{f\_\{1\}\}\_\{1\}=\\Phi^\{f\_\{2\}\}\_\{1\}\. The collapse propagates to any method whose global score is bounded below by a positive multiple ofΦt1f\\Phi^\{f\}\_\{t\_\{1\}\}, including SHAP, LIME, and the temporally\-aware perturbation methods \(TimeSHAP, ShapTime\) that condition perturbations on coalitions or windows of features rather than on the conditional distribution given the mediator\.

##### Part 2: violation of \(C1\) by gradient methods\.

Gradient\-based methods probeffvia partial derivatives at points in input space\. We use a separate construction in which the data manifold is low\-dimensional, so off\-manifold directions are unambiguous\.

Assume Scenario II,

The partial derivative

∂f3∂x\(1\)=−γ​δ≠0\\frac\{\\partial f\_\{3\}\}\{\\partial x^\{\(1\)\}\}=\-\\gamma\\delta\\neq 0is nonzero everywhere, including at every𝐱∈supp​\(p\)\\mathbf\{x\}\\in\\mathrm\{supp\}\(p\)\. Gradient\-based methods such as vanilla saliency, integrated gradients, and SmoothGrad assign nonzero attribution toX\(1\)X^\{\(1\)\}underf3f\_\{3\}despite the absence of an edge\. This is a violation of \(C1\): the method’s score depends onff’s extension offsupp​\(p\)\\mathrm\{supp\}\(p\), registering sensitivity in directions orthogonal to the manifold that play no role in the model’s on\-manifold dependency structure\.

∎