Software Frameworks for Explainable AI in Time Series Classification: A Systematic Review

# Software Frameworks for Explainable AI in Time Series Classification: A Systematic Review
Source: [https://arxiv.org/html/2608.21449](https://arxiv.org/html/2608.21449)
Louis PeterOrcID:[0009\-0001\-7478\-1888](https://orcid.org/0009-0001-7478-1888)Affiliation:Technische Hochschule Mittelhessen \- University of Applied Science, Friedberg, GermanyAffiliation:Hessian Center for AI \(hessian\.AI\), Darmstadt, GermanyE\-mail[\{louis\.peter,nils\.gumpfer,jana\.fischer,jennifer\.hannig\}@kite\.thm\.de](mailto:{louis.peter,nils.gumpfer,jana.fischer,jennifer.hannig}@kite.thm.de)Nils GumpferOrcID:[0000\-0001\-8644\-9885](https://orcid.org/0000-0001-8644-9885)Affiliation:Technische Hochschule Mittelhessen \- University of Applied Science, Friedberg, GermanyAffiliation:Hessian Center for AI \(hessian\.AI\), Darmstadt, GermanyE\-mail[\{louis\.peter,nils\.gumpfer,jana\.fischer,jennifer\.hannig\}@kite\.thm\.de](mailto:{louis.peter,nils.gumpfer,jana.fischer,jennifer.hannig}@kite.thm.de)Jana FischerOrcID:[0009\-0005\-0445\-0847](https://orcid.org/0009-0005-0445-0847)Affiliation:Technische Hochschule Mittelhessen \- University of Applied Science, Friedberg, GermanyAffiliation:Hessian Center for AI \(hessian\.AI\), Darmstadt, GermanyE\-mail[\{louis\.peter,nils\.gumpfer,jana\.fischer,jennifer\.hannig\}@kite\.thm\.de](mailto:{louis.peter,nils.gumpfer,jana.fischer,jennifer.hannig}@kite.thm.de)Christin SeifertOrcID:[0000\-0002\-6776\-3868](https://orcid.org/0000-0002-6776-3868)Affiliation:Hessian Center for AI \(hessian\.AI\), Darmstadt, GermanyE\-mail[\{louis\.peter,nils\.gumpfer,jana\.fischer,jennifer\.hannig\}@kite\.thm\.de](mailto:{louis.peter,nils.gumpfer,jana.fischer,jennifer.hannig}@kite.thm.de)Affiliation:Marburg University, Marburg, GermanyE\-mail[christin\.seifert@uni\-marburg\.de](mailto:[email protected])Jennifer Hannig \(✉\)OrcID:[0000\-0002\-2789\-5540](https://orcid.org/0000-0002-2789-5540)Affiliation:Technische Hochschule Mittelhessen \- University of Applied Science, Friedberg, GermanyAffiliation:Hessian Center for AI \(hessian\.AI\), Darmstadt, GermanyE\-mail[\{louis\.peter,nils\.gumpfer,jana\.fischer,jennifer\.hannig\}@kite\.thm\.de](mailto:{louis.peter,nils.gumpfer,jana.fischer,jennifer.hannig}@kite.thm.de)

###### Abstract

Time series arise in a wide range of application domains and are analyzed using machine learning in decision\-critical settings\. Time series classification \(TSC\) is one of the most widely studied and relevant tasks\. In this context, ensuring the transparency and trustworthiness of TSC models has become an important requirement, motivating the use of explainable artificial intelligence \(XAI\) methods\. Despite growing interest, research on XAI for TSC remains fragmented, and a systematic understanding of the available software frameworks for explanation generation, their evaluation practices, and practical limitations is still lacking\. Prior work largely focused on individual explanation methods, while cross\-framework consistency, time\-series\-specific evaluation, and reproducibility have received little attention\. In this survey, we analyze existing software frameworks for explanation generation and evaluation in TSC\. We compare them along multiple dimensions, including supported XAI methods, evaluation metrics, usability, benchmarking support, and reproducibility, providing the first time\-series\-specific survey of frameworks with implementation comparisons and an analysis of frequency\-domain support\. We identify six frameworks that explicitly support time series and reveal common limitations: only one method supports frequency\-domain explanations despite their relevance; only two evaluation metrics have been developed specifically for time series; and identical XAI methods can yield substantially different explanations across frameworks\. Based on these findings, we discuss open challenges and outline directions for future research, highlighting the need for unified, time\-series\-specific XAI frameworks that enable faithful, reproducible, and time\-series\-aware explanations\.

###### Keywords:

Explainable Artificial Intelligence Time Series Classification XAI Evaluation XAI Software Frameworks

## 1Introduction

Time series data are a central part of many real\-world applications in healthcare, industrial monitoring, finance, and audio processing\. The increasing use of deep learning models for time series classification \(TSC\) has led to substantial gains in predictive performance, but these improvements often come at the cost of interpretability and transparency\. In high\-risk domains, understanding and justifying automated decisions is critical, both to foster user trust and to comply with regulatory requirements such as the European Union’s Artificial Intelligence Act \(AI Act\), in particular Article 14, which requires human oversight to ensure that users can interpret and effectively oversee AI outputs\[[11](https://arxiv.org/html/2608.21449#bib.bib11)\]\. These regulatory developments further highlight the necessity of reliable and faithful explanation methods for TSC\.

Explainable artificial intelligence \(XAI\) has consequently emerged as an important research area\. However, most existing XAI research has been developed and evaluated primarily in the context of image data\[[26](https://arxiv.org/html/2608.21449#bib.bib26)\]\. As a result, many explanation methods are applied to time series without being specifically adapted or systematically evaluated\[[28](https://arxiv.org/html/2608.21449#bib.bib28)\], leaving their faithfulness and practical usefulness for time series data unclear\.

Explanations for time series are less intuitive than for other data modalities, increasing the risk of misleading interpretations\[[27](https://arxiv.org/html/2608.21449#bib.bib27)\]\. While image classification tasks usually allow even non\-experts to visually distinguish between classes such as cats and dogs, many real\-world time series lack a clear notion of interpretable components\. The raw time series signal often appears noisy and does not exhibit visually interpretable structure, even for domain experts\. A representative example is an audio signal of a spoken digit, which is not visually interpretable in the time domain \(see the left part of Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)\)\. In such cases, explanations based solely on the time domain are often insufficient, and alternative representations in the frequency and time\-frequency domains are usually considered\. This highlights the importance of incorporating these representations to generate meaningful explanations\.

Figure 1:The electrocardiogram \(ECG\) signal is a multivariate time series in which each channel corresponds to a distinct measurement location \(lead\)\. All channels reflect the same underlying cardiac process with strong temporal and synchronous cross\-channel dependencies\. An example with right bundle branch block \(RBBB\) shows characteristic patterns in leads V1 and V6, highlighting the need for multichannel analysis\.In contrast, the electrocardiogram \(ECG\) is a multivariate time series for which meaningful and clinically interpretable patterns are well defined and visible in the time domain\. Pathology\-related patterns can be reliably interpreted by domain experts, particularly cardiologists, and often arise from complex cross\-channel dependencies, reflecting spatial and temporal interactions of cardiac activity \(see Fig\.[1](https://arxiv.org/html/2608.21449#S1.F1)\)\. Consequently, explaining multivariate time series requires XAI methods that explicitly account for cross\-channel dependencies, which can be synchronous \(i\.e\., dependencies between channels occurring at the same time step\) or asynchronous\.

Overall, time series pose unique challenges for explainability\. Meaningful patterns often emerge from complex temporal dynamics and cross\-channel dependencies, including interactions across time, frequency, or time\-frequency domains\. As a result, XAI for TSC differs fundamentally from image\-based explanations and motivates the need for the systematic comparison, evaluation, and development of XAI methods tailored to the unique properties of time series\.

Prior surveys have identified key challenges of XAI for time series data\. Theissler et al\.\[[34](https://arxiv.org/html/2608.21449#bib.bib34)\]and Rojat et al\.\[[27](https://arxiv.org/html/2608.21449#bib.bib27)\]reviewed existing XAI methods and emphasized the need for systematic evaluation but focused primarily on methods rather than the frameworks111Throughout, we use “framework” to refer to a*software*framework, i\.e\., a library or tool for generating and/or evaluating explanations\.used to implement, evaluate, and benchmark them in practice\. Although multiple frameworks for evaluation exist, the survey by Le et al\.\[[24](https://arxiv.org/html/2608.21449#bib.bib24)\]showed that, as of 2022, only a single framework explicitly supported time series data\. At the same time, Theissler et al\.\[[34](https://arxiv.org/html/2608.21449#bib.bib34)\]called for unified frameworks to enable comparative and reproducible evaluation of XAI methods for time series\. It remains unclear how far this call has been addressed\. In particular, there is no consolidated understanding of how recent XAI frameworks support TSC, how they differ in explanation generation and evaluation, or how they align with emerging regulatory requirements\.

Addressing this gap motivates the present survey, which provides the first systematic review and comparative analysis of recent XAI frameworks for TSC\. In contrast to prior surveys that focus on XAI methods\[[34](https://arxiv.org/html/2608.21449#bib.bib34),[27](https://arxiv.org/html/2608.21449#bib.bib27)\]or analyze frameworks across different data types\[[24](https://arxiv.org/html/2608.21449#bib.bib24)\], we provide the first framework\-centric analysis specific to XAI for TSC\. We analyze how existing frameworks support the generation, evaluation, and benchmarking of explanations for TSC and assess their usability and practical limitations\.

1. RQ1:Which XAI frameworks currently support TSC, and how do they differ in usability and practical applicability? \(Sec\.[3\.1](https://arxiv.org/html/2608.21449#S3.SS1)\)
2. RQ2:How comprehensively do these frameworks support explanation methods and evaluation metrics, and to what extent do they support univariate and multivariate time series? \(Sec\.[3\.1](https://arxiv.org/html/2608.21449#S3.SS1)\)
3. RQ3:To what extent do current frameworks rely on reusing generic XAI methods and evaluation metrics, and how well do they support alternative signal representations? \(Sec\.[3\.2](https://arxiv.org/html/2608.21449#S3.SS2)\-[3\.3](https://arxiv.org/html/2608.21449#S3.SS3)\)
4. RQ4:How do existing frameworks support benchmarking of XAI methods for TSC, particularly with respect to dataset support and available ground\-truth information? \(Sec\.[3\.4](https://arxiv.org/html/2608.21449#S3.SS4)\)
5. RQ5:To what extent do differences in framework design and implementation affect the explanations and evaluation results produced by ostensibly identical XAI methods? \(Sec\.[3\.5](https://arxiv.org/html/2608.21449#S3.SS5)\)

By addressing these questions, this survey provides a consolidated view of the current landscape of XAI frameworks for TSC, highlights systematic strengths and shortcomings, and identifies open challenges for robust, comparable, and time\-series\-aware XAI frameworks\.

## 2Methodology

We performed a systematic review of available XAI frameworks for TSC\. In this work, an XAI framework denotes any software – such as library or tool – that facilitates either the comparative generation of explanations and/or the evaluation of XAI methods\. The review process is illustrated in Fig\.[2](https://arxiv.org/html/2608.21449#S2.F2)\. We searched GitHub, which is the dominant platform for hosting open\-source projects, using the following queries, resulting in 750 entries222The search was conducted on Dec 5th, 2025\.:"explainable\-ai time\-series","explainable\-ai evaluation","xai time\-series","xai evaluation", and"explanation time\-series"\.

After removing duplicates, 589 repositories remained\. To exclude unmaintained or incomplete repositories and ensure community relevance, we filtered repositories with fewer than four stars333GitHub stars represent the number of users who have marked a repository as a favorite\.and fewer than four forks444GitHub forks represent user copies of a repository to work on\.resulting in 115 repositories\. We then manually assessed the remaining repositories and excluded those that 1\.\) did not claim the support of time series in the corresponding publication, README file, or documentation,or2\.\) contained less than two XAI methods and less than two evaluation metrics,or3\.\) were not installable as a python package,or4\.\) had no associated publication or an insufficient README file555A README was regarded insufficient if it provided neither a tutorial nor documentation for usage\.\.

Figure 2:Flowchart of selection and review process\.We identified six candidate repositories\. We further examined the frameworks dependencies to identify additional frameworks, resembling a backward search for frameworks; however, all identified dependencies met at least one exclusion criterion\. As a result, the final survey included six XAI frameworks\. For each framework, we conducted a systematic review of the source code repositories, associated publications, documentation, supported datasets, and – where available – the original publications describing the implemented XAI methods and evaluation metrics\. The analysis was structured around the following dimensions:

#### Frameworks\.

For each framework, we record the number of XAI methods and evaluation metrics, the supported machine\-learning backends, whether it is available at a package index, and the corresponding GitHub repository metadata \(last update, stars, and forks\)\. To assess general usability, we adopted the usability scores proposed by Le et al\.\[[24](https://arxiv.org/html/2608.21449#bib.bib24)\], which evaluate the usability of frameworks along three dimensions:active maintenance, interaction with community, anddocumentation\. Each dimension is scored on a scale from 0 to 5 based on predefined criteria\.

#### XAI Methods\.

We categorized the XAI methods into counterfactual, gradient\-based, and perturbation\-based approaches; methods that did not fit these categories were grouped underother\. We examined whether methods explicitly claim to support time series\. For time\-series\-specific methods, we analyzed their applicability to univariate or multivariate time series and the domains in which explanations are generated \(time, frequency, or time\-frequency\)\. For methods implemented in multiple frameworks, we compared the resulting explanations to assess cross\-framework reproducibility\.

#### Evaluation Metrics\.

We investigated whether evaluation metrics are compatible with time series \(as explicitly stated\) or specific to time series \(i\.e\., originally developed for time series in the corresponding publication\)\. For metrics specific to time series, we analyzed the domains in which explanations can be evaluated \(time, frequency, or time\-frequency\)\. We categorized metrics into perturbation\-based and ground\-truth\-based metrics; metrics that did not fit these categories were grouped underother\. For metrics implemented in multiple frameworks, we compared the resulting scores to assess reproducibility across implementations\.

#### Benchmarking Capabilities\.

We assessed whether frameworks support benchmark datasets for comparing and evaluating XAI methods\. Datasets were considered if they can be accessed from the installed packages\. We further examined whether the datasets provide ground\-truth information \(known or expected reference explanations\) and whether they contain uni\- or multivariate time series\.

Figure 3:Dependency ecosystem of XAI frameworks \(arrows indicate dependencies\)\. White nodes support time series; black nodes are core dependencies without time series support\. For XAI method frameworks,on line, colback=black\!80, colframe=black, boxrule=0pt, arc=2pt,left=0pt, right=0pt, top=0pt, bottom=0pt\]Xindicates the number of time\-series\-specific XAI methods\. For XAI evaluation frameworks,on line, colback=black\!80, colframe=black, boxrule=0pt, arc=2pt,left=0pt, right=0pt, top=0pt, bottom=0pt\]X/Yindicates the number of time\-series\-specific XAI methods \(X\) and evaluation metrics \(Y\)\.

## 3Results

We present our findings along the five questions introduced above\. We first characterize the six frameworks and their usability \(Sec\.[3\.1](https://arxiv.org/html/2608.21449#S3.SS1)\), then analyze XAI methods \(Sec\.[3\.2](https://arxiv.org/html/2608.21449#S3.SS2)\), evaluation metrics \(Sec\.[3\.3](https://arxiv.org/html/2608.21449#S3.SS3)\), and benchmarking capabilities \(Sec\.[3\.4](https://arxiv.org/html/2608.21449#S3.SS4)\), and finally assess cross\-framework reproducibility \(Sec\.[3\.5](https://arxiv.org/html/2608.21449#S3.SS5)\)\.

### 3\.1Frameworks

Of the six identified frameworks, three support both explanation generation and evaluation by providing XAI evaluation metrics in addition to XAI methods \(time\_interpret\[[10](https://arxiv.org/html/2608.21449#bib.bib10)\], Quantus\[[17](https://arxiv.org/html/2608.21449#bib.bib17)\], and XTSC\-Bench\[[19](https://arxiv.org/html/2608.21449#bib.bib19)\]\), while the remaining three focus exclusively on XAI methods \(TSinterpret\[[18](https://arxiv.org/html/2608.21449#bib.bib18)\], tsCaptum\[[30](https://arxiv.org/html/2608.21449#bib.bib30)\], and SIGN\-XAI\-2\[[16](https://arxiv.org/html/2608.21449#bib.bib15)\]\)\. Table[1](https://arxiv.org/html/2608.21449#S3.T1)summarizes the frameworks and their main characteristics\. All analyzed frameworks support PyTorch\. TSInterpret has the broadest backend support, with native compatibility with PyTorch, scikit\-learn, and TensorFlow; XTSC\-Bench inherits these backends as it builds on TSInterpret\. Usability scores \(active maintenance, interaction with the community, and documentation\) are consistently high, with mean values of \(4\.3, 3\.6, 4\.6\) for XAI method frameworks and \(4, 3\.3, 5\) for XAI evaluation frameworks\. Quantus, which has the highest numbers of GitHub stars and forks, supports the largest set of XAI methods \(26\) by reusing or wrapping implementations from Captum\[[22](https://arxiv.org/html/2608.21449#bib.bib22)\], Zennit\[[1](https://arxiv.org/html/2608.21449#bib.bib1)\], and tf\-explain\[[25](https://arxiv.org/html/2608.21449#bib.bib25)\]\. However, none of these XAI methods are specific for time series \(see Fig\.[3](https://arxiv.org/html/2608.21449#S2.F3)\)\. Quantus also provides the largest collection of evaluation metrics \(36\)\. The largest set of time\-series\-specific XAI methods \(9, all of which support multivariate time series\) and metrics \(2\) is provided by time\_interpret\. SIGN\-XAI\-2 extends Zennit, while tsCaptum wraps Captum and applies chunking to reduce computational complexity, despite not providing time\-series\-specific methods\.

Table 1:Overview of the frameworks with a reference to the papers proposing frameworks or to the software release \(URLs are given in the reference entries\)\. : last update on GitHub; : number of stars on GitHub; : number of forks on GitHub; : availability local \(L\) or as installable package \(P\); : supported machine\-learning backend \(TensorFlow \(T\), PyTorch \(P\), and Scikit\-learn \(S\)\); : Usability scores in order of active maintenance, interaction with community, and documentation \(3 scores denoted as X–X–X, each in the range 0–5 from lowest to best\); : number of supported XAI methods \(implemented methods and, in parenthesis, wrapped or reused implementations of other frameworks\); : number of methods specific to univariate time series; : number of methods specific to multivariate time series;: number of methods generating explanations in the frequency domain; : number of supported XAI metrics \(implemented metrics and, in parenthesis, wrapped or reused implementations of other frameworks\); : number of metrics specific to time series\.XAI MethodsMetricsXAI Method FrameworksTSInterpret\[[18](https://arxiv.org/html/2608.21449#bib.bib18)\]11/202514418PT, P, S5\-4\-56 \+ \(10\)24000tsCaptum\[[30](https://arxiv.org/html/2608.21449#bib.bib30)\]11/2024101PP3\-2\-4\(5\)00000SIGN\-XAI\-2\[[16](https://arxiv.org/html/2608.21449#bib.bib15)\]12/202560PP5\-5\-52 \+ \(9\)10100XAI Evaluation FrameworksQuantus\[[17](https://arxiv.org/html/2608.21449#bib.bib17)\]07/202563183PT, P5\-4\-5\(26\)000360time\_interpret\[[10](https://arxiv.org/html/2608.21449#bib.bib10)\]09/20257110PP5\-4\-518090172XTSC\-Bench\[[19](https://arxiv.org/html/2608.21449#bib.bib19)\]10/202340LT, P, S2\-2\-5\(16\)\(2\)\(4\)0\(11\)0
### 3\.2XAI Methods

The frameworks contain 51 XAI methods, including 16 perturbation\-based, 25 gradient\-based, 4 counterfactual, and 6 other methods \(see Table[Appendix A](https://arxiv.org/html/2608.21449#Sx1)in the Appendix\)\. Only 16 of these methods explicitly state that they were developed for time series, although the remaining methods may still be applicable\. The majority of these 16 methods target multivariate time series, while only three are restricted to univariate time series \(DFT\-LRP\[[35](https://arxiv.org/html/2608.21449#bib.bib35)\], LEFTIST\[[14](https://arxiv.org/html/2608.21449#bib.bib14)\], and NUN\-CF\[[9](https://arxiv.org/html/2608.21449#bib.bib9)\]\)\. The multivariate methods are in principle also applicable to univariate signals \(with two exceptions\), whereas the three univariate\-specific methods are not designed for the multivariate case\. Two XAI methods can be used to adapt existing XAI methods to time series\. Temporal Saliency Rescaling calculates two relevance scores\[[20](https://arxiv.org/html/2608.21449#bib.bib20)\]: one for all time points and one for all features by masking them individually; the final explanation is the product of the two\. Time Forward Tunnel \(TFT\) calculates the relevance for each time point using only the time points preceding it, disallowing the usage of future time points\[[10](https://arxiv.org/html/2608.21449#bib.bib10)\]\. This can be used to force XAI methods to respect the temporal aspect of time series data\. These methods can be options to apply XAI methods of other domains in TSC tasks while considering temporal structure\.

For a time series as input, only a single method can generate explanations in the frequency and time\-frequency domains\. DFT\-LRP combines Fourier transforms with Layer\-wise Relevance Propagation \(LRP\)\[[2](https://arxiv.org/html/2608.21449#bib.bib2)\]to generate explanations for the time, frequency, and time\-frequency domains and is developed for univariate data\[[35](https://arxiv.org/html/2608.21449#bib.bib35)\]\. The applied Fourier transforms can also be combined with other methods than LRP\. In many TSC tasks, class\-relevant patterns are present in multiple domains \(time, frequency, and time\-frequency\)\. Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)illustrates an example of such a TSC task: the digit nine spoken by a female \(Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)upper part\) and by a male speaker \(Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)lower part\) from the AudioMNIST data set\[[6](https://arxiv.org/html/2608.21449#bib.bib6)\]\. The classification task is to distinguish between female and male speakers\.

![Refer to caption](https://arxiv.org/html/2608.21449v1/audiomnist_time_and_freq.png)Figure 4:Spoken digit nine from the AudioMNIST data by a female \(upper\) and a male \(lower\) speaker\. DFT\-LRP explanations \(SIGN\-XAI\-2 framework\) are shown in the time \(left\) and frequency domain \(right\)\. Red dots indicate features \(time steps or frequencies\) with positive relevance scores; higher intensity corresponds to higher relevance\. The expected ground truth in the frequency domain is highlighted in orange\.We trained the model from\[[35](https://arxiv.org/html/2608.21449#bib.bib35)\]using samples of the digit nine, with a balanced set of female and male speakers \(same preprocessing as in\[[35](https://arxiv.org/html/2608.21449#bib.bib35)\], 1,200 records, 66\.7/16\.7/16\.7% train/validation/test split, 100% test accuracy\)\. We classified a single instance from a male and a female recording and generated explanations for the resulting predictions using DFT\-LRP within the SIGN\-XAI\-2 framework\. In the time domain \(Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)left side\), the explanations are scattered across the signal and concentrated on positive and negative peaks\. In contrast, the frequency domain \(Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)right side\) clearly reveals the characteristic difference between a female and a male pronunciation of the digit nine\. For adult speakers, the fundamental frequency typically ranges from 165 to 255 Hz for female voices, whereas male voices usually exhibit fundamental frequencies between 90 to 155 Hz\[[5](https://arxiv.org/html/2608.21449#bib.bib5)\]\. The frequency regions corresponding to these ranges are highlighted in Fig\.[4](https://arxiv.org/html/2608.21449#S3.F4)and serve as ground\-truth information\. For both samples, the highest relevance scores are concentrated within their ground\-truth regions\.

### 3\.3Evaluation Metrics

The frameworks support a total of 52 evaluation metrics: 27 perturbation\-based, 15 ground\-truth\-based, and 10 other \(see Tablein the Appendix\)\. There is a small overlap in supported metrics, as only one metric is supported by all three XAI evaluation frameworks\. XTSC\-Bench builds entirely on Quantus for the implementation of its metrics, therefore, the metrics of XTSC\-Bench are a subset of the metrics available in Quantus\. Out of the 52 evaluation metrics, 44 metrics are compatible with time series, six are not compatible \(restricted to images\) and only two are specifically developed for time series\. The two metrics specific to time series are Mask Information and Mask Entropy\[[7](https://arxiv.org/html/2608.21449#bib.bib7)\], they work on subsequences of time series measuring how well a predicted relevance fits to a ground\-truth mask\. None of the identified metrics are specifically developed for evaluating explanations within the frequency or time\-frequency domain\.

### 3\.4Benchmarking Capabilities

Out of the six analyzed frameworks, three support datasets \(XTSC\-Bench, TSInterpret, and time\_interpret\) for the comparative evaluation of explanations\. The datasets available in each framework are listed in Table[2](https://arxiv.org/html/2608.21449#S3.T2)\. XTSC\-Bench offers six synthetic datasets proposed by Ismail et al\.\[[20](https://arxiv.org/html/2608.21449#bib.bib20)\], which contain ground\-truth information and are variable in length and number of channels\. Additionally, it supports the University of California Riverside \(UCR\) and University of East Anglia \(UEA\) time series archives\[[8](https://arxiv.org/html/2608.21449#bib.bib8),[4](https://arxiv.org/html/2608.21449#bib.bib4)\], which are well established in AI research for TSC and comprise 128 and 30 datasets, respectively\. Five datasets are supported by time\_interpret, including three synthetic datasets and two real\-world clinical datasets \(of which one is not openly accessible\)\. The synthetic datasets are variable in length and number of channels and provide ground\-truth information\. The clinical datasets contain patient laboratory data and biomarkers but lack ground\-truth information\. The synthetic datasets of both XTSC\-Bench and time\_interpret enable benchmarking of XAI methods for TSC due to their ground\-truth information\. TSInterpret also supports the UCR and UEA time series archives\.

Table 2:Datasets of frameworks\.Frameworks \(F\): Frameworks supporting the dataset;Channels and Length \(L\): Univariate \( \) or multivariate \( \), synthetic datasets have variable \(v\) lengths and channel number;Synthetic \(S\): ✓ indicates dataset is synthetic, mixed \(m\), and✗otherwise;Ground Truth \(GT\): ✓ indicates dataset contains GT information and✗otherwise; \*no open access;†Archives consist of multiple datasets; here we only indicate their variability at the collection level\.DatasetFrameworksChannelsLSGTArma\[[7](https://arxiv.org/html/2608.21449#bib.bib7)\]time\_interpret✓✓v✓✓BioBank\[[31](https://arxiv.org/html/2608.21449#bib.bib31)\]time\_interpret\*\*\*\*\*Hawkes\[[3](https://arxiv.org/html/2608.21449#bib.bib3)\]time\_interpret✓✓v✓✓Hidden Markow Model\[[7](https://arxiv.org/html/2608.21449#bib.bib7)\]time\_interpret✓✓v✓✓Mimic III\[[21](https://arxiv.org/html/2608.21449#bib.bib21)\]time\_interpret✗3148✗✗Gaussian, Harmonic, Pseudo Periodic, Autoregressive \(AR\), Continuous AR, Narma\[[20](https://arxiv.org/html/2608.21449#bib.bib20)\]time\_interpret✓✓v✓✓UCR Archive†\[[8](https://arxiv.org/html/2608.21449#bib.bib8)\]TSInterpret,XTSC\-Bench✓✗vm✗UEA Archive†\[[4](https://arxiv.org/html/2608.21449#bib.bib4)\]TSInterpret,XTSC\-Bench✗✓vm✗
### 3\.5Reproducibility

We compared explanations and evaluation scores produced by XAI methods and evaluation metrics with different implementations across frameworks\. Five XAI methods \(Saliency, Occlusion, Feature Ablation, Deconvolution, and Integrated Gradients, others were excluded due to different parameter usage or different backend support\) were compared on a convolutional neural network \(CNN\) with the architecture described in Gumpfer et al\.\[[15](https://arxiv.org/html/2608.21449#bib.bib16)\], trained for 20 epochs on right bundle branch block \(RBBB\) records and an equal amount of healthy ECG records of the PTB\-XL dataset\[[36](https://arxiv.org/html/2608.21449#bib.bib36)\]\(full\-length 12\-lead ECGs, no preprocessing, 3,316 records, 80/10/10% train/validation/test split, 96\.98% test accuracy\)\. We explained the model predictions on the test set using Integrated Gradients \(IG\)\[[32](https://arxiv.org/html/2608.21449#bib.bib32)\]as it is implemented in TSInterpret via Captum and in SIGN\-XAI\-2 via Zennit\. IG attributes relevance by integrating gradients along a straight\-line path between a baseline and the input\. To ensure comparability, we used identical parameters in both frameworks \(zero baseline and 50 interpolation steps\)\. One example of resulting explanations is shown in Fig\.[5](https://arxiv.org/html/2608.21449#S3.F5), illustrating one channel \(lead V1\) of the ECG example from Fig\.[1](https://arxiv.org/html/2608.21449#S1.F1)\. The relevance distributions differ noticeably and only the Zennit\-based implementation \(Fig\.[5](https://arxiv.org/html/2608.21449#S3.F5), right\) explains the expected“M”\-shaped pattern characteristic of RBBB\[[33](https://arxiv.org/html/2608.21449#bib.bib33)\]\. To verify the results, we analyzed the whole test set of 443 instances: Captum\- and Zennit\-based IG yielded a mean Spearman rank correlation of−0\.14\-0\.14\(median−0\.17\-0\.17, std 0\.11; 0 of 443 instances hadρ\>0\.5\\rho\>0\.5\) and a top\-5% Jaccard overlap of 0\.26 \(±0\.09\\pm 0\.09\)\. These results confirm that the divergence in Fig\.[5](https://arxiv.org/html/2608.21449#S3.F5)reflects systematic disagreement rather than an instance\-specific or thresholding artifact\. The other investigated methods provided identical explanations across frameworks, when configured with matching parameters\. We also compared evaluation metrics across frameworks, but only Area under the Receiver Operating Characteristic Curve \(ROC\-AUC\)\[[12](https://arxiv.org/html/2608.21449#bib.bib12)\]is implemented by two \(Quantus and time\_interpret\)\. Unlike the explanations, the metric produced consistent values across both implementations\.

Figure 5:Electrocardiogram \(ECG\) example with right bundle branch block \(RBBB\)\. Left: Illustration of characteristic“M”\-shaped pattern \(rSR’\)\. Middle and right: Explanations generated using Integrated Gradients as implemented in TSInterpret \(middle, based on Captum\) and SIGN\-XAI\-2 \(right, based on Zennit\) for the same ECG instance, illustrating differences in relevance distributions across implementations\. Relevance scores are indicated by red dots; higher intensity indicates higher relevance\.

## 4Open Challenges & Further Research

Compared to 2022, when only one framework supported time series data\[[24](https://arxiv.org/html/2608.21449#bib.bib24)\], we now identified five additional frameworks that claim to support time series, indicating increased research activity\. This development suggests that earlier calls for time\-series\-aware XAI frameworks\[[34](https://arxiv.org/html/2608.21449#bib.bib34)\]have been acknowledged by the community\. However, our analysis shows that explicit support for time\-series\-specific properties remains limited\.

#### Method Applicability\.

Although many supported XAI methods are technically applicable to TSC, only a small fraction has been explicitly developed for time series\. Two frameworks, Quantus and tsCaptum, do not include any XAI methods that are specifically designed for time series, but instead rely exclusively on reusing or wrapping methods developed for other data types\. The remaining frameworks also partially rely on generic explanation methods\. However, the suitability of the underlying methods for time series is often not explicitly validated\. Further research should investigate to which extent these methods account for time\-series\-specific properties like temporal and cross\-channel dependencies\.

#### Cross\-Channel Dependencies\.

While most supported XAI methods can highlight relevant regions in multiple channels, the dependencies of these relevant parts are not represented and would require an additional dimension of explanation\. This issue also manifests in the common use of heatmap\-based relevance visualizations, which are unable to highlight class\-relevant synchronous and asynchronous cross\-channel dependencies\. PAX\-TS\[[23](https://arxiv.org/html/2608.21449#bib.bib23)\], a perturbation\-based method, addresses this for forecasting tasks, but at higher computational cost\. More generally, gradient\-based approaches are efficient for high\-dimensional inputs yet cannot explain cross\-channel dependencies, whereas perturbation\-based methods can, at increased cost, suggesting potential benefits of hybrid approaches for future research\.

#### Frequency and Time\-Frequency Domains\.

Another major limitation is the insufficient support for frequency and time\-frequency domain explanations\. In many TSC tasks, including audio, biomedical, and sensor\-based applications, time\-domain explanations alone are often insufficient, as class\-relevant patterns are present in multiple domains \(time, frequency, and time\-frequency\)\. DFT\-LRP is currently the only XAI method of the analyzed frameworks that generates explanations in multiple domains for models using time\-domain inputs\. However, Fourier transformations, used by DFT\-LRP, can also be combined with other XAI methods to extend their explanations to the frequency and time\-frequency domains\. Currently, among the analyzed frameworks only SIGN\-XAI\-2 implements a frequency\-aware method, which remain rare in the literature as well\.

#### Evaluation Metrics\.

Similar limitations exist for evaluation metrics\. Although many metrics implemented in current frameworks are technically applicable to time series, it is often unclear to which extent they account for time\-series\-specific properties\. An assumption that some metrics make is the independence of feature relevance, which is violated by temporal and cross\-channel dependencies in time series\. Only two of the 52 analyzed metrics were developed specifically for time series, in contrast to the literature, which offers time\-series\-specific metrics such as the Attribution Stability Indicator\[[29](https://arxiv.org/html/2608.21449#bib.bib29)\], Swap Time Points, and Mean Time Points\[[28](https://arxiv.org/html/2608.21449#bib.bib28)\]\. Representations of time series in the frequency domain are not considered by any of the evaluation metrics\. The application of existing perturbation\-based metrics to evaluate explanations in the frequency domain is problematic, because small perturbations can impact the entire signal in the time domain\. Future research should investigate to which extent supported metrics account for time\-series\-specific properties like temporal and cross\-channel dependencies as well as frequency and time\-frequency evaluations\. We encourage researchers to develop dedicated metrics for these properties\.

#### Benchmarking and Ground\-Truth Availability\.

Only half of the frameworks support benchmarking datasets and only two include ground\-truth information\. All datasets with ground truth are based on synthetic or predefined data distributions\. It remains unclear to what extent they represent real time series\. Extending existing real\-world datasets with ground\-truth annotations would allow for more reliable benchmarking and transfer to real applications\.

#### Reproducibility and Consistency\.

Our reproducibility analysis shows that explanation results may depend on the choice of the XAI framework\. We observed substantial differences across frameworks for explanations generated by IG\. For one framework, the class\-relevant pattern was not visible in the resulting explanations\. Prior work by Le et al\.\[[24](https://arxiv.org/html/2608.21449#bib.bib24)\]has shown that evaluation metrics can exhibit substantial variability across implementations\. In contrast, we did not observe discrepancies for evaluation metrics in our experiments\. However, this observation is likely limited due to the small overlap of evaluation metrics across frameworks, which restricts the scope of reproducibility analysis for metrics\. We emphasize that researchers need to take care when comparing or reproducing published results \(without knowledge of the used framework\), this highlights current limitations of XAI research in generating reliable explanations\.

#### Transferability\.

Although this survey focuses on TSC, many of the identified limitations may also affect other time series tasks, such as forecasting and anomaly detection, which are even less studied within current XAI research for time series despite their practical relevance\.

#### Limitations\.

The systematic search underlying this review was conducted entirely on GitHub, which allowed us to specifically filter for software contributions\. We acknowledge that this procedure might miss research software published on other platforms\. Furthermore, our cross\-framework reproducibility analysis is limited in scope: only five explanation methods and a single evaluation metric were implemented in more than one framework\. Our results therefore demonstrate that explanations can depend on the framework implementation, but do not establish how prevalent this is across methods and metrics\.

## 5Conclusion

We presented the first systematic survey of XAI frameworks for TSC and identified substantial limitations in the current landscape\. These include limited support for time\-series\-specific XAI methods and evaluation metrics, minimal incorporation of alternative signal representations such as frequency and time\-frequency domains for XAI methods, a lack of ground\-truth annotations for real\-world benchmark datasets, and a lack of methods for explaining cross\-channel dependencies in multivariate time series\. We argue that researchers should exercise caution when applying methods or evaluation metrics for TSC from existing frameworks, as different implementations can yield substantially different results\. We provide Tables[1](https://arxiv.org/html/2608.21449#S3.T1)and[2](https://arxiv.org/html/2608.21449#S3.T2)as references to select frameworks and datasets for TSC tasks \(Tables[Appendix A](https://arxiv.org/html/2608.21449#Sx1)andin the Appendix can serve as additional resources for selecting time\-series specific methods and metrics\)\.

Publicly funded, community\-driven efforts in other disciplines, such as the platform Galaxy\[[13](https://arxiv.org/html/2608.21449#bib.bib13)\]in bioinformatics, highlight the potential of shared infrastructure for reproducible and collaborative research\. A standardized, community\-driven initiative would greatly benefit XAI research by enabling faithful, reproducible, and time\-series\-aware explanations as well as robust, systematic comparison and evaluation of XAI methods\.

#### Acknowledgements

This work was supported by the German Federal Ministry of Research, Technology and Space \(BMFTR\) through ExperTeam4KI \(grant no\. 16IS24063\)\. We gratefully acknowledge support from the hessian\.AI Service Center \(funded by the BMFTR, grant no\. 16IS22091\) and the hessian\.AI Innovation Lab \(funded by the Hessian Ministry for Digital Strategy and Innovation, grant no\. S\-DIW04/0013/003\)\. This work was partially funded by the Deutsche Forschungsgemeinschaft \(DFG, German Research Foundation\) under project ID 536124560\.

#### Disclosure of Interests\.

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]\(2026\)Software for dataset\-wide xai: from local explanations to global insights with zennit, corelay, and virelay\.PLOS One21\(1\),pp\. 1–38\.Note:https://github\.com/chr5tphr/zennitExternal Links:[Document](https://dx.doi.org/10.1371/journal.pone.0336683)Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1)\.
- \[2\]S\. Bach, A\. Binder, G\. Montavon, F\. Klauschen, K\. Müller, and W\. Samek\(2015\)On pixel\-wise explanations for non\-linear classifier decisions by layer\-wise relevance propagation\.PLOS One10\(7\),pp\. 1–46\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0130140)Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p2.1)\.
- \[3\]E\. Bacry, M\. Bompaire, P\. Deegan, S\. Gaïffas, and S\. V\. Poulsen\(2018\)Tick: a python library for statistical learning, with an emphasis on hawkes processes and time\-dependent models\.Journal of Machine Learning Research18\(214\),pp\. 1–5\.Cited by:[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.5.1.1.1)\.
- \[4\]A\. Bagnall, H\. A\. Dau, J\. Lines, M\. Flynn, J\. Large, A\. Bostrom, P\. Southam, and E\. Keogh\(2018\)The UEA multivariate time series classification archive, 2018\.Computing Research Repositoryabs/1811\.00075\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1811.00075),1811\.00075Cited by:[§3\.4](https://arxiv.org/html/2608.21449#S3.SS4.p1.1),[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.10.1.1.1)\.
- \[5\]R\. J\. Baken and R\. F\. Orlikoff\(2000\)Clinical measurement of speech and voice\.2\. edition,Singular Thomson Learning\.External Links:ISBN 1565938690Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p3.1)\.
- \[6\]S\. Becker, J\. Vielhaben, M\. Ackermann, K\. Müller, S\. Lapuschkin, and W\. Samek\(2024\)AudioMNIST: exploring explainable artificial intelligence for audio analysis on a simple benchmark\.Journal of the Franklin Institute361\(1\),pp\. 418–428\.External Links:[Document](https://dx.doi.org/10.1016/j.jfranklin.2023.11.038),ISSN 0016\-0032Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p2.1)\.
- \[7\]J\. Crabbé and M\. Van Der Schaar\(2021\)Explaining time series predictions with dynamic masks\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),M\. Meila and T\. Zhang \(Eds\.\),Vol\.139,online,pp\. 2166–2177\.Cited by:[§3\.3](https://arxiv.org/html/2608.21449#S3.SS3.p1.1),[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.3.1.1.1),[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.6.1.1.1)\.
- \[8\]H\. A\. Dau, A\. Bagnall, K\. Kamgar, C\. M\. Yeh, Y\. Zhu, S\. Gharghabi, C\. A\. Ratanamahatana, and E\. Keogh\(2019\)The UCR time series archive\.IEEE/CAA Journal of Automatica Sinica6\(6\),pp\. 1293–1305\.External Links:[Document](https://dx.doi.org/10.1109/JAS.2019.1911747)Cited by:[§3\.4](https://arxiv.org/html/2608.21449#S3.SS4.p1.1),[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.9.1.1.1)\.
- \[9\]E\. Delaney, D\. Greene, and M\. T\. Keane\(2021\)Instance\-based counterfactual explanations for time series classification\.InCase\-Based Reasoning Research and Development,A\. A\. Sánchez\-Ruiz and M\. W\. Floyd \(Eds\.\),Cham,pp\. 32–47\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-86957-1%5F3),ISBN 978\-3\-030\-86957\-1Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p1.1)\.
- \[10\]J\. Enguehard\(2023\)Time Interpret: A unified model interpretability library for time series\.Computing Research Repositoryabs/2306\.02968\.Note:https://github\.com/josephenguehard/time\_interpretExternal Links:[Document](https://dx.doi.org/10.48550/ARXIV.2306.02968),2306\.02968Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.8.1)\.
- \[11\]European Union\(2024\)Regulation \(EU\) 2024/1689 of the European Parliament and of the council laying down harmonised rules on artificial intelligence \(Artificial Intelligence Act\)\.Official Journal of the European Union\.Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p1.1)\.
- \[12\]T\. Fawcett\(2006\)An introduction to ROC analysis\.Pattern Recognition Letters27\(8\),pp\. 861–874\.External Links:[Document](https://dx.doi.org/10.1016/j.patrec.2005.10.010),ISSN 0167\-8655Cited by:[§3\.5](https://arxiv.org/html/2608.21449#S3.SS5.p1.1)\.
- \[13\]T\. Galaxy Community\(2024\)The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update\.Nucleic Acids Research52\(W1\),pp\. W83–W94\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkae410),https://academic\.oup\.com/nar/article\-pdf/52/W1/W83/58436305/gkae410\.pdf,ISSN 0305\-1048Cited by:[§5](https://arxiv.org/html/2608.21449#S5.p2.1)\.
- \[14\]M\. Guillemé, V\. Masson, L\. Rozé, and A\. Termier\(2019\)Agnostic local explanation for time series classification\.In2019 IEEE 31st International Conference on Tools with Artificial Intelligence \(ICTAI\),Portland, Oregon, USA,pp\. 432–439\.External Links:[Document](https://dx.doi.org/10.1109/ICTAI.2019.00067)Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p1.1)\.
- \[15\]N\. Gumpfer, B\. Dinov, S\. Sossalla, M\. Guckert, and J\. Hannig\(2024\)Towards trustworthy ai in cardiology: a comparative analysis of explainable ai methods for electrocardiogram interpretation\.InArtificial Intelligence in Medicine,J\. Finkelstein, R\. Moskovitch, and E\. Parimbelli \(Eds\.\),Cham,pp\. 350–361\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-66535-6%5F36),ISBN 978\-3\-031\-66535\-6Cited by:[§3\.5](https://arxiv.org/html/2608.21449#S3.SS5.p1.1)\.
- \[16\]N\. Gumpfer, J\. Prim, T\. Keller, B\. Seeger, M\. Guckert, and J\. Hannig\(2023\)SIGNed explanations: unveiling relevant features by reducing bias\.Information Fusion99,pp\. 101883\.Note:https://github\.com/TimeXAIgroup/signxai2External Links:[Document](https://dx.doi.org/10.1016/j.inffus.2023.101883),ISSN 1566\-2535Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.5.1.1)\.
- \[17\]A\. Hedström, L\. Weber, D\. Krakowczyk, D\. Bareeva, F\. Motzkus, W\. Samek, S\. Lapuschkin, and M\. M\.\-C\. Höhne\(2023\)Quantus: an explainable AI toolkit for responsible evaluation of neural network explanations and beyond\.Journal of Machine Learning Research24,pp\. 34:1–34:11\.Note:https://github\.com/understandable\-machine\-intelligence\-lab/QuantusCited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.7.1.1)\.
- \[18\]J\. Höllig, C\. Kulbach, and S\. Thoma\(2022\)TSInterpret: A unified framework for time series interpretability\.Note:https://github\.com/fzi\-forschungszentrum\-informatik/TSInterpretExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2208.05280),2208\.05280Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.3.1.1)\.
- \[19\]J\. Höllig, S\. Thoma, and F\. Grimm\(2023\)XTSC\-Bench: quantitative benchmarking for explainers on time series classification\.In2023 International Conference on Machine Learning and Applications \(ICMLA\),Jacksonville, Florida, USA,pp\. 1126–1131\.Note:https://github\.com/JHoelli/XTSC\-BenchExternal Links:[Document](https://dx.doi.org/10.1109/ICMLA58977.2023.00168)Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.9.1.1)\.
- \[20\]A\. A\. Ismail, M\. Gunady, H\. Corrada Bravo, and S\. Feizi\(2020\)Benchmarking deep learning interpretability in time series predictions\.InAdvances in Neural Information Processing Systems \(NeurIPS\),H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,online,pp\. 6441–6452\.Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p1.1),[§3\.4](https://arxiv.org/html/2608.21449#S3.SS4.p1.1),[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.8.1.1.1)\.
- \[21\]A\. E\. W\. Johnson, T\. J\. Pollard, L\. Shen, L\. H\. Lehman, M\. Feng, M\. Ghassemi, B\. Moody, P\. Szolovits, L\. Anthony Celi, and R\. G\. Mark\(2016\)MIMIC\-III, a freely accessible critical care database\.Scientific Data3\(1\),pp\. 160035\.External Links:[Document](https://dx.doi.org/10.1038/sdata.2016.35),ISBN 2052\-4463Cited by:[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.7.1.1.1)\.
- \[22\]N\. Kokhlikyan, V\. Miglani, M\. Martin, E\. Wang, B\. Alsallakh, J\. Reynolds, A\. Melnikov, N\. Kliushkina, C\. Araya, S\. Yan, and O\. Reblitz\-Richardson\(2020\)Captum: A unified and generic model interpretability library for PyTorch\.Computing Research Repositoryabs/2009\.07896\.Note:https://github\.com/meta\-pytorch/captumExternal Links:[Document](https://dx.doi.org/10.48550/arXiv.2009.07896),2009\.07896Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1)\.
- \[23\]T\. Kreuzer, J\. Zdravkovic, and P\. Papapetrou\(2025\)PAX\-TS: model\-agnostic multi\-granular explanations for time series forecasting via localized perturbations\.Computing Research Repositoryabs/2508\.18982\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2508.18982),2508\.18982Cited by:[§4](https://arxiv.org/html/2608.21449#S4.SS0.SSSx2.p1.1)\.
- \[24\]P\. Q\. Le, M\. Nauta, V\. B\. Nguyen, S\. Pathak, J\. Schlötterer, and C\. Seifert\(2023\)Benchmarking eXplainable AI \- A survey on available toolkits and open challenges\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence \(IJCAI\),E\. Elkind \(Ed\.\),Macau,pp\. 6665–6673\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2023/747)Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p6.1),[§1](https://arxiv.org/html/2608.21449#S1.p7.1.1),[§2](https://arxiv.org/html/2608.21449#S2.SS0.SSSx1.p1.1),[§4](https://arxiv.org/html/2608.21449#S4.SS0.SSSx6.p1.1),[§4](https://arxiv.org/html/2608.21449#S4.p1.1)\.
- \[25\]R\. Meudec\(2021\)tf\-explain\.Note:Software\. https://github\.com/sicara/tf\-explainExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.5711704)Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1)\.
- \[26\]M\. Nauta, J\. Trienes, S\. Pathak, E\. Nguyen, M\. Peters, Y\. Schmitt, J\. Schlötterer, M\. van Keulen, and C\. Seifert\(2023\)From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI\.ACM Computing Surveys55\(13s\),pp\. 295:1–295:42\.External Links:[Document](https://dx.doi.org/10.1145/3583558),ISSN 0360\-0300Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p2.1)\.
- \[27\]T\. Rojat, R\. Puget, D\. Filliat, J\. D\. Ser, R\. Gelin, and N\. D\. Rodríguez\(2021\)Explainable artificial intelligence \(XAI\) on timeseries data: A survey\.Computing Research Repositoryabs/2104\.00950\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2104.00950),2104\.00950Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p3.1),[§1](https://arxiv.org/html/2608.21449#S1.p6.1),[§1](https://arxiv.org/html/2608.21449#S1.p7.1)\.
- \[28\]U\. Schlegel, H\. Arnout, M\. El\-Assady, D\. Oelke, and D\. A\. Keim\(2019\)Towards a rigorous evaluation of xai methods on time series\.In2019 IEEE/CVF International Conference on Computer Vision Workshop \(ICCVW\),Seoul, South Korea,pp\. 4197–4201\.External Links:[Document](https://dx.doi.org/10.1109/ICCVW.2019.00516)Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p2.1),[§4](https://arxiv.org/html/2608.21449#S4.SS0.SSSx4.p1.1)\.
- \[29\]U\. Schlegel and D\. A\. Keim\(2023\)Introducing the attribution stability indicator: a measure for time series XAI attributions\.Computing Research Repositoryabs/2310\.04178\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2310.04178),2310\.04178Cited by:[§4](https://arxiv.org/html/2608.21449#S4.SS0.SSSx4.p1.1)\.
- \[30\]D\. I\. Serramazza, T\. L\. Nguyen, and G\. Ifrim\(2024\)A short tutorial for multivariate time series explanation using tsCaptum\.Software Impacts22,pp\. 100723\.Note:https://github\.com/mlgig/tscaptumExternal Links:[Document](https://dx.doi.org/10.1016/j.simpa.2024.100723),ISSN 2665\-9638Cited by:[§3\.1](https://arxiv.org/html/2608.21449#S3.SS1.p1.1),[Table 1](https://arxiv.org/html/2608.21449#S3.T1.17.4.1)\.
- \[31\]C\. Sudlow, J\. Gallacher, N\. Allen, V\. Beral, P\. Burton, J\. Danesh, P\. Downey, P\. Elliott, J\. Green, M\. Landray, B\. Liu, P\. Matthews, G\. Ong, J\. Pell, A\. Silman, A\. Young, T\. Sprosen, T\. Peakman, and R\. Collins\(2015\)UK Biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age\.PLOS Medicine12\(3\),pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1371/journal.pmed.1001779)Cited by:[Table 2](https://arxiv.org/html/2608.21449#S3.T2.14.4.1.1.1)\.
- \[32\]M\. Sundararajan, A\. Taly, and Q\. Yan\(2017\)Axiomatic attribution for deep networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),Vol\.70,Sydney, NSW, Australia,pp\. 3319–3328\.Cited by:[§3\.5](https://arxiv.org/html/2608.21449#S3.SS5.p1.1)\.
- \[33\]B\. Surawicz, R\. Childers, B\. J\. Deal, and L\. S\. Gettes\(2009\)AHA/ACCF/HRS recommendations for the standardization and interpretation of the electrocardiogram\.Circulation119\(10\),pp\. e235–e240\.External Links:[Document](https://dx.doi.org/10.1161/CIRCULATIONAHA.108.191095),https://www\.ahajournals\.org/doi/pdf/10\.1161/CIRCULATIONAHA\.108\.191095Cited by:[§3\.5](https://arxiv.org/html/2608.21449#S3.SS5.p1.1)\.
- \[34\]A\. Theissler, F\. Spinnato, U\. Schlegel, and R\. Guidotti\(2022\)Explainable AI for time series classification: a review, taxonomy and research directions\.IEEE Access10,pp\. 100700–100724\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2022.3207765)Cited by:[§1](https://arxiv.org/html/2608.21449#S1.p6.1),[§1](https://arxiv.org/html/2608.21449#S1.p7.1),[§4](https://arxiv.org/html/2608.21449#S4.p1.1)\.
- \[35\]J\. Vielhaben, S\. Lapuschkin, G\. Montavon, and W\. Samek\(2024\)Explainable AI for time series via Virtual Inspection Layers\.Pattern Recognition150,pp\. 110309\.External Links:[Document](https://dx.doi.org/10.1016/j.patcog.2024.110309),ISSN 0031\-3203Cited by:[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p2.1),[§3\.2](https://arxiv.org/html/2608.21449#S3.SS2.p3.1)\.
- \[36\]P\. Wagner, N\. Strodthoff, R\. Bousseljot, D\. Kreiseler, F\. I\. Lunze, W\. Samek, and T\. Schaeffter\(2020\)PTB\-XL, a large publicly available electrocardiography dataset\.Scientific Data7\(1\),pp\. 154\.External Links:[Document](https://dx.doi.org/10.1038/s41597-020-0495-6),ISBN 2052\-4463Cited by:[§3\.5](https://arxiv.org/html/2608.21449#S3.SS5.p1.1)\.

## Appendix A

XAI methods of frameworks\.Compatibility: ✓ indicates method is specific for time series, otherwise marked as – \(not explicitly stated\);Dimensionality: supports uni\- \(U\), multivariate \(M\) time series or both \(B\);Frequency: ✓ generates explanations for the frequency and time\-frequency domain;Frameworks: ✓ indicates that the framework implements the method, \(✓\) if the framework wrapped or reused implementations of other frameworks, and✗otherwise\. Abbr\.: Counterfactuals \(CF\)FrameworksMethodsCompatibility

Dimensionality

Frequency

Quantus

TSInterpret

timeinterpret

tsCaptum

XTSC\-Bench

SIGN\-XAI\-2

\\endfirstheadTable A\.0 –continued from previous pageFrameworksMethodsCompatibility

Dimensionality

Frequency

Quantus

TSInterpret

timeinterpret

tsCaptum

XTSC\-Bench

SIGN\-XAI\-2

\\endheadContinued on next page