Causal Discovery with Inverted Self-attention for Multivariate Time Series

# Causal Discovery with Inverted Self-attention for Multivariate Time Series
Source: [https://arxiv.org/html/2607.28212](https://arxiv.org/html/2607.28212)
11institutetext:School of Computer Science, University of Technology Sydney, Sydney, Australia
11email:\{Yusen\.Liu, Yifan\.Yin, Huan\.Huo\}@uts\.edu\.au22institutetext:Department of Technology, Management and Economics, Technical University of Denmark, Lyngby, Denmark
22email:xiuli@dtu\.dk33institutetext:TianFu YongXing Laboratory, Chengdu, China
33email:ethan\-wy@foxmail\.com44institutetext:Faculty of Data Science, City University of Macau, Macau, China
44email:tqzhu@cityu\.edu\.mo###### Abstract

Causal discovery in multivariate time series data is challenging due to complex interactions, high dimensionality, and nonlinear dependencies among variables\. Existing methods often struggle to capture these complexities, resulting in inaccurate causal structures\. To address this issue, we propose a novel framework that leverages self\-attention mechanisms within the transformer architecture for causal discovery\. Our approach introduces a novel inverted causal self\-attention mechanism \(CSAM\) that emphasizes latent and indirect causal relationships by inverting tokens and inducing sparsity in attention scores, focusing on significant causal interactions and reducing spurious correlations\. Additionally, we develop a global causal algorithm to identify global causal links, providing a holistic metric for causal influence, along with a causal verification module to ensure robustness in the identified causal relationships, enhancing the reliability of our framework\. Experiments on both linear and nonlinear datasets, along with ablation studies and sensitivity analyses, show that our framework outperforms existing methods, demonstrating its potential for causal discovery in complex multivariate time series\.

## 1Introduction

Time series analysis is widely applied in fields like economics, environmental science, and healthcare\. The temporal dependencies and dynamic nature of time series data create unique challenges, requiring advanced analytical techniques for tasks such as classification and forecasting\. Beyond these tasks, understanding causal relationships between variables in multivariate time series holds particular importance, as it not only provides insights into underlying mechanisms but also improves forecasting and classification performance, supporting more informed decision\-making across various domains\.

However, causal discovery in time series data remains challenging due to the complex interactions between variables over time\. A primary difficulty is distinguishing causality from correlation, as temporal dependencies can obscure true causal structures, leading to potential misinterpretations\[[13](https://arxiv.org/html/2607.28212#bib.bib22)\]\. This challenge is further intensified by the high dimensionality and potential nonlinearity present in multivariate data\[[15](https://arxiv.org/html/2607.28212#bib.bib18)\]\. Traditional methods often struggle to handle these complexities, which can limit research and applications; inaccurate causal links may result in flawed conclusions and less effective interventions, impacting decision\-making in critical areas\[[17](https://arxiv.org/html/2607.28212#bib.bib15)\]\.

Existing methods for causal discovery in time series data, including distance\-based\[[16](https://arxiv.org/html/2607.28212#bib.bib61)\], index\-based\[[22](https://arxiv.org/html/2607.28212#bib.bib20)\], and traditional statistical methods\[[4](https://arxiv.org/html/2607.28212#bib.bib1)\], primarily capture linear relationships but are often sensitive to noise and may confuse causation with correlation, limiting their effectiveness in dynamic multivariate contexts\[[2](https://arxiv.org/html/2607.28212#bib.bib11)\]\. Kernel\-based approaches, such as Kernel Granger Causality \(KGC\)\[[8](https://arxiv.org/html/2607.28212#bib.bib4)\], extend causal discovery to nonlinear relationships\. Constraint\-based methods, including Time Series Fast Causal Inference \(tsFCI\)\[[3](https://arxiv.org/html/2607.28212#bib.bib31)\]and PCMCI\[[14](https://arxiv.org/html/2607.28212#bib.bib30)\], detect causal links even with latent confounders\. Score\-based methods, such as NOTEARS\[[24](https://arxiv.org/html/2607.28212#bib.bib33)\]and DYNOTEARS\[[11](https://arxiv.org/html/2607.28212#bib.bib56)\], offer additional tools for uncovering causal structures in multivariate time series, addressing some limitations of traditional methods\. Despite these advances, there has been limited exploration of deep learning’s potential for causal discovery\[[10](https://arxiv.org/html/2607.28212#bib.bib7)\], especially given its capacity to model complex, high\-dimensional data\.

To further address high dimensionality and nonlinearity challenges in multivariate time series data, we propose a novel framework that leverages an inverted self\-attention mechanism for causal discovery\. First, we propose an inverted causal self\-attention mechanism \(CSAM\) that highlights latent and indirect causal relationships\. Specifically, we apply Sparsemax in the attention score matrix, inducing sparsity to focus on significant causal interactions and reduce spurious correlations\. Then, we develop a global algorithm to integrate attention scores across model components, providing a metric for causal influence that captures complex causal structures in time series data\. This approach enables the model to handle nonlinear interactions while remaining robust to noise and confounders, improving interpretability and reducing false positives\. Finally, we integrate Permutation Importance \(PI\)\[[12](https://arxiv.org/html/2607.28212#bib.bib23)\]to verify that identified causal relationships are robust, enhancing the reliability of our causal discovery pipeline\. In summary, the main contributions of this paper are:

- −\-We propose a novel framework that leverages an inverted causal self\-attention mechanism for causal discovery, enabling the capture of complex, nonlinear causal relationships in multivariate time series\.
- −\-We introduce a sparsity\-inducing modification to the attention score matrix, improving interpretability and reducing spurious correlations\.
- −\-We develop a global causal algorithm to identify global causal links, along with a causal verification module to ensure robustness in the identified relationships\.
- −\-We validate our framework through experiments on high\-dimensional and complex time series data, with ablation studies and sensitivity analyses demonstrating the contribution of each component\.

The structure of this paper is as follows: Section[2](https://arxiv.org/html/2607.28212#S2)reviews related work on causal discovery in time series and attention mechanisms\. Section[3](https://arxiv.org/html/2607.28212#S3)introduces the definitions and assumptions of this paper\. Section[4](https://arxiv.org/html/2607.28212#S4)details our proposed methodology\. Section[5](https://arxiv.org/html/2607.28212#S5)presents the experiments and results\. Finally, Section[6](https://arxiv.org/html/2607.28212#S6)concludes the paper and suggests future research directions\.

## 2Related Work

##### Causal Discovery for Time Series\.

Causal discovery in time series data aims to identify causal relationships, with Granger causality as a foundational method\. It posits that if past values of a time seriesYYimprove the prediction of another seriesXX, thenYYGranger\-causesXX\[[4](https://arxiv.org/html/2607.28212#bib.bib1)\]\. Building on this, KGC\[[8](https://arxiv.org/html/2607.28212#bib.bib4)\]extends Granger’s framework to nonlinear relationships via kernel methods\. Distance\-based methods\[[16](https://arxiv.org/html/2607.28212#bib.bib61)\]also capture nonlinear relationships but are sensitive to noise and require careful parameter tuning\. Additionally, constraint\-based approaches like tsFCI\[[3](https://arxiv.org/html/2607.28212#bib.bib31)\]detect causal links even with latent confounders\. PCMCI\[[14](https://arxiv.org/html/2607.28212#bib.bib30)\]combines the PC algorithm\[[18](https://arxiv.org/html/2607.28212#bib.bib8)\]with momentary conditional independence tests for efficient causal structure identification\. Score\-based methods, such as NOTEARS\[[24](https://arxiv.org/html/2607.28212#bib.bib33)\]and DYNOTEARS\[[11](https://arxiv.org/html/2607.28212#bib.bib56)\], use continuous optimization to capture temporal and nonlinear dependencies\. Further,\[[19](https://arxiv.org/html/2607.28212#bib.bib6),[10](https://arxiv.org/html/2607.28212#bib.bib7)\]leverage neural networks to infer causality directly\. Together, these methods offer a range of tools for analyzing causal relationships in time series\.

##### Attention Mechanisms\.

Attention mechanisms enable models to focus on relevant data segments and are widely used in fields like NLP and computer vision\. First introduced by\[[1](https://arxiv.org/html/2607.28212#bib.bib35)\]for machine translation, attention improved both model performance and interpretability\. The Transformer architecture by\[[20](https://arxiv.org/html/2607.28212#bib.bib36)\], relying entirely on attention, set new benchmarks in NLP\. In time series analysis, attention mechanisms have shown strong potential in models like Informer\[[25](https://arxiv.org/html/2607.28212#bib.bib37)\]and Crossformer\[[23](https://arxiv.org/html/2607.28212#bib.bib38)\]\. Attention\-based models have also been applied in causal discovery, with\[[10](https://arxiv.org/html/2607.28212#bib.bib7)\]incorporating attention into convolutional neural networks to improve causal effect estimation\. In contrast to these approaches, we propose a novel CSAM to identify latent and nonlinear causal relationships in multivariate time series data, facilitating a deeper understanding of dynamic interactions\.

## 3Definitions and Assumptions

###### Definition 1

\(Multivariate Time Series\)A multivariate time series is defined as anMM\-dimensional series𝐗=\(𝐗1,𝐗2,…,𝐗M\)\\mathbf\{X\}=\(\\mathbf\{X\}^\{1\},\\mathbf\{X\}^\{2\},\\ldots,\\mathbf\{X\}^\{M\}\)of lengthTT, where each component𝐗i\\mathbf\{X\}^\{i\}represents an individual time series\.

###### Definition 2

\(Causal Graph\)Given a multivariate time series𝐗\\mathbf\{X\}, its causal graph is a directed acyclic graph \(DAG\)G=\(V,E\)G=\(V,E\), whereV=\{1,2,…,M\}V=\\\{1,2,\\dots,M\\\}represents the set of variables \(time series\), andEEis the set of directed edges\. An edge\(i,j\)∈E\(i,j\)\\in Eexists if and only if𝐗i\\mathbf\{X\}^\{i\}Granger\-causes𝐗j\\mathbf\{X\}^\{j\}\.

###### Assumption 1

\(Causal Markov\)The causal Markov condition states that each variable is conditionally independent of non\-effects given its direct causes, focusing the analysis on direct causal dependencies\.

###### Assumption 2

\(Faithfulness\)Faithfulness assumes that any conditional independencies in the data reflect the true causal structure, ensuring that observed independencies correspond to actual causal relationships\.

###### Assumption 3

\(Strict Causal Ordering\)Strict temporal ordering in the causal graph implies that if𝐗i\\mathbf\{X\}^\{i\}causes𝐗j\\mathbf\{X\}^\{j\}, theni<ji<j, disallowing contemporaneous causation and simplifying temporal dependency analysis\.

###### Assumption 4

\(Bounded Dependencies\)For each edge\(i,j\)∈E\(i,j\)\\in E, a constantϵ\>0\\epsilon\>0exists such that the minimum absolute difference between the prediction error of𝐗j\\mathbf\{X\}^\{j\}with and without𝐗i\\mathbf\{X\}^\{i\}as a predictor exceedsϵ\\epsilonfor sufficiently largeTT, ensuring a minimum "signal strength" for true causal dependencies\.

## 4Methods

Following our proposed definitions and assumptions, we introduce a novel framework that systematically addresses the identification of Granger causality networks in multivariate time series𝐗\\mathbf\{X\}\. The model aims to delineate causal interactions among series components through matrixAA\. The full time series𝐗t\\mathbf\{X\}\_\{t\}is input to a causal self\-attention module \(CSAM\), which differs from traditional self\-attention in token handling and internal dynamics\. This module’s output feeds into a transformer, generating predictions for each series𝐗ti\\mathbf\{X\}^\{i\}\_\{t\}and identifying causal links with other series during training\. Granger causality is then derived from𝐗t\\mathbf\{X\}\_\{t\}through a global algorithm and verification strategy, as detailed in Sections[4\.2](https://arxiv.org/html/2607.28212#S4.SS2)and[4\.3](https://arxiv.org/html/2607.28212#S4.SS3)\.

### 4\.1Causal Self\-attention Mechanism

Transformers effectively capture complex dependencies in sequential data through self\-attention mechanisms\. To leverage this for causal discovery, we introduce an inverted self\-attention mechanism, CSAM, designed to identify causal relationships in multivariate time series data\. Traditional self\-attention computes attention scores based on the affinity between positions in the input sequence, formulated as:

Attention​\(Q,K,V\)=softmax​\(Q​KTdk\)​V\\text\{Attention\}\(Q,K,V\)=\\text\{softmax\}\\left\(\\frac\{QK^\{T\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)V\(1\)whereQQ,KK, andVVrepresent the matrices for queries, keys, and values, respectively, anddkd\_\{k\}is the dimensionality of the keys\. In multivariate time series analysis, each tokenX=\{x1,x2,x3,…\}X=\\\{x\_\{1\},x\_\{2\},x\_\{3\},\\ldots\\\}represents a combination of different variables, as shown in Fig\.[1](https://arxiv.org/html/2607.28212#S4.F1)\(a\)\. Each token corresponds to a single time point, and a sequence of tokens\{X1,X2,X3,…\}\\\{X\_\{1\},X\_\{2\},X\_\{3\},\\ldots\\\}represents consecutive time points\. Using these tokens in the self\-attention mechanism allows for the computation of inter\-temporal relationships\.

![Refer to caption](https://arxiv.org/html/2607.28212v1/x1.png)Figure 1:Comparison of token representations in \(a\) traditional self\-attention and \(b\) the proposed causal self\-attention mechanism\.Contrastingly, in Granger causality analysis, the focus is on identifying relationships among different time series rather than across time intervals\. To achieve this, we invert the token representation\. As shown in Fig\.[1](https://arxiv.org/html/2607.28212#S4.F1)\(b\), each inverted tokenX^\\hat\{X\}consists of sequential observations within a single time series,X^=\{x1,x2,x3,…\}\\hat\{X\}=\\\{x^\{1\},x^\{2\},x^\{3\},\\ldots\\\}, representing one complete time series\. A collection of these inverted tokens,\{X1^,X2^,X3^,…\}\\\{\\hat\{X\_\{1\}\},\\hat\{X\_\{2\}\},\\hat\{X\_\{3\}\},\\ldots\\\}, represents multiple time series\. Using these inverted tokens in the self\-attention framework allows for the analysis of cross\-series relationships, aligning with Granger causality objectives\. This approach maintains the temporal sequence within each series while enabling a detailed exploration of causal dynamics across the multivariate time series\.

After transposing the tokens, we apply the CSAM to reveal Granger causality among the time series, as illustrated in Fig\.[2](https://arxiv.org/html/2607.28212#S4.F2)\. The inverted tokens are initially transformed through multiplication with the weight matricesWk^W^\{\\hat\{k\}\},Wq^W^\{\\hat\{q\}\}, andWv^W^\{\\hat\{v\}\}, producing the key \(K^\\hat\{K\}\), query \(Q^\\hat\{Q\}\), and value \(V^\\hat\{V\}\) matrices:

K^=X^⋅Wk^,Q^=X^⋅Wq^,V^=X^⋅Wv^\\hat\{K\}=\\hat\{X\}\\cdot W^\{\\hat\{k\}\},\\quad\\hat\{Q\}=\\hat\{X\}\\cdot W^\{\\hat\{q\}\},\\quad\\hat\{V\}=\\hat\{X\}\\cdot W^\{\\hat\{v\}\}\(2\)It is imperative to note that, diverging from the conventional dimensions observed in standard transformers, which are typically indexed byK∈ℝN×dk\{K\}\\in\\mathbb\{R\}^\{N\\times d\_\{k\}\},Q∈ℝN×dq\{Q\}\\in\\mathbb\{R\}^\{N\\times d\_\{q\}\},V∈ℝN×dv\{V\}\\in\\mathbb\{R\}^\{N\\times d\_\{v\}\}, the dimensions of our key \(K^\\hat\{K\}\), query \(Q^\\hat\{Q\}\), and value \(V^\\hat\{V\}\) matrices are structured asK^∈ℝT×dk\\hat\{K\}\\in\\mathbb\{R\}^\{T\\times d\_\{k\}\},Q^∈ℝT×dq\\hat\{Q\}\\in\\mathbb\{R\}^\{T\\times d\_\{q\}\},V^∈ℝT×dv\\hat\{V\}\\in\\mathbb\{R\}^\{T\\times d\_\{v\}\}, WhereNNstands for the number of variables andTTstands for the time steps\. Focusing on a specific time series designated byii, we extract theii\-th row from theQ^\\hat\{Q\}matrix and engage in a matrix multiplication with the transposedKT^\\hat\{K^\{T\}\}matrix\. This operation is instrumental in deriving the initial attention scoresei→\\vec\{e\_\{i\}\}that encapsulate potential causal influences exerted by all time series on the series of interestii:

ei→=Q^i×K^Tdk\\vec\{e\_\{i\}\}=\\frac\{\\hat\{Q\}\_\{i\}\\times\\hat\{K\}^\{T\}\}\{\\sqrt\{d\_\{k\}\}\}\(3\)
To enhance the interpretability of the resultant weight vector, we apply theSparseMaxfunction\[[9](https://arxiv.org/html/2607.28212#bib.bib60)\]\. This transformation imparts sparsity to the vector, thereby facilitating a more straightforward interpretation by accentuating the most salient causal relationships and diminishing the noise from negligible interactions\. TheSparseMaxfunction is a differentiable alternative to the softmax function, designed to provide sparser probabilities\. Formally, theSparseMaxfunction for a vector𝐳∈ℝd\\mathbf\{z\}\\in\\mathbb\{R\}^\{d\}is defined as the solution to the following optimization problem:

SparseMax​\(𝐳\)=argmin𝐩∈Δd−1​‖𝐩−𝐳‖2\\textit\{SparseMax\}\(\\mathbf\{z\}\)=\\underset\{\\mathbf\{p\}\\in\\Delta^\{d\-1\}\}\{\\textit\{argmin\}\}\\left\\\|\\mathbf\{p\}\-\\mathbf\{z\}\\right\\\|^\{2\}\(4\)whereΔd−1\\Delta^\{d\-1\}denotes the\(d−1\)\(d\-1\)\-dimensional simplex, i\.e\., the set of𝐩∈ℝd\\mathbf\{p\}\\in\\mathbb\{R\}^\{d\}such that∑i=1dpi=1\\sum\_\{i=1\}^\{d\}p\_\{i\}=1andpi≥0p\_\{i\}\\geq 0for allii\. TheSparseMaxoperation projects the input vector𝐳\\mathbf\{z\}onto the simplex, resulting in a sparse probability distribution where many elements can be exactly zero\.

α→i=SparseMax​\(ei→\)\\vec\{\\alpha\}\_\{i\}=\\textit\{SparseMax \}\(\\vec\{e\_\{i\}\}\)\(5\)After applyingSparseMax, the non\-zero elements in the sparse vectorsαi→\\vec\{\\alpha\_\{i\}\}are added to the causal tensor𝐂i​j​t\\mathbf\{C\}\_\{ijt\}as potential causes of theii\-variables, wherettdenotes different tokens\. Additionally,αi→\\vec\{\\alpha\_\{i\}\}is duplicated across thedvd\_\{v\}dimension to form an attention map𝐀𝐢\\mathbf\{A\_\{i\}\}for the CSAM\. The attention map represents the refined causal influence structure, with each element indicating the degree of influence one time series has on another within the multivariate context\.

𝐀𝐢=Replicate​\(α→i,dv\)=\[α→iα→i⋯α→i\]\\mathbf\{A\_\{i\}\}=\\textit\{Replicate\}\(\\vec\{\\alpha\}\_\{i\},d\_\{v\}\)=\\begin\{bmatrix\}\\vec\{\\alpha\}\_\{i\}&\\vec\{\\alpha\}\_\{i\}&\\cdots&\\vec\{\\alpha\}\_\{i\}\\end\{bmatrix\}\(6\)
![Refer to caption](https://arxiv.org/html/2607.28212v1/x2.png)Figure 2:Illustration of the process of our causal self\-attention mechanism \(CSAM\)\.The final stage performs an element\-wise multiplication \(Hadamard product\) between the attention map𝐀𝐢\\mathbf\{A\_\{i\}\}and the𝐕\\mathbf\{V\}matrix, encapsulating the weighted significance of each value element based on the derived causal relationships\. This product is then fed into a fully connected layer to produce the final processed tokenX^′\\hat\{X\}^\{\\prime\}, which serves as input to the transformer model\.

X^′=FC​\(𝐀𝐢⊙𝐕\)\\hat\{X\}^\{\\prime\}=\\text\{FC\}\(\\mathbf\{A\_\{i\}\}\\odot\\mathbf\{V\}\)\(7\)

### 4\.2Global Causal Algorithm

Algorithm[1](https://arxiv.org/html/2607.28212#algorithm1)presents the global causal algorithm, which systematically identifies potential causal links in the dataset based on causal strengths derived from the attention mechanism\. This approach aggregates causal information from each token during training to output a set of potential causes𝐏\\mathbf\{P\}\. Specifically, the attention mechanism is applied to each token during training, generating a causal tensor𝐂i​j​t\\mathbf\{C\}\_\{ijt\}, wherettrepresents different tokens\. The algorithm then calculates the causal strengthsHi​jH\_\{ij\}between each pair of time seriesiiandjjby summing𝐂i​j​t\\mathbf\{C\}\_\{ijt\}across all tokens:

Hi​j=∑t𝐂i​jtH\_\{ij\}=\\sum\_\{t\}\\mathbf\{C\}\_\{ij\}^\{t\}\(8\)Next, the algorithm sorts the causal strengthsHi​j\{H\_\{ij\}\}in descending order and selects a thresholdτ\\taubased on a predefined parameterkk, whereτ\\tauis set as thekk\-th largest value in the sorted list\.

τ=sorted​\(Hi​j\)​\[k\]\\tau=\\text\{sorted\}\(\{H\_\{ij\}\}\)\[k\]\(9\)We select the thresholdτ\\taufollowing the approach in\[[10](https://arxiv.org/html/2607.28212#bib.bib7)\]\. If the causal strengthHi​jH\_\{ij\}between any two time seriesiiandjjmeets or exceedsτ\\tau, the algorithm infers a causal relationship fromiitojjand includes it in the set of potential causes𝐏\\mathbf\{P\}:

𝐏=\{\(i→j\)∣Hi​j≥τ\}\\mathbf\{P\}=\\\{\(i\\rightarrow j\)\\mid H\_\{ij\}\\geq\\tau\\\}\(10\)
Data:Causality tensor𝐂\\mathbf\{C\}, order parameterkk

Result:potential causes set

𝐏\\mathbf\{P\}
1Function*GlobalCausality\(*CC,kk*\)*:

2for*i=1i=1toNN*do

3for*j=1j=1toNN*do

4

Hi​j←∑t𝐂i​jtH\_\{ij\}\\leftarrow\\sum\_\{t\}\\mathbf\{C\}\_\{ij\}^\{t\};

5

6

7

h​s←sorted​\(\{Hi​j\}\)hs\\leftarrow\\textbf\{sorted\}\(\\\{H\_\{ij\}\\\}\);

8

τ←h​s​\[k\]\\tau\\leftarrow hs\[k\];

9

𝐏←∅\\mathbf\{P\}\\leftarrow\\emptyset;

10for*i=1i=1toNN*do

11for*j=1j=1toNN*do

12if*Hi​j≥τH\_\{ij\}\\geq\\tau*then

13

𝐏←𝐏∪\{\(i→j\)\}\\mathbf\{P\}\\leftarrow\\mathbf\{P\}\\cup\\\{\(i\\rightarrow j\)\\\};

14

15

16

17return

𝐏\\mathbf\{P\};

18

Algorithm 1Global Causal Algorithm
### 4\.3Causal Verification

Our methodology uses Permutation Importance \(PI\)\[[12](https://arxiv.org/html/2607.28212#bib.bib23)\]to validate causal relationships\. PI quantifies the effect of variable reordering on prediction error by shuffling the values of a candidate cause variable, disrupting potential causal pathways and assessing the significance of the temporal structure\.

Given a neural network𝒩j\\mathcal\{N\}\_\{j\}trained on the complete dataset to predict the target time series𝐗j\\mathbf\{X\}\_\{j\}, we denote the baseline loss on the original data asℒG\\mathcal\{L\}\_\{G\}\. For each potential causal factor𝐗i∈𝐏\\mathbf\{X\}\_\{i\}\\in\\mathbf\{P\}, we create a permuted dataset by randomly rearranging the values of𝐗i\\mathbf\{X\}\_\{i\}while keeping the rest of the data unchanged\. This permutation preserves the original data distribution, allowing us to re\-evaluate the prediction for𝐗j\\mathbf\{X\}\_\{j\}without retraining the network\. The perturbed dataset produces a new loss metricℒI\\mathcal\{L\}\_\{I\}\. A significant increase in loss fromℒG\\mathcal\{L\}\_\{G\}toℒI\\mathcal\{L\}\_\{I\}suggests a strong causal influence of𝐗i\\mathbf\{X\}\_\{i\}on𝐗j\\mathbf\{X\}\_\{j\}\. If the increase is minor, it indicates that the temporal ordering of𝐗i\\mathbf\{X\}\_\{i\}is not crucial for predicting𝐗j\\mathbf\{X\}\_\{j\}, questioning its causal role\. The set of validated causes𝐂j\\mathbf\{C\}\_\{j\}for each time series is determined through this process, forming the basis for constructing the Granger causality diagramG​\(V,E\)G\(V,E\), whereVVrepresents the time series nodes andEEthe directed edges validated by the causal sets\.

## 5Experiments

### 5\.1Experimental Setup

Our implementation was based on Python 3\.8\.18 and PyTorch 1\.12\.1, with experiments conducted on a server equipped with two NVIDIA A40 GPUs\. We employed Mean Squared Error as the loss function and utilized the Adam optimizer with a learning rate of 0\.001, modified by a scheduler with a step size of 10 and gamma of 0\.98\. Hyper parameters were set through preliminary tuning, with a batch size of 32 for 500 epochs\. To reduce overfitting, dropout was applied\. Input and output window sizes were configured to 10 and 1, respectively, while both causal attention and transformer modules employed 8 heads and 3 layers\.

##### Datasets

Our experiments used linear and non\-linear multivariate time series datasets, as summarized below\.Linear Datasets:1\)Hénon Maps:Chaotic time series data from 6 interconnected Hénon maps, totaling 2,048 data points for training\[[6](https://arxiv.org/html/2607.28212#bib.bib41)\]\. 2\)FINANCE:A 25\-variable financial time series dataset\[[5](https://arxiv.org/html/2607.28212#bib.bib43)\], utilized for evaluating causal discovery algorithms\.Non\-linear Datasets:1\)Lorenz\-96 Model:A 10\-variable atmospheric dynamics model generating 2,048 samples\[[7](https://arxiv.org/html/2607.28212#bib.bib42)\]\. 2\)fMRI:Simulated BOLD signals derived from dynamic causal modeling for fMRI data, with 1,200 samples for analysis\[[21](https://arxiv.org/html/2607.28212#bib.bib25)\]\.

##### Baseline Methods

We compared our method with six standard models in Granger causal discovery:BGranger\[[4](https://arxiv.org/html/2607.28212#bib.bib1)\],KGC\[[8](https://arxiv.org/html/2607.28212#bib.bib4)\],tsFCI\[[3](https://arxiv.org/html/2607.28212#bib.bib31)\],TCDF\[[10](https://arxiv.org/html/2607.28212#bib.bib7)\],PCMCI\[[14](https://arxiv.org/html/2607.28212#bib.bib30)\], andDYNOTEARS\[[11](https://arxiv.org/html/2607.28212#bib.bib56)\]\. These methods represent diverse approaches to causal discovery\. BGranger provides a foundational benchmark by testing if past values of one variable improve predictions of another\. Constraint\-based methods like tsFCI and PCMCI use conditional independence tests to infer causal structures\. KGC, a kernel\-based method, extends Granger causality to non\-linear relationships\. TCDF enhances inference with attention mechanisms, and DYNOTEARS, an optimization\-based method, applies continuous optimization for dynamic causal discovery\.

##### Evaluation Metrics

To evaluate performance, we usePrecision\(P\)\(P\),Recall\(R\)\(R\), andF1\-Score\(F1\)\(F\_\{1\}\)to measure accuracy\. Precision represents the fraction of correctly identified causal relationships among those inferred, Recall indicates the fraction of actual causal relationships detected, and F1\-Score provides a balanced measure between Precision and Recall\.

Table 1:Performance comparison of the proposed method and baselines\.![Refer to caption](https://arxiv.org/html/2607.28212v1/x3.png)Figure 3:Comparison of causality matrices from various methods on the fMRI dataset, where dark blue indicates true causality and red highlights errors\.

### 5\.2Overall Performance

The quantitative results in Table[1](https://arxiv.org/html/2607.28212#S5.T1)demonstrate the effectiveness of our causal discovery framework across different datasets\. Our method consistently achieves high precision scores, from 0\.673 for the FINANCE dataset to 0\.900 for the Hénon dataset, indicating its strong ability to minimize false\-positive causal inferences\. Furthermore, our approach maintains competitive recall scores, showing its capacity to capture a significant portion of the ground\-truth causal relationships\. Notably, our method outperforms existing state\-of\-the\-art techniques on the non\-linear Lorenz\-96 and fMRI datasets, achieving the highest F1\-scores of 0\.769 and 0\.837, respectively\. This performance underscores the robustness of our framework in uncovering complex causal structures within non\-linear systems, a key requirement for real\-world applications\.

The visual comparison in Fig\.[3](https://arxiv.org/html/2607.28212#S5.F3)further supports the quantitative results\. The inferred causality matrix from our method closely matches the ground truth, with minimal false positives and negatives\. This accuracy is due to the combined effects of the proposed CSAM, the sparsity\-inducing modifications, and the global integration algorithm\. CSAM captures complex, non\-linear causal dependencies by leveraging self\-attention mechanisms within the transformer architecture\. The sparsity\-inducing modifications enhance the reliability of identified causal relationships by reducing spurious correlations and noise\. Finally, the global integration algorithm provides a holistic metric for causal influence, facilitating the discovery of both direct and indirect causal links while accounting for temporal dynamics and inter\-dependencies among variables\.

![Refer to caption](https://arxiv.org/html/2607.28212v1/x4.png)\(a\)Precision
![Refer to caption](https://arxiv.org/html/2607.28212v1/x5.png)\(b\)Recall
![Refer to caption](https://arxiv.org/html/2607.28212v1/x6.png)\(c\)F1\-Score

Figure 4:Ablation study results across datasets\.
### 5\.3Ablation Study

To evaluate the contributions of our model’s components, we conducted an ablation study focusing on the CSAM and sparsity\-inducing mechanism\. Our study compares the performance of the complete model \(Ours\-full\) with two variants:Ours\-n2nandOurs\-soft\. TheOurs\-n2nvariant, which predicts all elements of the multivariate time series simultaneously, shows reduced precision compared toOurs\-full, indicating that parallel prediction disperses focus and weakens causal detection, as shown in Fig\.[4\(a\)](https://arxiv.org/html/2607.28212#S5.F4.sf1)\. TheOurs\-softvariant, replacing SparseMax with Softmax for attention scoring, also shows a precision drop across datasets, as Softmax fails to create the desired sparsity, reducing specificity in causal inference\. As seen in Fig\.[4\(b\)](https://arxiv.org/html/2607.28212#S5.F4.sf2), these variants affect recall as well\. WhileOurs\-n2nperforms similarly toOurs\-fullin some datasets, it underperforms in complex cases like Lorenz\-96 and fMRI\.Ours\-softalso exhibits inconsistencies, especially in datasets where sparsity is key to accurate causal detection\. The F1\-scores in Fig\.[4\(c\)](https://arxiv.org/html/2607.28212#S5.F4.sf3)confirm thatOurs\-fulloutperforms the variants, demonstrating the importance of both CSAM and sparsity\-inducing mechanisms\.

### 5\.4Sensitivity Analysis

The CSAM, rooted in the transformer framework, inherently relies on multiple attention heads and blocks, both of which are crucial hyper\-parameters in the model’s architecture\. In our parameter analysis, the number of Attention Heads was varied among 2, 4, 8, and 16, with 8 set as the default value\. Similarly, the number of Attention Blocks was adjusted to 1, 2, 3, and 4, with 3 being the default\.

Table 2:Sensitivity analysis\.
Number of Attention Heads:In the CSAM, the number of attention heads controls how many attention mechanisms are applied to the input data\. Each head processes the sequence separately, allowing the model to capture various dependencies\. As shown in Table[2](https://arxiv.org/html/2607.28212#S5.T2), increasing attention heads to 16 helps capture more complex dependencies but may cause over\-fitting\. Conversely, reducing heads to 2 simplifies the model, potentially losing important information\.

Number of Attention Blocks:Attention blocks determine the depth of attention applied across the time series data, with each block adding a layer of processing to refine the model’s understanding\. Table[2](https://arxiv.org/html/2607.28212#S5.T2)shows that increasing blocks to 4 enhances the model’s ability to capture complex dependencies but risks over\-fitting, while reducing blocks to 1 may fail to capture sufficient details\.

## 6Conclusion and Future Work

This study introduces a novel framework for causal discovery in multivariate time series data\. Our proposed inverted causal self\-attention mechanism \(CSAM\) provides a unique token representation to analyze causal relationships\. By integrating a global causal algorithm with causal verification, the framework effectively handles spurious correlations\. Experiments on four benchmarks demonstrate that our method outperforms existing approaches\. For future research, we aim to extend the method to complex datasets across different domains and explore approaches focused on identifying true causal frequencies and causal lags\.

\{credits\}

#### 6\.0\.1Acknowledgements

This work was supported by the China Scholarships Council \(Grant No\. 202208410132\), and the Tianfu Yongxing Laboratory Organized Research Project Funding \(No\. 2023CXXM14\)\.

## References

- \[1\]D\. Bahdanau, K\. Cho, and Y\. Bengio\(2015\)Neural machine translation by jointly learning to align and translate\.ICLR\.Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]I\. Ebert\-Uphoff and Y\. Deng\(2012\)Causal discovery for climate research using graphical models\.Journal of Climate25\(17\),pp\. 5648–5665\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1)\.
- \[3\]D\. Entner and P\. O\. Hoyer\(2010\)On causal discovery from time series data using fci\.Probabilistic graphical models,pp\. 121–128\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[4\]C\. W\. Granger\(1969\)Investigating causal relations by econometric models and cross\-spectral methods\.Econometrica: journal of the Econometric Society,pp\. 424–438\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[5\]S\. Kleinberg\(2013\)Causality, probability, and time\.Cambridge University Press\.Cited by:[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px1.p1.1)\.
- \[6\]D\. Kugiumtzis\(2013\)Direct\-coupling information measure from nonuniform embedding\.Physical Review E87\(6\),pp\. 062918\.Cited by:[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px1.p1.1)\.
- \[7\]E\. N\. Lorenz\(1996\)Predictability: a problem partly solved\.InProc\. of Seminar on predictability,Vol\.1\.Cited by:[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px1.p1.1)\.
- \[8\]D\. Marinazzo, M\. Pellicoro, and S\. Stramaglia\(2008\)Kernel method for nonlinear granger causality\.Physical review letters100\(14\),pp\. 144103\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[9\]A\. Martins and R\. Astudillo\(2016\)From softmax to sparsemax: a sparse model of attention and multi\-label classification\.InICML,pp\. 1614–1623\.Cited by:[§4\.1](https://arxiv.org/html/2607.28212#S4.SS1.p4.1)\.
- \[10\]M\. Nauta, D\. Bucur, and C\. Seifert\(2019\)Causal discovery with attention\-based convolutional neural networks\.Machine Learning and Knowledge Extraction1\(1\)\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2607.28212#S4.SS2.p1.20),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[11\]R\. Pamfil, N\. Sriwattanaworachai, S\. Desai, P\. Pilgerstorfer, K\. Georgatzis, P\. Beaumont, and B\. Aragam\(2020\)Dynotears: structure learning from time\-series data\.InAISTATS,pp\. 1595–1605\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[12\]J\. P\. Pereira, E\. S\. Stroes, A\. H\. Zwinderman, and E\. Levin\(2022\)Covered information disentanglement: model transparency via unbiased permutation importance\.InProc\. of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 7984–7992\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p4.1),[§4\.3](https://arxiv.org/html/2607.28212#S4.SS3.p1.1)\.
- \[13\]J\. Runge, S\. Bathiany, E\. Bollt, G\. Camps\-Valls, D\. Coumou, E\. Deyle, C\. Glymour, M\. Kretschmer, M\. D\. Mahecha, J\. Muñoz\-Marí,et al\.\(2019\)Inferring causation from time series in earth system sciences\.Nature communications10\(1\),pp\. 2553\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p2.1)\.
- \[14\]J\. Runge, P\. Nowack, M\. Kretschmer, S\. Flaxman, and D\. Sejdinovic\(2019\)Detecting and quantifying causal associations in large nonlinear time series datasets\.Science advances5\(11\),pp\. eaau4996\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4),[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px2.p1.1)\.
- \[15\]J\. Runge\(2018\)Causal network reconstruction from time series: from theoretical assumptions to practical estimation\.Chaos: An Interdisciplinary Journal of Nonlinear Science28\(7\)\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p2.1)\.
- \[16\]T\. Schreiber\(2000\)Measuring information transfer\.Physical review letters85\(2\)\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4)\.
- \[17\]S\. Shih, F\. Sun, and H\. Lee\(2019\)Temporal pattern attention for multivariate time series forecasting\.Machine Learning108,pp\. 1421–1441\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p2.1)\.
- \[18\]P\. Spirtes and C\. Glymour\(1991\)An algorithm for fast recovery of sparse causal graphs\.Social science computer review9\(1\),pp\. 62–72\.Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4)\.
- \[19\]A\. Tank, I\. Covert, N\. Foti, A\. Shojaie, and E\. B\. Fox\(2021\)Neural granger causality\.IEEE TPAMI44\(8\),pp\. 4267–4279\.Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4)\.
- \[20\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.NeurIPS30\.Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px2.p1.1)\.
- \[21\]Y\. Yan, L\. Dahmani, J\. Ren, L\. Shen, X\. Peng, R\. Wang, C\. He, C\. Jiang, C\. Gong, Y\. Tian,et al\.\(2020\)Reconstructing lost bold signal in individual participants using deep machine learning\.Nature communications11\(1\),pp\. 5046\.Cited by:[§5\.1](https://arxiv.org/html/2607.28212#S5.SS1.SSS0.Px1.p1.1)\.
- \[22\]S\. M\. Zema\(2022\)Directed acyclic graph based information shares for price discovery\.Journal of Economic Dynamics and Control139,pp\. 104434\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1)\.
- \[23\]Y\. Zhang and J\. Yan\(2022\)Crossformer: transformer utilizing cross\-dimension dependency for multivariate time series forecasting\.InICML,Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]X\. Zheng, B\. Aragam, P\. K\. Ravikumar, and E\. P\. Xing\(2018\)Dags with no tears: continuous optimization for structure learning\.NeurIPS31\.Cited by:[§1](https://arxiv.org/html/2607.28212#S1.p3.1),[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px1.p1.4)\.
- \[25\]H\. Zhou, S\. Zhang, J\. Peng, S\. Zhang, J\. Li, H\. Xiong, and W\. Zhang\(2021\)Informer: beyond efficient transformer for long sequence time\-series forecasting\.InProc\. of the AAAI conference on artificial intelligence,Vol\.35,pp\. 11106–11115\.Cited by:[§2](https://arxiv.org/html/2607.28212#S2.SS0.SSS0.Px2.p1.1)\.