X-LogSMask: Expand Transformer for Graph-Structured Data

# X-LogSMask: Expand Transformer for Graph-Structured Data
Source: [https://arxiv.org/html/2607.01553](https://arxiv.org/html/2607.01553)
Leyan Li &Rennong Yang &Zhenxing Zhang &Liping Hu

###### Abstract

Transformers have become general\-purpose architectures, but their all\-to\-all self\-attention is poorly matched to graph data, whose interactions are sparse, structured and multi\-scale\. Existing Graph Transformers address this mismatch through structural encodings, hybrid message\-passing modules or learned attention constraints, often introducing additional complexity and limited interpretability\. Here we introduce X\-LogSMask, an explainable multi\-headlogarithmicstructuralmaskthat injects symmetrically normalized graph topology directly into attention logits\. The logarithmic transform converts structural connectivity into a topology\-aware gating signal, suppressing unsupported node interactions while preserving feature\-dependent attention\. By assigning different powers of the normalized adjacency matrix to different attention heads, X\-LogSMask gives each head a defined structural radius and supports multi\-hop information propagation within a single layer\. We further show that a standard Transformer encoder can be interpreted as one\-step message passing on a complete graph, motivating X\-LogSMask as a topology\-constrained alternative to unrestricted self\-attention\. Across 20 node\-, edge\- and graph\-level benchmarks, Transformers equipped with X\-LogSMask achieve state\-of\-the\-art performance on 13 datasets and remain competitive in a lightweight one\-layer configuration\. These results show that simple, interpretable structural masks can make self\-attention an effective graph\-learning operator without changing the Transformer architecture\. The code is available at[https://github\.com/LiLeyan\-0120/X\-LogSMask](https://github.com/LiLeyan-0120/X-LogSMask)\.

*Keywords*Graph transformer, graph neural network, explainable learning, logarithmic structural mask, multi\-hop attention

## 1Introduction

Graphs provide a natural representation for relational systems, including citation networks, social networks, molecules, biological systems and transportation networks\. Unlike sequences or images, graphs are defined by irregular neighborhoods, sparse connectivity and dependencies that may span multiple topological scales\. Graph neural networks \(GNNs\) address this structure by propagating information along observed edges, thereby aligning model computation with graph topology\[[16](https://arxiv.org/html/2607.01553#bib.bib9),[34](https://arxiv.org/html/2607.01553#bib.bib10)\]\. By contrast, the standard Transformer was developed around self\-attention, which permits every token to interact with every other token\[[29](https://arxiv.org/html/2607.01553#bib.bib3)\]\. This design gives Transformers strong representational flexibility, but it also creates a mismatch when the input is a graph\.

![Refer to caption](https://arxiv.org/html/2607.01553v1/x1.png)Figure 1:Transformer for all kinds of data\.Transformers have been successfully extended beyond language to vision, temporal modelling and other structured domains\[[9](https://arxiv.org/html/2607.01553#bib.bib4),[3](https://arxiv.org/html/2607.01553#bib.bib6),[10](https://arxiv.org/html/2607.01553#bib.bib11),[39](https://arxiv.org/html/2607.01553#bib.bib13),[13](https://arxiv.org/html/2607.01553#bib.bib12)\]\. Their success suggests that self\-attention can serve as a general\-purpose information aggregation operator when equipped with an appropriate inductive bias\. In sequential data, this role is played by positional encodings, which supply the order information absent from permutation\-invariant attention\. For graph data, however, the required inductive bias is not a one\-dimensional position but a sparse, relational and multi\-hop topology\. Without such a bias, all\-to\-all self\-attention can mix topologically unrelated nodes and weaken the structural constraints that make message passing effective\.

Recent Graph Transformer models have therefore sought to inject graph structure into attention\. One line of work incorporates attention mechanisms into GNN architectures, as in Graph Attention Networks\[[30](https://arxiv.org/html/2607.01553#bib.bib14)\]\. Another combines message\-passing modules with Transformer blocks in hybrid pipelines\[[38](https://arxiv.org/html/2607.01553#bib.bib18),[24](https://arxiv.org/html/2607.01553#bib.bib19)\]\. A third line directly modifies the attention mechanism through positional encodings, structural biases or learned attention constraints, including Graphormer, GradFormer and Eigenformer\[[35](https://arxiv.org/html/2607.01553#bib.bib15),[19](https://arxiv.org/html/2607.01553#bib.bib16),[12](https://arxiv.org/html/2607.01553#bib.bib17)\]\. These methods have shown that structural priors are important for graph\-aware self\-attention\. However, they often introduce additional architectural complexity, rely on implicit structural effects, or provide limited control over what each attention head captures\.

Here we introduce X\-LogSMask, an explainable multi\-head logarithmic structural mask for adapting Transformers to graph\-structured data\. X\-LogSMask is constructed from a symmetrically normalized adjacency matrix with self\-loops and is injected additively into the attention logits\. The logarithmic transform converts normalized structural connectivity into a topology\-aware gating signal in attention space\. As a result, unsupported node interactions are strongly suppressed, while feature\-dependent attention scores remain discriminative\. This design preserves the core Transformer architecture, but replaces unrestricted communication with topology\-constrained message passing\. X\-LogSMask further assigns different powers of the normalized adjacency matrix to different attention heads\. Each head is therefore associated with a defined structural radius, ranging from local neighborhoods to higher\-order graph context\. This head\-wise decomposition gives multi\-head attention an explicit structural interpretation and allows multi\-hop information propagation within a single Transformer layer\. It also reduces the need to stack many message\-passing layers, which can aggravate over\-smoothing and over\-squashing in conventional GNNs\.

Table 1:Summary of the Representative Graph Transformer ModelsModelYearGraph Structural EncodingFormulaic RepresentationGraphormer\[[35](https://arxiv.org/html/2607.01553#bib.bib15)\]2021Central, spatial, and edge encodingαi​j∝exp⁡\(\(𝐡i​𝐖Q\)​\(𝐡j​𝐖K\)Td\+bϕ​\(vi,vj\)\+1N​∑n=1N𝐱en​\(𝐰nE\)T\)\\alpha\_\{ij\}\\propto\\exp\\\!\\left\(\\frac\{\(\\mathbf\{h\}\_\{i\}\\mathbf\{W\}^\{Q\}\)\(\\mathbf\{h\}\_\{j\}\\mathbf\{W\}^\{K\}\)^\{\\mathrm\{T\}\}\}\{\\sqrt\{d\}\}\+b\_\{\\phi\(v\_\{i\},v\_\{j\}\)\}\+\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbf\{x\}\_\{e\_\{n\}\}\(\\mathbf\{w\}\_\{n\}^\{E\}\)^\{\\mathrm\{T\}\}\\right\)Gradformer\[[19](https://arxiv.org/html/2607.01553#bib.bib16)\]2024Decay mask with learnable constraintsαi​j∝exp⁡\(𝐪i​𝐤jTd⋅λReLU​\(ψ​\(vi,vj\)−s​p\)\)\\alpha\_\{ij\}\\propto\\exp\\\!\\left\(\\frac\{\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\mathrm\{T\}\}\}\{\\sqrt\{d\}\}\\cdot\\lambda^\{\\mathrm\{ReLU\}\(\\psi\(v\_\{i\},v\_\{j\}\)\-sp\)\}\\right\)Eigenformer\[[12](https://arxiv.org/html/2607.01553#bib.bib17)\]2024Structural attention biasαi​j∝exp⁡\(𝐪i​𝐤jTd\+β​∑k=1K𝐮k​\(i\)​𝐮k​\(j\)λk\)\\alpha\_\{ij\}\\propto\\exp\\\!\\left\(\\frac\{\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\mathrm\{T\}\}\}\{\\sqrt\{d\}\}\+\\beta\\sum\_\{k=1\}^\{K\}\\frac\{\\mathbf\{u\}\_\{k\}\(i\)\\mathbf\{u\}\_\{k\}\(j\)\}\{\\lambda\_\{k\}\}\\right\)Ours2025X\-LogSMaskαi​j∝exp⁡\(𝐪i​𝐤jT\+log⁡\(𝐃−1/2​𝐆head​𝐃−1/2\+ϵ\)d\)\\alpha\_\{ij\}\\propto\\exp\\\!\\left\(\\frac\{\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\mathrm\{T\}\}\+\\log\\\!\\left\(\\mathbf\{D\}^\{\-1/2\}\\mathbf\{G\}\_\{\\mathrm\{head\}\}\\mathbf\{D\}^\{\-1/2\}\+\\epsilon\\right\)\}\{\\sqrt\{d\}\}\\right\)Our contributions are as follows:

1. 1\.We propose X\-LogSMask, a compact and theoretically motivated structural bias that replaces positional encodings in the vanilla Transformer\. X\-LogSMask is constructed from a symmetrically normalized adjacency matrix and a logarithmic transform, and is injected additively into the attention logits to suppress irrelevant communications while preserving the discriminative scaling properties of attention\.
2. 2\.An explainable multi\-head mechanism is designed for LogSMask, where different heads attend to different powers of the normalized adjacency matrix\. This head\-wise decomposition enables each attention head to specialize in a specific structural radius, accelerating multi\-hop information propagation within a single layer\. Thus, our method supports an efficient 1\-layer Transformer solution that is practical for large\-scale graph tasks\.

The article is organized as follows\. Section II distinguishes X\-LogSMask from other Graph Transformer designs\. Section III establishes the theoretical foundation that links the standard Transformer encoder to message passing\. Section IV details the architectural design of X\-LogSMask\. Section V validates the model through experiments on node\-, edge\-, and graph\-level benchmarks\. Section VI concludes the article and discusses future research directions\.

## 2What’s the difference between ours and others

The Introduction section outlined three principal paradigms for adapting Transformers to graph\-structured data; our approach belongs to the third paradigm, which directly augments the self\-attention mechanism with graph\-derived structural biases\. To clarify distinctions among methods in this family, Table[1](https://arxiv.org/html/2607.01553#S1.T1)summarizes representative structural\-encoding strategies and their formal instantiations\.

Broadly speaking, existing methods fall into two conceptual classes\. Additive approaches \(e\.g\., Graphormer\[[35](https://arxiv.org/html/2607.01553#bib.bib15)\]and Eigenformer\[[12](https://arxiv.org/html/2607.01553#bib.bib17)\]\) incorporate structural encodings by adding bias terms to the attention logits, thereby preserving topological information in a form that is straightforwardly learnable\. Multiplicative approaches \(e\.g\., GradFormer\[[19](https://arxiv.org/html/2607.01553#bib.bib16)\]\) instead scale the attention scores, directly modulating message\-passing intensity and providing an effective mechanism to suppress spurious interactions between distant or non\-adjacent nodes\. Each strategy has complementary advantages: additive biases facilitate rich structural representation and gradient\-based learning, whereas multiplicative biases exert stronger, direct control over inter\-node communication\.

X\-LogSMask is additive in formulation but leverages a minimal\-value logarithmic masking scheme that unifies the strengths of both paradigms\. By applying a log\-transformed, symmetrically normalized adjacency\-derived mask to the attention logits and assigning head\-specific powers of the adjacency to different heads, X\-LogSMask \(i\) retains the representational flexibility of additive encodings, and \(ii\) enforces selective attenuation of irrelevant connections akin to multiplicative bias\. This hybrid behavior enables explicit regulation of message\-passing strength while maintaining efficient structural encoding\.

Moreover, unlike prior methods that require deep stacking to capture long\-range dependencies, our mask embeds multi\-hop structural cues within a single layer via head\-wise specialization: lower\-order heads focus on local neighborhoods while higher\-order heads capture progressively longer\-range interactions\. This design accelerates information propagation, reduces variance introduced by unconstrained multi\-head attention, and yields both improved training efficiency and empirical performance\.

In summary, compared to competing designs, X\-LogSMask offers a low\-complexity, architecturally conservative modification to the Transformer that \(i\) preserves the original attention mechanism, \(ii\) injects interpretable, multi\-scale structural priors, and \(iii\) scales practically to a range of graph tasks\. These characteristics make it a flexible building block for extending Transformer architectures to other structured modalities, including sequential and visual domains\.

## 3Preliminary

Table 2:Correspondence between MPNN and Transformer encoder operations\.MPNNFormulaic representationTransformerImplementationMessage generationmi​j\(l\)=ϕ\(l\)​\(𝐡i\(l−1\),𝐡j\(l−1\),ei​j\)m\_\{ij\}^\{\(l\)\}=\{\\phi^\{\(l\)\}\}\\left\(\{\{\\bf\{h\}\}\_\{i\}^\{\(l\-1\)\},\{\\bf\{h\}\}\_\{j\}^\{\(l\-1\)\},\{e\_\{ij\}\}\}\\right\)Self\-attention calculationϕ\(l\)​\(⋅,⋅,1\)=softmax​\(𝐪i​𝐤jTd\)⋅𝐯j\{\\phi^\{\(l\)\}\}\(\\cdot,\\cdot,1\)=\{\\rm\{softmax\}\}\\left\(\{\\frac\{\{\{\{\\bf\{q\}\}\_\{i\}\}\{\\bf\{k\}\}\_\{j\}^\{\\rm\{T\}\}\}\}\{\{\\sqrt\{d\}\}\}\}\\right\)\\cdot\{\{\\bf\{v\}\}\_\{j\}\}Aggregation𝐡i\(l\),agg=⊕j∈𝒩​\(i\)mi​j\(l\)\{\\bf\{h\}\}\_\{i\}^\{\(l\),\{\\rm\{agg\}\}\}=\{\\oplus\_\{j\\in\{\\cal N\}\(i\)\}\}m\_\{ij\}^\{\(l\)\}Attention output⊕j∈𝒩​\(i\)=LN\(∑j∈𝒱mi​j\(l\)\+𝐡i\(l−1\)\)\{\\oplus\_\{j\\in\{\\cal N\}\(i\)\}\}=\{\\rm\{LN\}\}\(\\sum\\limits\_\{j\\in\{\\cal V\}\}\{m\_\{ij\}^\{\(l\)\}\}\+\{\\bf\{h\}\}\_\{i\}^\{\(l\-1\)\}\)Node update𝐡i\(l\)=ψ\(l\)​\(𝐡i\(l−1\),𝐡i\(l\),agg\)\{\\bf\{h\}\}\_\{i\}^\{\(l\)\}=\{\\psi^\{\(l\)\}\}\(\{\\bf\{h\}\}\_\{i\}^\{\(l\-1\)\},\{\\bf\{h\}\}\_\{i\}^\{\(l\),\{\\rm\{agg\}\}\}\)FFN outputψ\(l\)=LN​\(𝐡i\(l\),agg\+FFN​\(𝐡i\(l\),agg\)\)\{\\psi^\{\(l\)\}\}=\{\\rm\{LN\}\}\(\{\\bf\{h\}\}\_\{i\}^\{\(l\),\{\\rm\{agg\}\}\}\+\{\\rm\{FFN\}\}\(\{\\bf\{h\}\}\_\{i\}^\{\(l\),\{\\rm\{agg\}\}\}\)\)###### Theorem 1\.

The standard Transformer encoder layer implements a one\-step message\-passing operator on the complete graph𝒢c=\(𝒱,ℰc\)\\mathcal\{G\}\_\{c\}=\(\\mathcal\{V\},\\mathcal\{E\}\_\{c\}\)withℰc=𝒱×𝒱\\mathcal\{E\}\_\{c\}=\\mathcal\{V\}\\times\\mathcal\{V\}, and thus constitutes a special case of message passing neural networks \(MPNN\)\.

###### Proof\.

Consider the standard formulation of an MPNN layer, which is defined by a message function, a permutation\-invariant aggregation operator, and an update function\. At layerll, the message sent from nodejjto nodeiiis given by

mi​j\(l\)=ϕ\(l\)​\(𝐡i\(l−1\),𝐡j\(l−1\),ei​j\)m\_\{ij\}^\{\(l\)\}=\\phi^\{\(l\)\}\\left\(\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\},\\mathbf\{h\}\_\{j\}^\{\(l\-1\)\},e\_\{ij\}\\right\)\(1\)where𝐡i\(l−1\)\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\}denotes the representation of nodeiifrom the preceding layer andei​je\_\{ij\}denotes the edge attribute\.

The aggregated representation is

𝐡i,agg\(l\)=⨁j∈𝒩​\(i\)mi​j\(l\)\\mathbf\{h\}\_\{i,\\mathrm\{agg\}\}^\{\(l\)\}=\\bigoplus\_\{j\\in\\mathcal\{N\}\(i\)\}m\_\{ij\}^\{\(l\)\}\(2\)
The layer output is obtained through

𝐡i\(l\)=ψ\(l\)​\(𝐡i\(l−1\),𝐡i,agg\(l\)\)\\mathbf\{h\}\_\{i\}^\{\(l\)\}=\\psi^\{\(l\)\}\\left\(\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\},\\mathbf\{h\}\_\{i,\\mathrm\{agg\}\}^\{\(l\)\}\\right\)\(3\)
For a Transformer encoder layer applied to the same node set, define𝐪i=𝐡i\(l−1\)​𝐖Q\\mathbf\{q\}\_\{i\}=\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\}\\mathbf\{W\}^\{Q\},𝐤j=𝐡j\(l−1\)​𝐖K\\mathbf\{k\}\_\{j\}=\\mathbf\{h\}\_\{j\}^\{\(l\-1\)\}\\mathbf\{W\}^\{K\}, and𝐯j=𝐡j\(l−1\)​𝐖V\\mathbf\{v\}\_\{j\}=\\mathbf\{h\}\_\{j\}^\{\(l\-1\)\}\\mathbf\{W\}^\{V\}\. The contribution of tokenjjto tokeniican then be expressed as an attention\-weighted message,

mi​j\(l\)=αi​j​𝐯j,αi​j=exp⁡\(𝐪i​𝐤jT/d\)∑r∈𝒱exp⁡\(𝐪i​𝐤rT/d\)m\_\{ij\}^\{\(l\)\}=\\alpha\_\{ij\}\\mathbf\{v\}\_\{j\},\\quad\\alpha\_\{ij\}=\\frac\{\\exp\\left\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\mathrm\{T\}\}/\\sqrt\{d\}\\right\)\}\{\\sum\_\{r\\in\\mathcal\{V\}\}\\exp\\left\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{r\}^\{\\mathrm\{T\}\}/\\sqrt\{d\}\\right\)\}\(4\)
Accordingly, the self\-attention output for tokeniiis

𝐳i\(l\)=∑j∈𝒱mi​j\(l\)\\mathbf\{z\}\_\{i\}^\{\(l\)\}=\\sum\_\{j\\in\\mathcal\{V\}\}m\_\{ij\}^\{\(l\)\}\(5\)
This expression is identical in form to MPNN aggregation when the neighborhood of each node is taken to be the full vertex set, that is,𝒩​\(i\)=𝒱\\mathcal\{N\}\(i\)=\\mathcal\{V\}for allii\. This choice corresponds precisely to message passing on the complete graph𝒢c=\(𝒱,𝒱×𝒱\)\\mathcal\{G\}\_\{c\}=\(\\mathcal\{V\},\\mathcal\{V\}\\times\\mathcal\{V\}\)\. The subsequent residual connection, layer normalization, and feed\-forward network implement an update map of the formψ\(l\)\\psi^\{\(l\)\}\. Therefore, a standard Transformer encoder layer can be regarded as an MPNN layer instantiated on𝒢c\\mathcal\{G\}\_\{c\}\. A step\-by\-step correspondence is provided in Table[2](https://arxiv.org/html/2607.01553#S3.T2)\.

###### Lemma 1\.

The fully\-connected nature of the Transformer arises from the undiscriminating message passing among tokens induced by the self\-attention mechanism\.

###### Proof\.

By construction, the softmax attention produces non\-negative, normalized coefficientsαi​j\\alpha\_\{ij\}for every pair\(i,j\)\(i,j\), where the normalization for fixediisums over allj∈𝒱j\\in\\mathcal\{V\}\. Consequently, the aggregated message for nodeiiis

𝐦i=∑j∈𝒱αi​j​𝐯j\\mathbf\{m\}\_\{i\}=\\sum\_\{j\\in\\mathcal\{V\}\}\\alpha\_\{ij\}\\mathbf\{v\}\_\{j\}\(6\)which includes contributions from every token in the input set\. In the MPNN view, this means the receptive neighborhood𝒩​\(i\)\\mathcal\{N\}\(i\)equals the entire vertex set𝒱\\mathcal\{V\}for allii, i\.e\., the induced communication graph is complete\. This absence of structural constraint on permitted message sources—each node both sends and receives messages to/from every other node—gives rise to the Transformer’s effective full connectivity\. ∎

![Refer to caption](https://arxiv.org/html/2607.01553v1/x2.png)Figure 2:Model architecture and structural masking mechanism of X\-LogSMask\. \(a\) Graph Transformer attention with structural bias\. Node features are linearly projected to query, key and value representations; the scaled query\-key energy is combined with a multi\-head X\-LogSMask before softmax, thereby constraining value aggregation by graph topology\. \(b\) Topology\-to\-mask construction\. The graph adjacency is augmented with self\-loops and symmetrically normalized to form𝐀^\\hat\{\\mathbf\{A\}\}\. A logarithmic transform converts normalized structural weights into attention\-logit biases, and powers of𝐀^\\hat\{\\mathbf\{A\}\}yield head\-specific masks from 1\-hop local structure to 4\-hop global context\. \(c\) Fusion mechanism\. Learned node\-information energy and graph\-information masks are added head by head, producing fused logits that couple feature\-dependent attention with multi\-hop structural priors\. \(d\) Multi\-hop context capture\. Conventional message\-passing GNNs expand the receptive field by stacking layers, which can aggravate over\-smoothing and over\-squashing\. X\-LogSMask assigns different hop ranges to different heads in a single Transformer layer, enabling parallel local\-to\-global message passing before concatenation and output\.
## 4Methodology

### 4\.1Model Architecture

Graph\-structured data are distinguished from one\- and two\-dimensional data by their non\-Euclidean connectivity patterns\. Let𝐀∈ℝn×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{n\\times n\}denote the graph adjacency matrix for a graph withnnnodes, where𝐀i​j=1\\mathbf\{A\}\_\{ij\}=1if nodesiiandjjare connected and𝐀i​j=0\\mathbf\{A\}\_\{ij\}=0otherwise\. Although𝐀\\mathbf\{A\}encodes critical topological priors, canonical neural architectures such as the Transformer do not natively incorporate such relational inductive biases\. A principal objective of this work is therefore to inject graph topology into the Transformer framework in a principled and computationally efficient manner\.

As illustrated in Fig\.[2](https://arxiv.org/html/2607.01553#S3.F2), we introduce X\-LogSMask \(Explainable Multi\-headLogarithmicStructuralMask\), a compact structural bias constructed from𝐀\\mathbf\{A\}and integrated additively into the attention logits\. X\-LogSMask modifies attention computation while preserving the Transformer’s core architecture and optimization benefits\. Compared to prior Graph Transformer designs, X\-LogSMask offers three key advantages:

1. 1\.It enforces topologically constrained message passing by suppressing attention between non\-adjacent nodes via an additive structural mask\.
2. 2\.It injects an explicit graph inductive bias directly into the attention matrix, improving the model’s sensitivity to graph topology\.
3. 3\.By assigning distinct powers of a normalized adjacency to different attention heads, it encodes multi\-hop structural information within a single layer, accelerating information propagation and reducing the randomness associated with vanilla multi\-head attention\.

### 4\.2X\-LogSMask

The standard Transformer’s all\-to\-all attention disregards graph topology and can induce excessive, uninformative communication across distant nodes\. X\-LogSMask remedies this by embedding structural constraints into the attention logits while maintaining the differentiable, end\-to\-end training behavior of the Transformer\.

#### 4\.2\.1Symmetric Normalization of the Adjacency Matrix

The raw adjacency matrix𝐀∈ℝn×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{n\\times n\}inherently suffers from uneven node degree distribution, which can distort graph signal propagation and bias message aggregation toward high\-degree nodes\. The phenomenon will be strengthened especially in the multi\-hop connections\. To stabilize propagation and mitigate degree\-related distortion, we adopt symmetric normalization with self\-loops:

𝐀~=𝐃−1/2​\(𝐀\+𝐈\)​𝐃−1/2\\tilde\{\\mathbf\{A\}\}=\\mathbf\{D\}^\{\-1/2\}\(\\mathbf\{A\}\+\\mathbf\{I\}\)\\mathbf\{D\}^\{\-1/2\}\(7\)where𝐈\\mathbf\{I\}is the identity matrix and𝐃\\mathbf\{D\}is the diagonal degree matrix with𝐃i​i=∑j\(𝐀i​j\+𝐈i​j\)\\mathbf\{D\}\_\{ii\}=\\sum\_\{j\}\(\\mathbf\{A\}\_\{ij\}\+\\mathbf\{I\}\_\{ij\}\)\. This normalization balances incoming and outgoing signals and improves numerical stability for subsequent matrix operations\.

#### 4\.2\.2Logarithmic Structural Mask

Directly using𝐀~\\tilde\{\\mathbf\{A\}\}as an additive attention bias would risk an excessive focus on strong connections while collapsing the influence of weaker, yet potentially informative, links\. To compress this dynamic range while preserving the relative ordering of connections, we apply a logarithmic transform\. The resulting structural mask𝐌\\mathbf\{M\}is defined element\-wise as:

𝐌i​j=log⁡\(𝐀~i​j\+ϵ\)\\mathbf\{M\}\_\{ij\}=\\log\(\\tilde\{\\mathbf\{A\}\}\_\{ij\}\+\\epsilon\)\(8\)whereϵ\\epsilonis a small constant \(e\.g\.,10−3010^\{\-30\}\) to ensure numerical stability for𝐀~i​j=0\\tilde\{\\mathbf\{A\}\}\_\{ij\}=0\.

Let𝐪i\(r\)\\mathbf\{q\}\_\{i\}^\{\(r\)\},𝐤j\(r\)\\mathbf\{k\}\_\{j\}^\{\(r\)\}, and𝐯j\(r\)\\mathbf\{v\}\_\{j\}^\{\(r\)\}denote the query, key, and value vectors in therr\-th attention head, and letdhd\_\{h\}be the dimensionality of each head\. X\-LogSMask is added to the query\-key energy before softmax:

αi​j\(r\)=exp⁡\(𝐪i\(r\)​𝐤j\(r\)T\+𝐌i​j\(r\)dh\)∑u∈𝒱exp⁡\(𝐪i\(r\)​𝐤u\(r\)T\+𝐌i​u\(r\)dh\)\\alpha\_\{ij\}^\{\(r\)\}=\\frac\{\\exp\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\(r\)\}\{\\mathbf\{k\}\_\{j\}^\{\(r\)\}\}^\{\\mathrm\{T\}\}\+\\mathbf\{M\}\_\{ij\}^\{\(r\)\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\(r\)\}\{\\mathbf\{k\}\_\{u\}^\{\(r\)\}\}^\{\\mathrm\{T\}\}\+\\mathbf\{M\}\_\{iu\}^\{\(r\)\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\}\(9\)where𝐌\(r\)\\mathbf\{M\}^\{\(r\)\}denotes the structural mask used by therr\-th head\. The corresponding head output is

𝐳i\(r\)=∑j∈𝒱αi​j\(r\)​𝐯j\(r\)\\mathbf\{z\}\_\{i\}^\{\(r\)\}=\\sum\_\{j\\in\\mathcal\{V\}\}\\alpha\_\{ij\}^\{\(r\)\}\\mathbf\{v\}\_\{j\}^\{\(r\)\}\(10\)
This transformation maps bounded structural weights to non\-positive log\-space biases\. Under the attention formulation above, the logarithmic mask acts as a scaled multiplicative structural gate after exponentiation\. Specifically, when therr\-th head uses a structural matrix𝐒\(r\)\\mathbf\{S\}^\{\(r\)\}with𝐌i​j\(r\)=log⁡\(𝐒i​j\(r\)\+ϵ\)\\mathbf\{M\}\_\{ij\}^\{\(r\)\}=\\log\(\\mathbf\{S\}\_\{ij\}^\{\(r\)\}\+\\epsilon\),

exp⁡\(𝐪i\(r\)​𝐤j\(r\)T\+log⁡\(𝐒i​j\(r\)\+ϵ\)dh\)=exp⁡\(𝐪i\(r\)​𝐤j\(r\)Tdh\)​\(𝐒i​j\(r\)\+ϵ\)1/dh\\begin\{gathered\}\\exp\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\(r\)\}\{\\mathbf\{k\}\_\{j\}^\{\(r\)\}\}^\{\\mathrm\{T\}\}\+\\log\\left\(\\mathbf\{S\}\_\{ij\}^\{\(r\)\}\+\\epsilon\\right\)\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\\\\ =\\exp\\left\(\\frac\{\\mathbf\{q\}\_\{i\}^\{\(r\)\}\{\\mathbf\{k\}\_\{j\}^\{\(r\)\}\}^\{\\mathrm\{T\}\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\\left\(\\mathbf\{S\}\_\{ij\}^\{\(r\)\}\+\\epsilon\\right\)^\{1/\\sqrt\{d\_\{h\}\}\}\\end\{gathered\}\(11\)Thus, feature\-dependent attention is preserved, while structurally weak or unsupported node interactions receive smaller pre\-softmax weights\. For absent connections, the small constantϵ\\epsilonyields a strong negative bias rather than an exact zero mask, so X\-LogSMask implements a near\-hard topological constraint in practice\.

#### 4\.2\.3Explainable Multi\-head Mechanism

A naive scheme that broadcasts the same first\-order structural mask across all attention heads would fail to capture multi\-scale topology without deep stacking\. To enable per\-head specialization, we construct head\-specific masks from successive powers of the normalized adjacency\. Formally, fork=1,…,hk=1,\\dots,hdefine

𝐀~k\\displaystyle\\tilde\{\\mathbf\{A\}\}^\{k\}=𝐀~⋅𝐀~​⋯​𝐀~⏟k​times\\displaystyle=\\underbrace\{\\tilde\{\\mathbf\{A\}\}\\cdot\\tilde\{\\mathbf\{A\}\}\\cdots\\tilde\{\\mathbf\{A\}\}\}\_\{k\\ \\text\{times\}\}\(12\)and the corresponding log\-masks

𝐌i​jk=log⁡\(𝐀~i​jk\+ϵ\)\\mathbf\{M\}^\{k\}\_\{ij\}=\\log\\big\(\\tilde\{\\mathbf\{A\}\}^\{k\}\_\{ij\}\+\\epsilon\\big\)\(13\)
We associate thekk\-th head with mask𝐌k\\mathbf\{M\}^\{k\}, thereby encouraging the head to attend preferentially to paths of lengthkk\. Intuitively, low\-order heads emphasize local neighborhoods while higher\-order heads capture longer\-range dependencies\. This structured division eliminates the randomness typical of standard multi\-head attention and enables multi\-scale graph analysis within a single layer—surpassing traditional message\-passing GNNs that require deep architectures to capture long\-range dependencies\. Empirical results show that our model achieves near\-state\-of\-the\-art performance with only 1 layer\.

Note that powers of𝐀~\\tilde\{\\mathbf\{A\}\}remain numerically stable under mild spectral conditions \(for example,‖𝐀~‖2≤1\\\|\\tilde\{\\mathbf\{A\}\}\\\|\_\{2\}\\leq 1when self\-loops are included\), and therefore additional normalization is typically unnecessary for modestkk\. By contrast, we observe empirically that an alternative pipeline—raising the raw adjacency matrix𝐀\+𝐈\\mathbf\{A\}\+\\mathbf\{I\}to thekk\-th power and only then applying a global normalization—produces severe degree imbalance in the resulting multi\-hop matrices\. Intuitively, this effect arises because\(𝐀\+𝐈\)k\(\\mathbf\{A\}\+\\mathbf\{I\}\)^\{k\}amplifies high\-degree nodes multiplicatively, and a posteriori normalization cannot fully compensate for the resultant heterogeneity\. This observation motivates our practical strategy of*normalize first, then raise to thekk\-th power*, which yields more balanced multi\-hop structural descriptors for downstream attention masking\.

Furthermore, many message\-passing GNNs deteriorate as depth increases due to over\-smoothing or node homogenization induced by repeated multi\-hop aggregation\. This issue is absent in our proposed Transformer with X\-LogSMask; instead, a problem with a similar origin is shifted to the number of heads in X\-LogSMask\. High\-order adjacency matrices lack strong practical utility\.

## 5Experiment

We evaluated X\-LogSMask across node\-, edge\- and graph\-level prediction tasks to test whether structural masking improves Transformer\-based graph learning\. The evaluation included 20 benchmark datasets and compared two X\-LogSMask configurations with message\-passing GNNs, Graph Transformer variants and task\-specific baselines\. We first assessed predictive performance across the three task families, then examined the contribution of individual components through ablation, lightweight one\-layer models and attention decomposition\.

### 5\.1Node Level

#### 5\.1\.1Experimental Settings

We evaluated node\-level classification on eight benchmark datasets: Cora, Citeseer, Pubmed, Computers, Photo, CS, Physics and WikiCS\[[25](https://arxiv.org/html/2607.01553#bib.bib40),[26](https://arxiv.org/html/2607.01553#bib.bib41),[21](https://arxiv.org/html/2607.01553#bib.bib42)\]\. These datasets cover citation, co\-purchase, co\-authorship and web\-network domains, and span small citation graphs to larger academic networks\. For each dataset, we strictly followed the train, validation and test splits used in established benchmark studies\[[20](https://arxiv.org/html/2607.01553#bib.bib37)\]\.

We compared X\-LogSMask with three representative message\-passing GNNs, GCN, GraphSAGE and GAT\[[16](https://arxiv.org/html/2607.01553#bib.bib9),[14](https://arxiv.org/html/2607.01553#bib.bib30),[30](https://arxiv.org/html/2607.01553#bib.bib14)\], and seven Graph Transformer variants, SGFormer, Polynormer, GOAT, NodeFormer, NAGphormer, GraphGPS and Exphormer\[[33](https://arxiv.org/html/2607.01553#bib.bib48),[8](https://arxiv.org/html/2607.01553#bib.bib29),[17](https://arxiv.org/html/2607.01553#bib.bib34),[32](https://arxiv.org/html/2607.01553#bib.bib46),[7](https://arxiv.org/html/2607.01553#bib.bib25),[24](https://arxiv.org/html/2607.01553#bib.bib19),[27](https://arxiv.org/html/2607.01553#bib.bib44)\]\. Baseline results were taken from recent comparative benchmarks under matched evaluation protocols\[[20](https://arxiv.org/html/2607.01553#bib.bib37)\]\.

During training, the small citation datasets showed rapid memorization by the Transformer\-based model\. In several runs, training accuracy reached 100% early, while validation and test accuracy remained lower\. To reduce this effect, we used random subgraph sampling as training\-time augmentation on small node\-level datasets\. This sampling procedure was not part of X\-LogSMask itself and was not used for graph\-level tasks\.

#### 5\.1\.2Node\-level Performance

On node classification, the full X\-LogSMask model achieved the best average rank among all compared methods, with an average rank of 3\.3 across eight datasets \(Table[3](https://arxiv.org/html/2607.01553#S5.T3)\)\. It obtained the highest accuracy on Photo, CS, Physics and WikiCS, and the second\-highest accuracy on Computers\. These gains were most evident on the larger node\-classification benchmarks, where structural masking improved the Transformer’s ability to aggregate graph\-dependent information\.

Table 3:Experimental results on node\-level tasks\. The top1stand2n​d2^\{nd\}are highlighted\.CoraCiteseerPubmedComput\-ersPhotoCSPhysicsWikiCSAvg\.Rk\.MetricAcc\.Acc\.Acc\.Acc\.Acc\.Acc\.Acc\.Acc\.\-MP\-GNNsGCN \[Kipf and Welling, 2017\]81\.6071\.6078\.8089\.6592\.7092\.9296\.1877\.4710\.6GraphSAGE \[Hamilton et al\., 2017\]82\.6871\.9379\.4191\.2094\.5993\.9196\.4974\.778\.1GAT \[Veličković et al\., 2018\]83\.0072\.5079\.0090\.7893\.8793\.6196\.1776\.918\.6Graph Transformer variantsGraphGPS \[Rampášek et al\., 2022\]82\.8472\.7379\.9491\.1995\.0693\.9397\.1278\.665\.1NAGphormer \[Chen et al\., 2023\]82\.1271\.4779\.7391\.2295\.4995\.7597\.3477\.165\.9NodeFormer \[Wu et al\., 2022\]82\.2072\.5079\.9086\.9893\.4695\.6496\.4574\.737\.6GOAT \[Kong et al\., 2023\]83\.1871\.9979\.1390\.9692\.9694\.2196\.2477\.008\.1Exphormer \[Shirzad et al\., 2023\]82\.7771\.6379\.4691\.4795\.3594\.9396\.8978\.545\.9SGFormer \[Wu et al\., 2023\]84\.5072\.6080\.3091\.9995\.1094\.7896\.6073\.464\.9Polynormer \[Deng et al\., 2024\]83\.2572\.3179\.2493\.6896\.4695\.5397\.2780\.103\.8Trans\. with X\-LogSMask \(1\-layer\)80\.8071\.0079\.2091\.2195\.6996\.5697\.6580\.365\.9Trans\. with X\-LogSMask \(ours\)82\.7071\.7079\.6092\.0196\.8696\.6297\.6880\.363\.3The one\-layer X\-LogSMask model remained competitive despite its reduced depth\. It ranked within the top three on Photo, CS, Physics and WikiCS, and achieved an average rank of 5\.9\. This result showed that head\-wise multi\-hop structural masks can provide useful receptive\-field expansion without requiring deep Transformer stacks\.

Performance was weaker on the smaller citation datasets\. On Cora, Citeseer and Pubmed, X\-LogSMask did not exceed the strongest competing Graph Transformer baselines\. The corresponding training curves showed early saturation of training accuracy, indicating that overfitting limited performance in these low\-data regimes\. This limitation is consistent with the need for random subgraph augmentation during node\-level training\.

### 5\.2Edge Level

#### 5\.2\.1Experimental Settings

We evaluated edge\-level performance in two settings: link prediction and edge regression\. For link prediction, we used Cora and Citeseer\[[25](https://arxiv.org/html/2607.01553#bib.bib40)\], with mean reciprocal rank \(MRR\) as the evaluation metric\. For edge regression, we used three temporal edge\-regression datasets from the TER benchmark, epic\-games\-plr, air\-traffic\-2015\-rlr and air\-traffic\-2019\-rlr\[[23](https://arxiv.org/html/2607.01553#bib.bib56)\]\. These datasets cover game\-rating prediction and flight\-delay estimation\.

For link prediction, X\-LogSMask was compared with three heuristic baselines, common neighbours \(CN\), Adamic\-Adar \(AA\) and resource allocation \(RA\), and eight neural baselines, GCN, SAGE, SEAL, Neo\-GNN, BUDDY, NCN, NCNC and LPFormer\[[16](https://arxiv.org/html/2607.01553#bib.bib9),[14](https://arxiv.org/html/2607.01553#bib.bib30),[37](https://arxiv.org/html/2607.01553#bib.bib52),[36](https://arxiv.org/html/2607.01553#bib.bib51),[5](https://arxiv.org/html/2607.01553#bib.bib53),[31](https://arxiv.org/html/2607.01553#bib.bib38),[28](https://arxiv.org/html/2607.01553#bib.bib5)\]\. For edge regression, we compared against moving\-average baselines and static message\-passing GNNs, including eGCN, eGSage, eGAT, eGTransf and their rich\-feature variants\.

Dense attention hasO​\(n2\)O\(n^\{2\}\)memory and computational complexity\. We therefore restricted these experiments to small\- and medium\-scale graphs, and did not use the edge\-level benchmarks to assess ultra\-large\-graph scalability\. Link\-prediction datasets were randomly split into training, validation and test sets in an 85:5:10 ratio\[[28](https://arxiv.org/html/2607.01553#bib.bib5)\]\. Edge\-regression datasets were split in a 7:1:2 ratio\.

#### 5\.2\.2Edge\-level Performance

For edge regression, X\-LogSMask also ranked first across all three datasets \(Table[4](https://arxiv.org/html/2607.01553#S5.T4)\)\. Both configurations achieved an average rank of 1\.0 and occupied the best or second\-best position in every MAE and RMSE column\. On epic\-games\-plr, X\-LogSMask reduced MAE to 0\.0149 and RMSE to 0\.0416, compared with 0\.1062 and 0\.1788 for the strongest baseline\. On the two air\-traffic datasets, the one\-layer and full models again produced nearly identical error values, showing that the structural mask provided most of the edge\-regression gain independently of model depth\.

Table 4:Experimental results on edge\-level regression tasks\. The top1stand2n​d2^\{nd\}are highlighted\.epic\-games\-plrair\-traffic\-2019\-rlrair\-traffic\-2015\-rlrAvg\. Rk\.MethodMAERMSEMAERMSEMAERMSE\-Non\-parametric MethodsMA\-all \(K=10\)0\.13220\.19540\.45100\.59310\.18770\.300512\.3MA\-src \(K=10\)0\.14800\.24990\.44210\.58330\.18640\.299311\.7MA\-dst \(K=10\)0\.29510\.43410\.44440\.58670\.18480\.297012\.0Graph Neural NetworkseGCN0\.11780\.18830\.39830\.54670\.16730\.29654\.3eGCN\-rich0\.10620\.17880\.41510\.55930\.17270\.29756\.7eGSage0\.12050\.18940\.40210\.55020\.17270\.29758\.0eGSage\-rich0\.11910\.18830\.40280\.55160\.16670\.29336\.0eGAT0\.11800\.18830\.42520\.56990\.16650\.29446\.0eGAT\-rich0\.11860\.18880\.39950\.54740\.16690\.29425\.0eGTransf0\.11910\.18940\.41740\.55970\.16780\.29648\.0eGTransf\-rich0\.11900\.18870\.40280\.55070\.16890\.29747\.0Trans\. with X\-LogSMask \(1\-layer\)0\.01490\.04160\.10660\.16510\.09960\.15881\.0Trans\. with X\-LogSMask \(ours\)0\.01490\.04160\.10650\.16540\.09890\.15911\.0For link prediction, both the one\-layer and full X\-LogSMask models achieved the best performance on Cora and Citeseer \(Table[5](https://arxiv.org/html/2607.01553#S5.T5)\)\. The models obtained MRR scores of 59\.9 on Cora and 71\.5 on Citeseer, with an average rank of 1\.0\. Compared with the strongest non\-X\-LogSMask baseline, LPFormer, this corresponded to gains of 20\.48 MRR points on Cora and 6\.08 MRR points on Citeseer\. The identical performance of the one\-layer and full configurations indicated that edge\-level structural signals were captured effectively without additional Transformer depth in this setting\.

Table 5:Experimental results on edge\-level link prediction tasks\. The top1stand2n​d2^\{nd\}are highlighted\.CoraCiteseerAvg\. Rk\.MetricMRRMRR\-Heuristic MethodsCN20\.9928\.3412\.5AA31\.8729\.379\.0RA30\.7927\.6110\.5Graph Neural NetworksGCN\[[16](https://arxiv.org/html/2607.01553#bib.bib9)\]32\.5050\.017\.0SAGE\[[14](https://arxiv.org/html/2607.01553#bib.bib30)\]37\.8347\.846\.5SEAL\[[37](https://arxiv.org/html/2607.01553#bib.bib52)\]26\.6939\.3610\.0Neo\-GNN\[[36](https://arxiv.org/html/2607.01553#bib.bib51)\]22\.6553\.979\.5BUDDY\[[5](https://arxiv.org/html/2607.01553#bib.bib53)\]26\.4059\.488\.0NCN\[[31](https://arxiv.org/html/2607.01553#bib.bib38)\]32\.9354\.975\.5NCNC\[[31](https://arxiv.org/html/2607.01553#bib.bib38)\]29\.0164\.036\.5LPFormer\[[28](https://arxiv.org/html/2607.01553#bib.bib5)\]39\.4265\.423\.0Trans\. with X\-LogSMask \(1\-layer\)59\.971\.51\.0Trans\. with X\-LogSMask \(ours\)59\.971\.51\.0

### 5\.3Graph Level

#### 5\.3\.1Experimental Settings

We evaluated graph\-level prediction on seven benchmark datasets: NCI1, D&D, PROTEINS, MUTAG, COLLAB, IMDB\-B and MOLHIV\[[22](https://arxiv.org/html/2607.01553#bib.bib39),[15](https://arxiv.org/html/2607.01553#bib.bib43)\]\. The first six datasets were taken from TU\-Dataset and cover molecular, protein and social\-network classification tasks\. MOLHIV was taken from the Open Graph Benchmark and was evaluated using AUROC\. For TU\-Dataset benchmarks, we used random training, validation and test splits in an 8:1:1 ratio\. For MOLHIV, we used the prescribed benchmark split\.

We compared X\-LogSMask with four message\-passing GNNs, GCN, GAT, GIN and GatedGCN\[[16](https://arxiv.org/html/2607.01553#bib.bib9),[30](https://arxiv.org/html/2607.01553#bib.bib14),[34](https://arxiv.org/html/2607.01553#bib.bib10),[2](https://arxiv.org/html/2607.01553#bib.bib36)\], and seven Graph Transformer variants, GT, SAN, Graphormer, GraphTrans, GMT, SAT and GraphGPS\[[11](https://arxiv.org/html/2607.01553#bib.bib20),[18](https://arxiv.org/html/2607.01553#bib.bib33),[35](https://arxiv.org/html/2607.01553#bib.bib15),[4](https://arxiv.org/html/2607.01553#bib.bib47),[1](https://arxiv.org/html/2607.01553#bib.bib23),[6](https://arxiv.org/html/2607.01553#bib.bib26),[24](https://arxiv.org/html/2607.01553#bib.bib19)\]\. Baseline values were taken from prior benchmark studies to maintain consistency with established evaluation protocols\[[19](https://arxiv.org/html/2607.01553#bib.bib16),[1](https://arxiv.org/html/2607.01553#bib.bib23)\]\.

#### 5\.3\.2Graph\-level Performance

The full X\-LogSMask model achieved the best average rank on graph\-level benchmarks, with an average rank of 1\.7 across seven datasets \(Table[6](https://arxiv.org/html/2607.01553#S5.T6)\)\. It ranked first on D&D, PROTEINS, MUTAG and MOLHIV, and second on IMDB\-B\. The largest improvement was observed on PROTEINS, where X\-LogSMask reached 80\.63% accuracy, compared with 75\.77% for the strongest non\-X\-LogSMask baseline\. On D&D and MUTAG, the model achieved 81\.20% and 88\.89% accuracy, exceeding the best competing results by 2\.48 and 1\.67 percentage points, respectively\.

Table 6:Experimental results on graph\-level tasks\. The top1stand2n​d2^\{nd\}are highlighted\.NCI1D&DPROTE\.MUTAGCOLLABIMDB\-BMOLHIVAvg\. Rk\.MetricAcc\.Acc\.Acc\.Acc\.Acc\.Acc\.AUROC\.\-MP\-GNNsGCN \[Kipf and Welling, 2017\]79\.6872\.0571\.7073\.4071\.9274\.3075\.9910\.0GAT \[Velicković et al\., 2018\]79\.88\-72\.0073\.9075\.8074\.70\-10\.4GIN \[Xu et al\., 2019b\]81\.7070\.7973\.7684\.5073\.3275\.1077\.076\.7GatedGCN \[Bresson and Laurent, 2017\]81\.17\-74\.6585\.0080\.7073\.20\-6\.8Graph Transformer variantsGT \[Dwivedi and Bresson, 2021\]80\.15\-73\.9483\.9079\.6373\.10\-9\.2SAN \[Kreuzer et al\., 2021\]80\.50\-74\.1178\.8079\.4272\.1077\.858\.8Graphormer \[Ying et al\., 2021\]81\.44\-75\.2980\.5281\.8073\.4074\.556\.2GraphTrans \[Cai and Lam, 2019\]82\.60\-75\.1887\.2279\.8174\.5076\.335\.0GMT \[Baek et al\., 2021\]\-78\.7275\.0983\.4480\.7473\.48\-6\.0SAT \[Chen et al\., 2022\]80\.69\-73\.3280\.5080\.0575\.90\-7\.8GraphGPS \[Rampášek et al\., 2022\]84\.21\-75\.7785\.0081\.4077\.4078\.802\.0Trans\. with X\-LogSMask \(1\-layer\)81\.2781\.2075\.6888\.8979\.0076\.0077\.684\.0Trans\. with X\-LogSMask \(ours\)82\.2481\.2080\.6388\.8980\.8077\.0078\.911\.7The gains were not uniform across all datasets\. On NCI1, X\-LogSMask achieved 82\.24% accuracy, ranking behind GraphGPS and GraphTrans\. On COLLAB, it reached 80\.80% accuracy, below Graphormer and GraphGPS\. These results show that X\-LogSMask provided strong overall graph\-level performance, but its advantage was dataset\-dependent\.

The 1\-layer lightweight configuration exhibits remarkable efficiency, matching the full\-version’s top performance on D&D and MUTAG while maintaining competitiveness on other datasets\. It outperforms most complex Graph Transformer variants and traditional MP\-GNNs, demonstrating that X\-LogSMask enables reduced model complexity without performance degradation\.

### 5\.4Further Discussion

#### 5\.4\.1Ablation

To quantify the contribution of each core component, we perform an ablation study on node\-level classification benchmarks\. Starting from the full model, we removed symmetric normalization, the logarithmic structural mask and the explainable multi\-head mechanism one at a time\. The results are summarized in Table[7](https://arxiv.org/html/2607.01553#S5.T7)\.

Table 7:Ablation Study on Key Components of the proposed Model\.ComponentsCoraCiteseerPubmedComputersPhotoCSPhysicsWikiCSSym\. Norm\.LogSMaskExpl\. MH✓✓✓82\.7071\.7079\.6091\.5796\.8696\.6297\.6880\.36×\\times✓✓77\.00↓5\.707\.70↓64\.0074\.50↓5\.103\.27↓88\.304\.84↓95\.0095\.69↓0\.9397\.16↓0\.524\.01↓76\.35✓×\\times✓58\.80↓23\.9057\.80↓13\.9072\.40↓7\.2076\.74↓14\.8387\.06↓9\.8095\.20↓1\.4296\.35↓1\.3371\.73↓8\.63✓✓×\\times77\.40↓5\.3068\.00↓3\.7076\.20↓3\.4092\.01↑0\.4495\.68↓1\.1895\.80↓0\.8297\.68=0\.0079\.42↓0\.94
- Note:↓= performance degradation,↑= performance improvement\.

Removing symmetric normalization caused the largest performance collapse\. Accuracy dropped by up to 95\.00% on Photo and 88\.30% on Computers, showing that degree normalization is essential for stable propagation and for preventing multi\-hop weights from becoming overly biased toward high\-degree nodes\. Removing the logarithmic structural mask also caused severe degradation, including a 23\.90% drop on Cora, indicating that the structural mask is the main source of graph\-specific inductive bias\. By contrast, removing the explainable multi\-head mechanism produced only modest changes, and occasionally a small gain, such as the 0\.44% increase on Computers\. This suggests that the multi\-head design mainly supports structural specialization and interpretability, rather than serving as the primary source of predictive capacity\.

#### 5\.4\.2Interpretability of X\-LogSMask

Self\-attention energy matrices provide a direct view of inter\-node message passing\. To inspect how X\-LogSMask decomposes attention, we visualized the raw energy term, the structural term and their sum across two layers and four heads of a pre\-trained model on PROTEINS, as shown in Fig\.[3](https://arxiv.org/html/2607.01553#S5.F3)\.

The raw energy term mainly captured node\-attribute similarity, whereas X\-LogSMask injected edge\-level structure\. Their additive combination therefore coupled content similarity with topological bias in a single attention score\. The heads also showed distinct hop preferences, indicating that different heads specialized in different structural radii\. Comparing the two layers further showed a clear progression: the first layer mixed node and edge information, whereas the final layer before classification became more GNN\-like, with edge structure exerting a stronger influence on node representations\.

![Refer to caption](https://arxiv.org/html/2607.01553v1/attention_visual.png)Figure 3:Layer\-wise attention decomposition in a pre\-trained 2\-layer, 4\-head X\-LogSMask Transformer on PROTEINS\. For each layer and head, we compare the raw attention energy, the X\-LogSMask structural term, and their additive combination\. The comparison shows that the energy term mainly captures node\-attribute similarity, whereas X\-LogSMask injects edge\-level structure; their sum therefore couples node\-content information and structural bias within a single attention score\. Head\-specific patterns further indicate that different heads focus on different hop ranges, allowing one layer to approximate multi\-hop connectivity without deep stacking\. Comparing the two layers shows a clear progression: the first layer still mixes node and edge information, whereas the final layer before classification becomes more GNN\-like, with edge structure exerting a stronger influence on node representations\.
#### 5\.4\.3Lightweight 1\-layer Solution

The head\-wise structural decomposition of X\-LogSMask allows a single Transformer layer to capture multi\-hop graph context\. Across the benchmark suite, the 1\-layer model remained competitive with deeper baselines and matched the full model on several edge\-level tasks, as shown in Tables[3](https://arxiv.org/html/2607.01553#S5.T3),[4](https://arxiv.org/html/2607.01553#S5.T4),[5](https://arxiv.org/html/2607.01553#S5.T5),[6](https://arxiv.org/html/2607.01553#S5.T6), and Fig\.[4](https://arxiv.org/html/2607.01553#S5.F4)\.

![Refer to caption](https://arxiv.org/html/2607.01553v1/x3.png)Figure 4:Average ranks of X\-LogSMask variants across task families\. The one\-layer configuration is compared with the full X\-LogSMask and the best non\-X baseline, with lower ranks indicating better relative performance and the shaded region denoting the top\-tier rank zone\.This result shows that structural masking can partially substitute for depth in graph learning\. It does not remove the quadratic cost of dense attention within a layer, but it reduces the need for repeated stacking and therefore lowers the overall depth\-related cost\. We note that the 1\-layer design trades breadth for depth: careful selection of head counts and head\-specific structural orders is important to maximize the effective receptive field without introducing redundancy\. The trade\-off is also clear: one\-layer X\-LogSMask is often strong, but the full model still performs better on several graph\-level datasets\.

## 6Conclusion

We present X\-LogSMask, a concise and interpretable modification to the self\-attention matrix\. It embeds graph topology as an additive structural bias while preserving the discriminative power of multiplicative attention, thereby enabling Transformers to natively handle graph\-structured data\. Through comprehensive experiments on node\-, edge\-, and graph\-level benchmarks, X\-LogSMask consistently improves performance over strong baselines and establishes new state\-of\-the\-art results\. Moreover, we present a lightweight 1\-layer solution for large\-scale graphs and validate the model’s interpretability\. The core insight of introducing structured, multi\-scale masks into attention can be adapted to other structured modalities, suggesting a general pathway for other domains\.

## Data availability

All datasets used in this study are publicly available benchmark datasets\. Cora, Citeseer, Pubmed, Computers, Photo, CS, Physics, and WikiCS can be accessed through the PyTorch Geometric dataset loaders for Planetoid, Amazon, Coauthor, and WikiCS datasets at[https://pytorch\-geometric\.readthedocs\.io/en/latest/modules/datasets\.html](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html)\. The edge\-regression datasets are available from the TER benchmark repositories on Hugging Face: epic\-games\-plr at[https://huggingface\.co/datasets/cash\-app\-inc/epic\-games\-plr](https://huggingface.co/datasets/cash-app-inc/epic-games-plr), air\-traffic\-2019\-rlr at[https://huggingface\.co/datasets/cash\-app\-inc/air\-traffic\-2019\-rlr](https://huggingface.co/datasets/cash-app-inc/air-traffic-2019-rlr), and air\-traffic\-2015\-rlr at[https://huggingface\.co/datasets/cash\-app\-inc/air\-traffic\-2015\-rlr](https://huggingface.co/datasets/cash-app-inc/air-traffic-2015-rlr)\. NCI1, D&D, PROTEINS, MUTAG, COLLAB, and IMDB\-B are available from TUDataset at[https://chrsmrrs\.github\.io/datasets/docs/datasets/](https://chrsmrrs.github.io/datasets/docs/datasets/)\. MOLHIV is available from the Open Graph Benchmark graph property prediction collection at[https://ogb\.stanford\.edu/docs/graphprop/\#ogbg\-mol](https://ogb.stanford.edu/docs/graphprop/#ogbg-mol)\. CIFAR\-10 is available from the original CIFAR\-10 data source at[https://www\.cs\.toronto\.edu/~kriz/cifar\.html](https://www.cs.toronto.edu/~kriz/cifar.html)\.

## Code availability

## Datasets

We utilize a total of 20 datasets to evaluate model performance across node\-, edge\-, and graph\-level tasks\. These graph\-learning experiments were executed on A100 GPUs, whereas the auxiliary CIFAR\-10 Vision Transformer experiments were conducted on a single A5000 GPU\.

- •Node\-level: CS, Physics \(OGB \[Hu et al\., 2020\]\); Computers, Photo \(Benchmarking GNN \[Dwivedi et al\., 2023\]\); Cora, Citeseer, Pubmed \[Sen et al\., 2008\]; WikiCS \[Zitnik & Leskovec, 2017\]\.
- •Edge\-level: Cora, Citeseer \[Sen et al\., 2008\]; epic\-games\-plr, air\-traffic\-2019\-rlr, air\-traffic\-2015\-rlr \[Ozmen et al\., 2024\]\.
- •Graph\-level: NCI1, D&D, PROTEINS, MUTAG, COLLAB, IMDB\-B \(TU\-Dataset \[Morris et al\., 2020\]\); MOLHIV \(OGB \[Hu et al\., 2020\]\)\.

These datasets span citation networks, social networks, molecular graphs, biological networks, traffic networks, game review platforms, and NFT transaction systems, with varied scales to ensure comprehensive evaluation of generalization across graph types and tasks\. Detailed statistics are in Tables[8](https://arxiv.org/html/2607.01553#Ax1.T8),[9](https://arxiv.org/html/2607.01553#Ax1.T9), and[10](https://arxiv.org/html/2607.01553#Ax1.T10)\.

Table 8:Datasets for Node\-Level TasksDataset\# Nodes\# EdgesPredict levelPredict taskMetricCora2,7085,278node7\-class classif\.AccuracyCiteseer3,3274,732node6\-class classif\.AccuracyPubmed19,71744,324node3\-class classif\.AccuracyComputers13,752245,861node10\-class classif\.AccuracyPhoto7,650119,081node8\-class classif\.AccuracyCS18,33381,894node15\-class classif\.AccuracyPhysics34,493247,962node5\-class classif\.AccuracyWikiCS11,701216,123node10\-class classif\.AccuracyTable 9:Datasets for Edge\-Level TasksDataset\# Nodes\# EdgesPredict levelPredict taskMetricCora2,7085,278edgelink predictionMRRCiteseer3,3274,732edgelink predictionMRRepic\-games\-plr1,15617,584edgeedge regressionMAE, RMSEair\-traffic\-2019\-rlr274484,551edgeedge regressionMAE, RMSEair\-traffic\-2015\-rlr2575,138,263edgeedge regressionMAE, RMSETable 10:Datasets for Graph\-Level TasksDataset\# Graphs\# Nodes \(avg\)\# Edges \(avg\)Predict levelPredict taskMetricNCI14,11029\.832\.3graph2\-class classif\.AccuracyD&D1,178284\.3715\.6graph2\-class classif\.AccuracyPROTEINS1,11339\.172\.8graph2\-class classif\.AccuracyMUTAG18817\.919\.7graph2\-class classif\.AccuracyCOLLAB5,00074\.52457\.7graph3\-class classif\.AccuracyIMDB\-B1,00019\.896\.5graph2\-class classif\.AccuracyMOLHIV41,12725\.527\.5graph2\-class classif\.AUROC\.Table 11:Hyper\-parameter configurations for the CIFAR\-10 image\-classification experiments\. All runs share identical data splits and random seeds\.RunModelPatch sizeEmbed\. dimPos\. EncodingLayersHeadsDropoutBatch sizeLROptimiserT1ViT2×22\\\!\\times\\\!2256✓880\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamWViT \+ X\-LogSMask2×22\\\!\\times\\\!2256×\\times880\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamWT2ViT2×22\\\!\\times\\\!2256✓680\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamWViT \+ X\-LogSMask2×22\\\!\\times\\\!2256×\\times680\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamWT3ViT2×22\\\!\\times\\\!2256✓640\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamWViT \+ X\-LogSMask2×22\\\!\\times\\\!2256×\\times640\.11281×10−31\\\!\\times\\\!10^\{\-3\}AdamW
## Extending X\-LogSMask to Vision Transformers

To test whether the structural mask transfers beyond graph data, we integrated X\-LogSMask into a standard Vision Transformer and evaluated it on CIFAR\-10\. To isolate the effect of the mask, we kept the training setup fixed, without learning\-rate warmup, extra augmentation or extensive hyperparameter tuning\. The two models were trained under identical settings on a single A5000 GPU, and the hyperparameters are listed in Table[11](https://arxiv.org/html/2607.01553#Ax1.T11)\.

X\-LogSMask conferred a modest yet consistent improvement in classification accuracy relative to the baseline ViT\. Across the three controlled runs, X\-LogSMask improved ViT accuracy from 82\.18% to 86\.07% in T1, from 80\.96% to 85\.08% in T2, and from 81\.44% to 84\.15% in T3, corresponding to gains of 3\.89, 4\.12 and 2\.71 percentage points\. These results show that the structural mask can also improve a grid\-based Transformer, although the main focus of this paper remains graph\-structured data\.

Table 12:Image\-classification accuracy on CIFAR\-10\.RunModelAccuracy/%T1ViT82\.18ViT \+ X\-LogSMask86\.07↑3\.89T2ViT80\.96ViT \+ X\-LogSMask85\.08↑4\.12T3ViT81\.44ViT \+ X\-LogSMask84\.15↑2\.71
- Note:↑indicates improvement brought by X\-LogSMask\.

## References

- \[1\]J\. Baek, M\. Kang, and S\. J\. Hwang\(2021\)Accurate learning of graph representations with graph multiset pooling\.InInternational Conference on Learning Representations,Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[2\]X\. Bresson and T\. Laurent\(2017\)Residual gated graph convnets\.arXiv preprint arXiv:1711\.07553\.External Links:1711\.07553Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[3\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in Neural Information Processing Systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p2.1)\.
- \[4\]D\. Cai and W\. Lam\(2020\)Graph transformer for graph\-to\-sequence learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7464–7471\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6243),1911\.07470Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[5\]B\. P\. Chamberlain, S\. Shirobokov, E\. Rossi, F\. Frasca, T\. Markovich, N\. Y\. Hammerla, M\. M\. Bronstein, and M\. Hansmire\(2023\)Graph neural networks for link prediction with subgraph sketching\.InInternational Conference on Learning Representations,Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.12.1.1.1)\.
- \[6\]D\. Chen, L\. O’Bray, and K\. Borgwardt\(2022\)Structure\-aware transformer for graph representation learning\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 3469–3489\.Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[7\]J\. Chen, K\. Gao, G\. Li, and K\. He\(2023\)NAGphormer: a tokenized graph transformer for node classification in large graphs\.InInternational Conference on Learning Representations,Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[8\]C\. Deng, Z\. Yue, and Z\. Zhang\(2024\)Polynormer: polynomial\-expressive graph transformer in linear time\.InInternational Conference on Learning Representations,Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[9\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423),[Link](https://aclanthology.org/N19-1423/)Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p2.1)\.
- \[10\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly,et al\.\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YicbFdNTTy)Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p2.1)\.
- \[11\]V\. P\. Dwivedi and X\. Bresson\(2021\)A generalization of transformer networks to graphs\.InAAAI Workshop on Deep Learning on Graphs: Methods and Applications,Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[12\]A\. Garg\(2024\)Graph transformers without positional encodings\.arXiv preprint arXiv:2401\.17791\.External Links:2401\.17791Cited by:[Table 1](https://arxiv.org/html/2607.01553#S1.T1.4.4.2.1.1.2.1.1.1),[§1](https://arxiv.org/html/2607.01553#S1.p3.1),[§2](https://arxiv.org/html/2607.01553#S2.p2.1)\.
- \[13\]Y\. Gong, Y\. Chung, and J\. Glass\(2021\)AST: audio spectrogram transformer\.InProceedings of Interspeech 2021,pp\. 571–575\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2021-698)Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p2.1)\.
- \[14\]W\. L\. Hamilton, Z\. Ying, and J\. Leskovec\(2017\)Inductive representation learning on large graphs\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1),[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.9.1.1.1)\.
- \[15\]W\. Hu, M\. Fey, M\. Zitnik, Y\. Dong, H\. Ren, B\. Liu, M\. Catasta, and J\. Leskovec\(2020\)Open graph benchmark: datasets for machine learning on graphs\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p1.1)\.
- \[16\]T\. N\. Kipf and M\. Welling\(2017\)Semi\-supervised classification with graph convolutional networks\.InInternational Conference on Learning Representations,External Links:1609\.02907,[Link](https://openreview.net/forum?id=SJU4ayYgl)Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p1.1),[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1),[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.8.1.1.1)\.
- \[17\]K\. Kong, J\. Chen, J\. Kirchenbauer, R\. Ni, C\. B\. Bruss, and T\. Goldstein\(2023\)GOAT: a global transformer on large\-scale graphs\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 17375–17390\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[18\]D\. Kreuzer, D\. Beaini, W\. L\. Hamilton, V\. Létourneau, and P\. Tossou\(2021\)Rethinking graph transformers with spectral attention\.InAdvances in Neural Information Processing Systems,Vol\.34\.Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[19\]C\. Liu, Z\. Yao, Y\. Zhan, X\. Ma, S\. Pan, and W\. Hu\(2024\)Gradformer: graph transformer with exponential decay\.InProceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence,pp\. 2171–2179\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2024/240)Cited by:[Table 1](https://arxiv.org/html/2607.01553#S1.T1.3.3.2.1.1.2.1.1.1),[§1](https://arxiv.org/html/2607.01553#S1.p3.1),[§2](https://arxiv.org/html/2607.01553#S2.p2.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[20\]Y\. Luo, L\. Shi, and X\. Wu\(2024\)Classic gnns are strong baselines: reassessing gnns for node classification\.InAdvances in Neural Information Processing Systems,Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p1.1),[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[21\]P\. Mernyei and C\. Cangea\(2020\)Wiki\-cs: a wikipedia\-based benchmark for graph neural networks\.arXiv preprint arXiv:2007\.02901\.External Links:2007\.02901Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p1.1)\.
- \[22\]C\. Morris, N\. M\. Kriege, F\. Bause, K\. Kersting, P\. Mutzel, and M\. Neumann\(2020\)TUDataset: a collection of benchmark datasets for learning with graphs\.External Links:2007\.08663,[Link](https://arxiv.org/html/2607.01553v1/www.graphlearning.io)Cited by:[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p1.1)\.
- \[23\]M\. Ozmen, F\. Regol, and T\. Markovich\(2024\)Benchmarking edge regression on temporal networks\.Journal of Data\-centric Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=4k4cocpuSw)Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p1.1)\.
- \[24\]L\. Rampášek, M\. Galkin, V\. P\. Dwivedi, A\. T\. Luu, G\. Wolf, and D\. Beaini\(2022\)Recipe for a general, powerful, scalable graph transformer\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p3.1),[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[25\]P\. Sen, G\. Namata, M\. Bilgic, L\. Getoor, B\. Galligher, and T\. Eliassi\-Rad\(2008\)Collective classification in network data\.AI Magazine29\(3\),pp\. 93–93\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p1.1),[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p1.1)\.
- \[26\]O\. Shchur, M\. Mumme, A\. Bojchevski, and S\. Günnemann\(2018\)Pitfalls of graph neural network evaluation\.arXiv preprint arXiv:1811\.05868\.External Links:1811\.05868Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p1.1)\.
- \[27\]H\. Shirzad, A\. Velingker, B\. Venkatachalam, D\. J\. Sutherland, and A\. K\. Sinop\(2023\)Exphormer: sparse transformers for graphs\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 31613–31632\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[28\]H\. Shomer, Y\. Ma, H\. Mao, J\. Li, B\. Wu, and J\. Tang\(2024\-08\)LPFormer: an adaptive graph transformer for link prediction\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,KDD ’24,pp\. 2686–2698\.External Links:[Link](http://dx.doi.org/10.1145/3637528.3672025),[Document](https://dx.doi.org/10.1145/3637528.3672025)Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p3.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.15.1.1.1)\.
- \[29\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p1.1)\.
- \[30\]P\. Veličković, G\. Cucurull, A\. Casanova, A\. Romero, P\. Liò, and Y\. Bengio\(2018\)Graph attention networks\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p3.1),[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[31\]X\. Wang, H\. Yang, and M\. Zhang\(2024\)Neural common neighbor with completion for link prediction\.InInternational Conference on Learning Representations,Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.13.1.1.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.14.1.1.1)\.
- \[32\]Q\. Wu, W\. Zhao, Z\. Li, D\. Wipf, and J\. Yan\(2022\)NodeFormer: a scalable graph structure learning transformer for node classification\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[33\]Q\. Wu, W\. Zhao, C\. Yang, H\. Zhang, F\. Nie, H\. Jiang, Y\. Bian, and J\. Yan\(2023\)SGFormer: simplifying and empowering transformers for large\-graph representations\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.01553#S5.SS1.SSS1.p2.1)\.
- \[34\]K\. Xu, W\. Hu, J\. Leskovec, and S\. Jegelka\(2019\)How powerful are graph neural networks?\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p1.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[35\]C\. Ying, T\. Cai, S\. Luo, S\. Zheng, G\. Ke, D\. He, Y\. Shen, and T\. Liu\(2021\)Do transformers really perform bad for graph representation?\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 28877–28888\.Cited by:[Table 1](https://arxiv.org/html/2607.01553#S1.T1.2.2.2.1.1.2.1.1.1),[§1](https://arxiv.org/html/2607.01553#S1.p3.1),[§2](https://arxiv.org/html/2607.01553#S2.p2.1),[§5\.3\.1](https://arxiv.org/html/2607.01553#S5.SS3.SSS1.p2.1)\.
- \[36\]S\. Yun, S\. Kim, J\. Lee, J\. Kang, and H\. J\. Kim\(2021\)Neo\-gnns: neighborhood overlap\-aware graph neural networks for link prediction\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 13683–13694\.Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.11.1.1.1)\.
- \[37\]M\. Zhang and Y\. Chen\(2018\)Link prediction based on graph neural networks\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§5\.2\.1](https://arxiv.org/html/2607.01553#S5.SS2.SSS1.p2.1),[Table 5](https://arxiv.org/html/2607.01553#S5.T5.5.10.1.1.1)\.
- \[38\]P\. Zhang, Y\. Yan, X\. Zhang, C\. Li, S\. Wang, F\. Huang, and S\. Kim\(2024\)TransGNN: harnessing the collaborative power of transformers and graph neural networks for recommender systems\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1285–1295\.Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p3.1)\.
- \[39\]H\. Zhou, S\. Zhang, J\. Peng, S\. Zhang, J\. Li, H\. Xiong, and W\. Wang\(2021\)Informer: beyond efficient transformer for long sequence time\-series forecasting\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 11106–11115\.Cited by:[§1](https://arxiv.org/html/2607.01553#S1.p2.1)\.