Why don't Graph Transformers attend to the same token?

# Why don’t Graph Transformers attend to the same token?
Source: [https://arxiv.org/html/2610.19324](https://arxiv.org/html/2610.19324)
\\workshoptitle

Geometric Distributional Deep Learning

###### Abstract

In large language models \(LLMs\), attention concentrates heavily on the first token\. This ‘attention sink’ is believed to protect deep models against over\-mixing\. Even though graph transformers face a related pressure, we find that a SoTA graph transformer, GRIT, rarely forms sinks on molecules, and that it sinks less when trained on larger graphs, the opposite of the trend in LLMs\. In principle, a graph can prevent sinks: we prove and show empirically that its symmetry and locality limit the attention any node can receive\. On molecules, however, with distinguishable atoms and global attention, neither limit prevents sinks; what stops them lies in the model\. We find that the attention scoring function is the main cause\. Replacing GRIT’s gated logit with a dot product raises its sink rate from0\.090\.09to0\.680\.68on identical graphs\. In Graph Transformers using dot\-product attention, removing the sink gives about the same error as removing a random node, while GRIT’s most\-attended nodes matter to its predictions\. Attention sinks are therefore a property of dot\-product attention rather than of Transformers\. In our experiments, they form when dot\-product scoring has a distinguishable node to concentrate on\.111The code can be found at[https://github\.com/alexandre\-bismuth/graph\-transformer\-sinks](https://github.com/alexandre-bismuth/graph-transformer-sinks)\.

## 1Introduction

A striking empirical property of Large Language Models \(LLMs\) is the*attention sink*\([Xiao et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib27)\)\. In Transformers, an attention sink is a token that receives a large share of all attention: in Llama 3\.1 405B, nearly80%80\\%of attention heads sink on the semantically meaningless <BOS\> token\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2), sink metric of[Gu et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib12)at thresholdε=0\.8\\varepsilon=0\.8;\)\. Far from wasting compute, sinks are thought to slow the mixing of information between tokens and so protect deep models against ‘over\-mixing’\([Dong et al\., 2021](https://arxiv.org/html/2610.19324#bib.bib9);[Barbero et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib1);[Di Giovanni et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib8);[Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2)\), and larger models and longer training contexts both bring stronger sinks\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2)\)\.

Graph Transformers face a related pressure: depth causes over\-smoothing\([Zhang et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib29)\), and global attention makes it worse, since softmax over ever more nodes must disperse toward uniform mixing\([Veličković et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib25)\)\. A designated sink node might be a natural defence, yet nothing in an ordinary graph plays the role of <BOS\>: a graph has no first token, and permutation equivariance forbids preferring one of two nodes the model cannot tell apart\. Sinks are documented beyond causally masked LLMs, in bidirectional diffusion language models\([Rulli et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib24)\), as massive activations in attentional GNNs\([Bini et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib5)\), and when LLMs read serialised graphs\([Guan et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib13)\); concurrent work argues that graphs lack natural sinks\([Liu et al\., 2026](https://arxiv.org/html/2610.19324#bib.bib15)\)\. What is missing is an account of*where*a sink can form on a graph at all, and of*which*architectural ingredient decides whether one does\.

We ask whether Graph Transformers form attention sinks and, if not, what prevents them\. We study GRIT\([Ma et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib16)\), a state\-of\-the\-art Graph Transformer without message passing, so attention alone mixes information between nodes\. We find that GRIT rarely forms sinks on molecules, and sinks even less when trained on larger graphs \(Section[2](https://arxiv.org/html/2610.19324#S2)\), but that swapping in dot\-product attention makes it sink far more often \(Section[4](https://arxiv.org/html/2610.19324#S4)\)\. We investigate two suspects\. The graph itself is not the barrier: we prove three bounds on received attention, from symmetry, locality, and the spread of attention scores, and find both graph\-side bounds loose on real molecules \(Section[3](https://arxiv.org/html/2610.19324#S3)\)\. The attention scoring function is the main cause, as the logit swap of Figure[1](https://arxiv.org/html/2610.19324#S1.F1)and accuracy\-matched controls show \(Section[4](https://arxiv.org/html/2610.19324#S4)\)\.

![Refer to caption](https://arxiv.org/html/2610.19324v1/fig1.png)Figure 1:Changing only the scoring function creates a sink\. Each atom of a2323\-atom ZINC test molecule is shaded by the attention it receives, averaged over queries and heads at each model’s peak\-sinking layer: blue below the uniform share1/231/23, red above\. Swapping only GRIT’s gated logit for a dot product \(middle\) moves attention off the other atoms and onto one, as in GraphGPS \(right\)\.
## 2GRIT rarely forms attention sinks on molecules

Graph structure enters GRIT\([Ma et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib16)\)through Relative Random Walk Probabilities \(RRWP\), which initialise a learned pair representation\. GRIT scores a pair by gating the sum of query and key with that representation, passing it through a signed square root, and clipping it to\[0,5\]\[0,5\]before a scalar readout; dot\-product attention instead takes the inner product of query and key, which can grow with the product of the two node norms:

zu​jGRIT=a⊤​min​\{5,ReLU⁡\(ssq⁡\(\(qu\+kj\)⊙wu​j\)\+bu​j\)\},zu​jdot=qu⊤​kj/h,z^\{\\mathrm\{GRIT\}\}\_\{uj\}=a^\{\\top\}\\min\\bigl\\\{5,\\operatorname\{ReLU\}\\bigl\(\\operatorname\{ssq\}\(\(q\_\{u\}\+k\_\{j\}\)\\odot w\_\{uj\}\)\+b\_\{uj\}\\bigr\)\\bigr\\\},\\qquad z^\{\\mathrm\{dot\}\}\_\{uj\}=q\_\{u\}^\{\\top\}k\_\{j\}/\\sqrt\{h\},\(1\)withssq⁡\(x\)=sgn⁡\(x\)​\|x\|\\operatorname\{ssq\}\(x\)=\\operatorname\{sgn\}\(x\)\\sqrt\{\|x\|\},wu​j,bu​jw\_\{uj\},b\_\{uj\}linear in the pair representation, andhhthe head dimension\. Clipping the scalar logit instead, as the official code does, gives the same sink rate \(Appendix[C](https://arxiv.org/html/2610.19324#A3)\)\. We compare against GraphGPS\([Rampášek et al\., 2022](https://arxiv.org/html/2610.19324#bib.bib23)\)with RWSE\([Dwivedi et al\., 2022a](https://arxiv.org/html/2610.19324#bib.bib10)\), a hybrid of message passing and dot\-product attention\. We train on ZINC\-12k, a regression task on graphs of roughly 23 nodes, and on Peptides\-func\([Dwivedi et al\., 2022b](https://arxiv.org/html/2610.19324#bib.bib11)\), a classification task on larger graphs, both whole and split by node count into four quartiles \(mean sizes 56 to 245\), each trained on 2,023 graphs at batch size 16 to remove the dataset\-size confound\. Our results are means with a95%95\\%CI on the full test set over three seeds \(five for GraphGPS and the quartiles\)\.

Our central measurement is thesink scoreof a node, the attention it receives averaged over all queries\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2);[Gu et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib12)\)\. Following[Gu et al\. \(2025\)](https://arxiv.org/html/2610.19324#bib.bib12), a node is a sink in a head when its sink score exceedsε=0\.3\\varepsilon=0\.3, about seven times the uniform share on a 23\-node graph\. Thesink rateis the largest fraction of heads that share one sink node, averaged over test graphs, at the peak\-sinking layer\.

On ZINC, GRIT’s sink rate is0\.090±0\.0280\.090\\pm 0\.028: fewer than one head in ten sinks on the same node, against0\.415±0\.1940\.415\\pm 0\.194for GraphGPS and 0\.95–1\.00 for small language models at the same context length\. GRIT stays below GraphGPS at every threshold from0\.10\.1to0\.80\.8\(Appendix[C](https://arxiv.org/html/2610.19324#A3)\)\. The trend with size also runs backwards\. Language models trained on longer contexts form stronger sinks\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2)\), whereas GRIT’s sink rate falls from0\.2380\.238on the smallest Peptides\-func quartile to0\.0450\.045on the largest\. As in language models, where sink tokens develop unusually large norms\([Queipo\-de\-Llano et al\., 2026](https://arxiv.org/html/2610.19324#bib.bib21)\), GRIT’s largest squared node norm grows across layers from2\.42\.4to4\.84\.8times the squared mean norm\. Nonetheless, sinks stay rare, so something in GRIT keeps large norms from attracting attention \(Section[4](https://arxiv.org/html/2610.19324#S4)\)\.

## 3What can prevent a sink?

Three things can prevent a node from accumulating attention from many others at once: symmetry, since an equivariant model must treat nodes the labelled graph does not distinguish alike; locality, since nodes that attend mostly to their neighbourhood leave little attention for distant ones; and scoring range, since softmax concentrates only when logits spread far apart\.

Notation\.We writeA∈ℝn×nA\\in\\mathbb\{R\}^\{n\\times n\}for the attention matrix of a single layer and head, with logitszu​vz\_\{uv\}, and define the sink score assj​\(A\)=1n​∑uAu​js\_\{j\}\(A\)=\\tfrac\{1\}\{n\}\\sum\_\{u\}A\_\{uj\}\. We assume permutation equivariance, i\.e\.A⁡\(P​Z\)=P​A​\(Z\)​P⊤A\(PZ\)=PA\(Z\)P^\{\\top\}for every permutation matrixPPand inputZZcontaining the graph, node features and positional encodings\. Let𝒪⁡\(j\)\\mathcal\{O\}\(j\)be the orbit ofjjunder the automorphisms ofZZ\(the nodes equivariant models must treat likejj\),BG​\(j,r\)B\_\{G\}\(j,r\)the ball of radiusrraroundjjin shortest\-path distancedGd\_\{G\},λr\(A\)=maxu∑v:dG​\(u,v\)\>rAu​v\\lambda\_\{r\}\(A\)=\\max\_\{u\}\\sum\_\{v:\\,d\_\{G\}\(u,v\)\>r\}A\_\{uv\}the largest attention any query sends beyond distancerr, andΔ=maxu⁡\(maxv⁡zu​v−minv⁡zu​v\)\\Delta=\\max\_\{u\}\(\\max\_\{v\}z\_\{uv\}\-\\min\_\{v\}z\_\{uv\}\)the largest logit spread of any query\.

###### Proposition 3\.1\(Three bounds on received attention\)

Under the setup above, for every nodejjand every radiusrr,

sj​\(A\)≤min⁡\{1\|𝒪⁡\(j\)\|,λr​\(A\)\+\(1−λr​\(A\)\)​\|BG​\(j,r\)\|n,eΔeΔ\+n−1\}\.s\_\{j\}\(A\)\\;\\leq\\;\\min\\left\\\{\\frac\{1\}\{\|\\mathcal\{O\}\(j\)\|\},\\;\\;\\lambda\_\{r\}\(A\)\+\\bigl\(1\-\\lambda\_\{r\}\(A\)\\bigr\)\\frac\{\|B\_\{G\}\(j,r\)\|\}\{n\},\\;\\;\\frac\{e^\{\\Delta\}\}\{e^\{\\Delta\}\+n\-1\}\\right\\\}\.In particular, on a vertex\-transitive labelled inputsj​\(A\)=1/ns\_\{j\}\(A\)=1/nfor every node, independently of the learned parameters, and reaching received attentionε\\varepsilonrequiresΔ≥log⁡\(\(n−1\)​ε/\(1−ε\)\)\\Delta\\geq\\log\\bigl\(\(n\-1\)\\varepsilon/\(1\-\\varepsilon\)\\bigr\)\.

The orbit term is the standard fact that equivariant maps are constant on orbits\([Morris et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib19)\)\. What we add is that every bound can be computed head by head, so we can test whether any bound is tight enough to prevent the formation of a sink \(Appendix[B](https://arxiv.org/html/2610.19324#A2)\)\. The symmetry bound is tight on symmetric graphs: every checkpoint with an equivariant positional encoding, even the dot\-product models that sink heavily on molecules, spreads attention evenly across cycles of identical atoms and bonds \(C8C\_\{8\}toC24C\_\{24\}; Appendix[B](https://arxiv.org/html/2610.19324#A2)\)\. Symmetry alone prevents the sink, whatever the architecture\. Conversely, Laplacian encodings, whose eigenvector signs and bases are arbitrary\([Lim et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib14)\), break equivariance, and a dot\-product Transformer then puts almost all attention \(0\.9990\.999\) on one node\.

On real molecules, neither graph\-side bound is tight: atoms rarely have symmetric counterparts, attention is nonlocal \(medianλ1≥0\.91\\lambda\_\{1\}\\geq 0\.91\), and the bounds rule out a sink in at most5\.1%5\.1\\%of heads in any run\. The third bound, on the spread of attention scores, does differ between architectures: GRIT clips its scores elementwise, so their spread never exceeds5​∥a∥15\\lVert a\\rVert\_\{1\}, whereas dot\-product scores can grow without limit with the attended node’s norm \(Proposition[A\.1](https://arxiv.org/html/2610.19324#A1.Thmtheorem1)\)\. Yet this cap does not explain why GRIT rarely sinks: in our trained models,5​∥a∥15\\lVert a\\rVert\_\{1\}ranges from2\.72\.7to11\.711\.7across layers, above the spread of2\.22\.2a sink needs at ZINC size, and removing the clip leaves GRIT’s sink rate low \(Section[4](https://arxiv.org/html/2610.19324#S4)\)\. The graph thus leaves GRIT and GraphGPS equally free to sink on the same molecules, yet their sink rates differ more than four\-fold: the models themselves must decide\.

## 4The attention scoring function drives the gap

On identical ZINC graphs, GraphGPS and GRIT differ in heads, positional encoding, dropout, and message passing as well as in scoring, so we change one component at a time \(Table[1](https://arxiv.org/html/2610.19324#S4.T1)\)\. Replacing only GRIT’s logit withzdotz^\{\\mathrm\{dot\}\}\(labelled GRIT\-dotlogit\), leaving RRWP, the pair representation and the values unchanged, raises its sink rate from0\.090±0\.0280\.090\\pm 0\.028to0\.680±0\.0980\.680\\pm 0\.098\(Figure[1](https://arxiv.org/html/2610.19324#S1.F1)\), above GraphGPS’s\. Removing message passing from GraphGPS does not lower its sink rate either \(0\.507±0\.1590\.507\\pm 0\.159\)\. Message passing is therefore not what suppresses sinks; the scoring function is\.

The swap also slows training \(test MAE0\.0640\.064to0\.3730\.373\), but two controls at matched accuracy show that this is not why it sinks\. Without any graph structure \(RRWP and bond types\), GRIT’s score and a dot product reach the same accuracy \(test MAE0\.5420\.542and0\.5430\.543, labelled GRIT\-blind and GRIT\-dot\) yet sink at rates of0\.2140\.214and0\.5320\.532\. Likewise, GraphGPS with a virtual node matches GRIT’s accuracy \(0\.0660\.066\) yet sinks six times as often \(0\.5590\.559\)\. Adding node RRWP and a bounded pair bias to GRIT\-dot \(labelled GRIT\-dotbias\) recovers over half the lost accuracy \(test MAE0\.2680\.268\) but leaves its sink rate unchanged \(0\.5210\.521\)\. Sink formation tracks mainly the form of the score; accuracy tracks access to graph structure\. Even a loss that rewards concentrated attention lifts GRIT’s sink rate only to0\.1310\.131\.

Table 1:Changing each component individually on ZINC \(mean±\\pm95%95\\%CI; all arms in Table[3](https://arxiv.org/html/2610.19324#A3.T3)\)\.*Ablation ratio*: test MAE after removing all attention onto the most\-attended node, divided by that for a random node\.Why large norms do not attract attention in GRIT\.A dot\-product logit grows linearly with the attended node’s hidden state, so a node with a large enough norm can claim almost all of a query’s attention \(Proposition[A\.1](https://arxiv.org/html/2610.19324#A1.Thmtheorem1)\)\. GRIT’s score instead saturates: in layers 2 to 10, at least96%96\\%of its coordinates sit at00or55, so the logit is a sum of readout weights selected by which coordinates fire, not by input magnitude\. Scaling up one node’s hidden state confirms this \(Table[4](https://arxiv.org/html/2610.19324#A3.T4)\): doubling it makes it a sink in up to38\.8%38\.8\\%of heads in GRIT\-dotlogit but at most0\.2%0\.2\\%in GRIT, and even a3232\-fold increase reaches only1\.3%1\.3\\%\. Removing the clip \(GRIT\-noclamp\) changes little: the model still sinks rarely \(0\.1130\.113\), and a doubled norm makes a sink in at most1\.4%1\.4\\%of heads, but an extreme32×32\\timesnorm increase does in up to54\.0%54\.0\\%\. GRIT thus grows high\-norm nodes without attending to them\.

Giving the model a token to sink on\.A virtual node is the graph counterpart of <BOS\>: chemically empty, connected to every node, akin to vision\-transformer registers\([Darcet et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib7)\), and exempt from the symmetry bound, since no automorphism can move it\. On the symmetric cycles of Section[3](https://arxiv.org/html/2610.19324#S3), where no real node may stand out, GraphGPS sends all of its attention to the virtual node, GRIT only about half to two\-thirds on average, and inconsistently across seeds\. On molecules, a virtual node leaves GRIT’s sink rate low \(0\.095±0\.0190\.095\\pm 0\.019\), even with a logit penalty toward it \(0\.079±0\.0270\.079\\pm 0\.027\)\. GraphGPS with a virtual node usually sinks onto an atom rather than onto the virtual node, and the sink’s value vector is about a tenth of the mean norm\. Symmetry says where a sink may appear; the scoring function decides whether one does\.

The ablation column of Table[1](https://arxiv.org/html/2610.19324#S4.T1)shows what these sinks are worth: removing all attention to the most\-attended node leaves the dot\-product models’ test MAE close to that for a random node \(1\.001\.00to1\.221\.22times\), but GRIT’s twice as high \(2\.062\.06\)\. In dot\-product models the sink is thus close to a no\-op, as in LLMs\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2)\), whereas GRIT’s most\-attended nodes matter to its predictions\.

## 5Discussion

Graph Transformers based on GRIT largely do not form attention sinks on molecules, and the graph is not the reason: symmetry forbids sinks only on inputs far more regular than a molecule, and the attention we measure is too nonlocal for locality to matter\. What matters most is the scoring function: GRIT’s logit barely responds to a node’s norm, and on molecules even a virtual node does not make it sink\. The LLM sink therefore looks like a property of dot\-product attention rather than of Transformers, consistent with sinks vanishing without softmax normalisation\([Gu et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib12)\)or under an output gate\([Qiu et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib20)\)\. Read as attentional networks over a complete graph\([Bronstein et al\., 2021](https://arxiv.org/html/2610.19324#bib.bib6)\), Transformers differ from GRIT in two ways, a first token and a dot\-product score, and our experiments separate the two: on molecules, dot\-product scoring sinks with or without a designated token, while GRIT’s gated scoring rarely sinks in either case\. Whether sinks protect Graph Transformers against over\-mixing remains open: our1010\-layer models show little over\-smoothing to prevent \(Appendix[C](https://arxiv.org/html/2610.19324#A3)\)\.

Our study has limits: one gated\-score architecture against dot\-product baselines, on molecular graphs alone, at1010layers; several GRIT virtual\-node quantities remain seed\-dependent; and we measure reception rather than mixing, since a perfect sink can also be a complete mixer\. Further architectures\([Yuan et al\., 2026](https://arxiv.org/html/2610.19324#bib.bib28), EGT, Graphormer, SAN;\)and non\-molecular graphs come next\.

## References

- Barbero et al\. \(2024\)Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João G\. M\. Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković\.Transformers need glasses\! Information over\-squashing in language tasks\.*Advances in Neural Information Processing Systems*, 37:98111–98142, 2024\.
- Barbero et al\. \(2025\)Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Petar Veličković, Razvan Pascanu, and Michael Bronstein\.Why do LLMs attend to the first token?In*Conference on Language Modeling*, 2025\.
- Biccari et al\. \(2026\)Umberto Biccari, Qian Huang, and Enrique Zuazua\.Interior interpretability with attention rollout: Contraction and propagation profiles in transformers\.*arXiv preprint arXiv:2607\.22367*, 2026\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning*, 2023\.
- Bini et al\. \(2024\)Lorenzo Bini, Marco Sorbi, and Stéphane Marchand\-Maillet\.Massive activations in graph neural networks: Decoding attention for domain\-dependent interpretability\.*arXiv preprint arXiv:2409\.03463*, 2024\.
- Bronstein et al\. \(2021\)Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković\.Geometric deep learning: Grids, groups, graphs, geodesics, and gauges\.*arXiv preprint arXiv:2104\.13478*, 2021\.
- Darcet et al\. \(2024\)Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski\.Vision transformers need registers\.In*International Conference on Learning Representations*, 2024\.
- Di Giovanni et al\. \(2023\)Francesco Di Giovanni, James Rowbottom, Benjamin P Chamberlain, Thomas Markovich, and Michael M Bronstein\.Understanding convolution on graphs via energies\.*Transactions on Machine Learning Research*, 2023\.
- Dong et al\. \(2021\)Yihe Dong, Jean\-Baptiste Cordonnier, and Andreas Loukas\.Attention is not all you need: Pure attention loses rank doubly exponentially with depth\.In*International Conference on Machine Learning*, pages 2793–2803\. PMLR, 2021\.
- Dwivedi et al\. \(2022a\)Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson\.Graph neural networks with learnable structural and positional representations\.In*International Conference on Learning Representations*, 2022a\.
- Dwivedi et al\. \(2022b\)Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini\.Long range graph benchmark\.*Advances in Neural Information Processing Systems*, 35:22326–22340, 2022b\.
- Gu et al\. \(2025\)Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin\.When attention sink emerges in language models: An empirical view\.In*International Conference on Learning Representations*, 2025\.
- Guan et al\. \(2025\)Zhong Guan, Likang Wu, Hongke Zhao, Ming He, and Jianping Fan\.Attention mechanisms perspective: Exploring LLM processing of graph\-structured data\.In*International Conference on Machine Learning*, pages 20612–20639\. PMLR, 2025\.
- Lim et al\. \(2023\)Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka\.Sign and basis invariant networks for spectral graph representation learning\.In*International Conference on Learning Representations*, 2023\.
- Liu et al\. \(2026\)Yang Liu, Dongxin Guo, Tom Zheng, Siu Ming Yiu, Liam Ning, and Jikun Wu\.Capacity\-controlled global attention for graph transformers\.*arXiv preprint arXiv:2604\.17324*, 2026\.
- Ma et al\. \(2023\)Liheng Ma, Chen Lin, Derek Lim, Adriana Romero\-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser\-Nam Lim\.Graph inductive biases in transformers without message passing\.In*International Conference on Machine Learning*, pages 23321–23337\. PMLR, 2023\.
- McIntosh\-Smith et al\. \(2024\)Simon McIntosh\-Smith, Sadaf R Alam, and Christopher Woods\.Isambard\-AI: a leadership class supercomputer optimised specifically for artificial intelligence\.*arXiv preprint arXiv:2410\.11199*, 2024\.
- Merity et al\. \(2017\)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations*, 2017\.
- Morris et al\. \(2024\)Matthew Morris, Bernardo Cuenca Grau, and Ian Horrocks\.Orbit\-equivariant graph neural networks\.In*International Conference on Learning Representations*, 2024\.
- Qiu et al\. \(2025\)Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin\.Gated attention for large language models: Non\-linearity, sparsity, and attention\-sink\-free\.In*Advances in Neural Information Processing Systems*, 2025\.
- Queipo\-de\-Llano et al\. \(2026\)Enrique Queipo\-de\-Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz\-Ziv\.Attention sinks and compression valleys in LLMs are two sides of the same coin\.In*International Conference on Learning Representations*, 2026\.
- Radford et al\. \(2019\)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.Technical report, OpenAI, 2019\.
- Rampášek et al\. \(2022\)Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini\.Recipe for a general, powerful, scalable graph transformer\.*Advances in Neural Information Processing Systems*, 35:14501–14515, 2022\.
- Rulli et al\. \(2025\)Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, and Alessio Devoto\.Attention sinks in diffusion language models\.*arXiv preprint arXiv:2510\.15731*, 2025\.
- Veličković et al\. \(2025\)Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu\.Softmax is not enough \(for sharp size generalisation\)\.In*International Conference on Machine Learning*, pages 61190–61211\. PMLR, 2025\.
- Wu et al\. \(2023\)Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie\.Demystifying oversmoothing in attention\-based graph neural networks\.In*Advances in Neural Information Processing Systems*, 2023\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations*, 2024\.
- Yuan et al\. \(2026\)Chaohao Yuan, Kangfei Zhao, Ercan Engin Kuruoglu, Liang Wang, Tingyang Xu, Wenbing Huang, Deli Zhao, Hong Cheng, and Yu Rong\.A survey of graph transformers: Architectures, theories and applications\.*ACM Computing Surveys*, 2026\.doi:10\.1145/3834764\.
- Zhang et al\. \(2024\)Xu Zhang, Yonghui Xu, Wei He, Wei Guo, and Lizhen Cui\.A comprehensive review of the oversmoothing in graph neural networks\.In*Computer Supported Cooperative Work and Social Computing*, pages 451–465\. Springer, 2024\.

## Acknowledgments

I would like to thank Professor Michael Bronstein for his class on Geometric Deep Learning at the University of Oxford, where this research on Graph Transformers started as a course project\.

The author acknowledges the use of resources provided by the Isambard\-AI National AI Research Resource \(AIRR\)\([McIntosh\-Smith et al\., 2024](https://arxiv.org/html/2610.19324#bib.bib17)\)\. Isambard\-AI is operated by the University of Bristol and is funded by the UK Government’s Department for Business, Innovation, Science and Trade \(DBIST\) via UK Research and Innovation; and the Science and Technology Facilities Council \[ST/AIRR/I\-A\-I/1023\]\.

## Appendix AProofs and supporting theory

### A\.1Proof of Proposition[3\.1](https://arxiv.org/html/2610.19324#S3.Thmtheorem1)

#### Proof\.

*Orbit term\.*For an automorphismPPof the labelled input,P​Z=ZPZ=Zand equivariance giveA⁡\(Z\)=P​A​\(Z\)​P⊤A\(Z\)=PA\(Z\)P^\{\\top\}, so withs⁡\(A\)=1n​A⊤​𝟏s\(A\)=\\tfrac\{1\}\{n\}A^\{\\top\}\\mathbf\{1\}andP⊤​𝟏=𝟏P^\{\\top\}\\mathbf\{1\}=\\mathbf\{1\}we getP​s​\(A\)=s⁡\(A\)Ps\(A\)=s\(A\): the sink score is constant on every orbit\. Row\-stochasticity gives∑jsj​\(A\)=1\\sum\_\{j\}s\_\{j\}\(A\)=1, so an orbit of sizemmhassj​\(A\)≤1/ms\_\{j\}\(A\)\\leq 1/m, and a vertex\-transitive input is one orbit of sizenn, forcingsj​\(A\)=1/ns\_\{j\}\(A\)=1/n\.

*Locality term\.*Foru∈BG​\(j,r\)u\\in B\_\{G\}\(j,r\), row\-stochasticity givesAu​j≤1A\_\{uj\}\\leq 1; foru∉BG​\(j,r\)u\\notin B\_\{G\}\(j,r\),dG​\(u,j\)\>rd\_\{G\}\(u,j\)\>rgivesAu​j≤∑v:dG​\(u,v\)\>rAu​v=ℓu\(r\)≤λr\(A\)A\_\{uj\}\\leq\\sum\_\{v:\\,d\_\{G\}\(u,v\)\>r\}A\_\{uv\}=\\ell\_\{u\}^\{\(r\)\}\\leq\\lambda\_\{r\}\(A\)\. Therefore

sj​\(A\)≤\(\|BG​\(j,r\)\|\+∑u∉BG​\(j,r\)ℓu\(r\)\)/n≤λr​\(A\)\+\(1−λr​\(A\)\)​\|BG​\(j,r\)\|/n,s\_\{j\}\(A\)\\;\\leq\\;\\bigl\(\|B\_\{G\}\(j,r\)\|\+\\textstyle\\sum\_\{u\\notin B\_\{G\}\(j,r\)\}\\ell\_\{u\}^\{\(r\)\}\\bigr\)/n\\;\\leq\\;\\lambda\_\{r\}\(A\)\+\(1\-\\lambda\_\{r\}\(A\)\)\\,\|B\_\{G\}\(j,r\)\|/n,where the first inequality is the per\-node form evaluated in the certificate of Appendix[B](https://arxiv.org/html/2610.19324#A2)\.

*Range term\.*Fix a queryuu\. Forv≠jv\\neq jthe definition ofΔ\\Deltagiveszu​v≥zu​j−Δz\_\{uv\}\\geq z\_\{uj\}\-\\Delta, soAu​j=ezu​j/∑vezu​v≤ezu​j/\(ezu​j\+\(n−1\)​ezu​j−Δ\)=eΔ/\(eΔ\+n−1\)A\_\{uj\}=e^\{z\_\{uj\}\}/\\sum\_\{v\}e^\{z\_\{uv\}\}\\leq e^\{z\_\{uj\}\}/\\bigl\(e^\{z\_\{uj\}\}\+\(n\-1\)e^\{z\_\{uj\}\-\\Delta\}\\bigr\)=e^\{\\Delta\}/\(e^\{\\Delta\}\+n\-1\)\. Averaging overuugives the same bound forsj​\(A\)s\_\{j\}\(A\), andeΔ/\(eΔ\+n−1\)≥εe^\{\\Delta\}/\(e^\{\\Delta\}\+n\-1\)\\geq\\varepsilongivesΔ≥log⁡\(\(n−1\)​ε/\(1−ε\)\)\\Delta\\geq\\log\(\(n\-1\)\\varepsilon/\(1\-\\varepsilon\)\)\. Taking the minimum of the three bounds proves the proposition\.□\\square

### A\.2Score spread

###### Proposition A\.1\(Bounded and unbounded score spread\)

For the GRIT score \([1](https://arxiv.org/html/2610.19324#S2.E1)\), writee^u​j∈\[0,5\]h\\widehat\{e\}\_\{uj\}\\in\[0,5\]^\{h\}for the clipped vector thataareads out; hence5∑i:ai<0ai≤zu​j≤5∑i:ai\>0ai5\\sum\_\{i:a\_\{i\}<0\}a\_\{i\}\\leq z\_\{uj\}\\leq 5\\sum\_\{i:a\_\{i\}\>0\}a\_\{i\}andΔ≤5​∥a∥1\\Delta\\leq 5\\lVert a\\rVert\_\{1\}whatever the node features, with the constant sharp over the box\[0,5\]h\[0,5\]^\{h\}\. For dot\-product scoring no such bound exists: ifxjx\_\{j\}is the input state of nodejjand⟨qu,WK​xj⟩≠0\\langle q\_\{u\},W\_\{K\}x\_\{j\}\\rangle\\neq 0for someu≠ju\\neq j, replacingxjx\_\{j\}byρ​xj\\rho x\_\{j\}makesΔ\\Deltagrow without bound inρ\\rho, and a bounded edge\-bias term cannot restore a bound\.

#### Proof\.

The clip and rectifier confine each coordinate to\[0,5\]\[0,5\], and a linear functional on that box is extremised at the two sign\-selected corners, whose difference is5​∑i\|ai\|5\\sum\_\{i\}\|a\_\{i\}\|\. For the dot product andu≠ju\\neq j, onlykj​\(ρ\)=ρ​WK​xj\+bKk\_\{j\}\(\\rho\)=\\rho W\_\{K\}x\_\{j\}\+b\_\{K\}moves, sozu​jz\_\{uj\}is affine inρ\\rhowith nonzero slope while the rest of rowuuis fixed, andΔ\\Deltais at least the gap between two entries of that row\.□\\square

Atε=0\.3\\varepsilon=0\.3a sink needs a spread of2\.242\.24atn=23n=23and4\.744\.74atn=269n=269, against5​∥a∥1=2\.745\\lVert a\\rVert\_\{1\}=2\.74to11\.6911\.69across the ZINC layers, so the bound is loose \(Section[3](https://arxiv.org/html/2610.19324#S3)\)\.

Reception is not mixing\.A sink score does not determine how much a layer mixes, the distinction that spectral and Dobrushin\-type analyses of over\-smoothing and attention propagation turn on\([Wu et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib26);[Biccari et al\., 2026](https://arxiv.org/html/2610.19324#bib.bib3)\)\. Withκ\(A\)=12maxu,t∥Au:−At:∥1\\kappa\(A\)=\\tfrac\{1\}\{2\}\\max\_\{u,t\}\\lVert A\_\{u:\}\-A\_\{t:\}\\rVert\_\{1\}the Dobrushin coefficient,diam⁡\(A​Y\)≤κ⁡\(A\)​diam⁡\(Y\)\\operatorname\{diam\}\(AY\)\\leq\\kappa\(A\)\\operatorname\{diam\}\(Y\)for any featuresYY\. The identityInI\_\{n\}and the uniform matrixUn=1n​𝟏𝟏⊤U\_\{n\}=\\tfrac\{1\}\{n\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}share the flat profiles=𝟏/ns=\\mathbf\{1\}/nyet haveκ⁡\(In\)=1\\kappa\(I\_\{n\}\)=1andκ⁡\(Un\)=0\\kappa\(U\_\{n\}\)=0, while a perfect sink𝟏​ev⊤\\mathbf\{1\}e\_\{v\}^\{\\top\}has the maximal profiles=evs=e\_\{v\}yetκ=0\\kappa=0\. We therefore measure mixing separately \(Appendix[C](https://arxiv.org/html/2610.19324#A3)\)\.

## Appendix BExperimental details

Table 2:Experimental hyperparameters\. Peptides\-func graphs are capped at 300 nodes to limit memory usage, and each quartile model is trained on 2,023 graphs of its quartile at batch size 16\. All models are trained with AdamW, linear warmup, cosine annealing, and gradient clipping on a single GPU\. Every other arm inherits its base model’s configuration, changing only the named component\.Arms\.GRIT\-dotlogit replaceszGRITz^\{\\mathrm\{GRIT\}\}byzdotz^\{\\mathrm\{dot\}\}in Eq\. \([1](https://arxiv.org/html/2610.19324#S2.E1)\) and keeps everything else, including RRWP, the pair representation and the values; GRIT\-noclamp removes the clip\. Like the official code, GRIT\-official clips the scalar logit to\[−5,5\]\[\-5,5\]rather than each coordinate; unlike that code, all our GRIT arms use one readoutaafor every head\. GRIT\-blind removes node and pair RRWP and bond types; GRIT\-dot is GRIT\-blind withzdotz^\{\\mathrm\{dot\}\}and no pair representation; GRIT\-dotbias adds to GRIT\-dot node RRWP and a bounded pair biasa⊤​clip​\(W​eu​j\)a^\{\\top\}\\mathrm\{clip\}\(We\_\{uj\}\)in the logit\. GRIT\-entropy\-1/2 add an attention\-entropy penalty of two strengths\. Virtual\-node arms join one node to every atom, with the positional encoding padded or recomputed \(PE rec\.\); GRIT \+ virtual node \+ bias adds a learned logit penalty toward it\. GraphGPS\-none removes the MPNN, GraphGPS\-GatedGCN replaces it, and Transformer \+ LapPE is GraphGPS\-none with sign\-ambiguous Laplacian encodings \(8 eigenvectors\)\. Seeds are fixed per run, we report seed\-controlled runs only, and checkpoints are best\-validation\.

Interventions\.*Sink ablation*: on 500 test graphs, the most\-attended node \(head\-averaged received attention summed over layers\) has its attention column zeroed in every layer, removing that mass \(Table[1](https://arxiv.org/html/2610.19324#S4.T1)\) or renormalising the rows \(Table[3](https://arxiv.org/html/2610.19324#A3.T3)\); the control ablates a uniformly random node\.*Norm intervention*: at layerℓ∈\{3,5,8\}\\ell\\in\\\{3,5,8\\\}the input state of one uniformly chosen real node is multiplied byρ\\rhobefore that layer’s attention, on 200 test graphs, and we read that layer’s attention only\.*Language models*: Pythia\-70M/160M\([Biderman et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib4)\)and GPT\-2 small\([Radford et al\., 2019](https://arxiv.org/html/2610.19324#bib.bib22)\)on 200 non\-overlapping WikiText\-2 test windows\([Merity et al\., 2017](https://arxiv.org/html/2610.19324#bib.bib18)\)ofTTtokens without a BOS token, with the same sink\-rate estimator\.

Certificate and probe\.For Proposition[3\.1](https://arxiv.org/html/2610.19324#S3.Thmtheorem1)we evaluate, on 300 test graphs per run, layer, and head, the row leakagesℓu\(r\)\\ell\_\{u\}^\{\(r\)\}andλr​\(A\)\\lambda\_\{r\}\(A\)forr≤3r\\leq 3, the ball volumes, the orbits via VF2 with capped enumeration \(a cap only shrinks orbits, so1/\|𝒪⁡\(j\)\|1/\|\\mathcal\{O\}\(j\)\|stays valid\), and the per\-node boundUj=min⁡\{1/\|𝒪⁡\(j\)\|,minr⁡\(\|BG​\(j,r\)\|\+∑u∉BG​\(j,r\)ℓu\(r\)\)/n\}U\_\{j\}=\\min\\\{1/\|\\mathcal\{O\}\(j\)\|,\\min\_\{r\}\(\|B\_\{G\}\(j,r\)\|\+\\sum\_\{u\\notin B\_\{G\}\(j,r\)\}\\ell\_\{u\}^\{\(r\)\}\)/n\\\}\. The locality bound was never violated, and within\-orbit deviation was at most8\.7×10−58\.7\\times 10^\{\-5\}for every equivariant\-PE run except GRIT\-dotbias \(0\.0160\.016, with22orbit\-bound violations in total\), which doubles as an equivariance test\. The probe passes cyclesC8C\_\{8\}–C24C\_\{24\}of identical atoms and bonds through the checkpoints; virtual\-node arms use the training\-time transform, so the virtual node is a singleton orbit beside thenncycle nodes and the two are reported separately\. On the cycle probe, received attention is uniform to7\.8×10−77\.8\\times 10^\{\-7\}in every equivariant\-PE checkpoint except GRIT\-dotbias and GRIT\-dotlogit, which amplify float32 rounding \(deviation up to0\.960\.96\) and are uniform to9\.6×10−49\.6\\times 10^\{\-4\}in float64\. Real nodes of virtual\-node arms stay uniform to2\.5×10−42\.5\\times 10^\{\-4\}\.

## Appendix CAdditional results

Training verification\.GRIT reaches test MAE0\.0640\.064on ZINC, close to the published result\([Ma et al\., 2023](https://arxiv.org/html/2610.19324#bib.bib16)\), and test AP0\.6340\.634on the full Peptides\-func dataset \(0\.6120\.612on the smallest quartile,0\.4990\.499on the largest\), below the published result; GraphGPS reaches0\.0720\.072on ZINC\. At GRIT’s 2,000\-epoch budget, GRIT\-dotlogit appears under\-trained, with an unconverged training loss\.

Sink ablations\.Table[1](https://arxiv.org/html/2610.19324#S4.T1)removes the attention onto the most\-attended node outright\. If each row is instead renormalised, so that this attention is spread over the other nodes, GRIT’s ablation ratio falls from2\.062\.06to1\.351\.35, but every gated arm \(1\.251\.25to1\.611\.61\) stays above every dot\-product arm \(1\.001\.00to1\.231\.23\) except GraphGPS\-none, whose ratio varies widely across seeds \(2\.032\.03; Table[3](https://arxiv.org/html/2610.19324#A3.T3)\)\.

Thresholds and size\.GRIT sinks less than each of the seven dot\-product arms we swept at every threshold fromε=0\.1\\varepsilon=0\.1to0\.80\.8\(closest atε=0\.8\\varepsilon=0\.8:0\.020\.02vs\.0\.050\.05\), and the threshold\-free peak sharen​maxj​sjn\\max\_\{j\}s\_\{j\}separates gated arms \(2\.52\.5to3\.63\.6\) from dot\-product arms \(6\.06\.0to13\.113\.1\) with no overlap \(Table[3](https://arxiv.org/html/2610.19324#A3.T3)\)\. On Peptides\-func, GRIT’s largest sink score falls with graph size \(0\.1350\.135to0\.0660\.066\) while its peak share rises \(7\.47\.4to16\.116\.1\), as a score spread that does not grow withnnpredicts\.

Language models\.With the same estimator, the three language models reach sink rates of 0\.95–1\.00 atT=23T=23and 0\.39–1\.00 atT=269T=269\. Compared to our experiments, these language models were trained on far longer contexts, which bring stronger sinks\([Barbero et al\., 2025](https://arxiv.org/html/2610.19324#bib.bib2)\)\.

Virtual\-node details\.At the peak\-sinking layer, the virtual node of GRIT \+ virtual node is never the most\-attended node, and it almost never receives enough attention to be a sink\. In GraphGPS \+ virtual node it is the most\-attended node in10%10\\%of graphs; when it is, its value norm is about a tenth of the mean\.

Over\-mixing\.From first to last layer the Dirichlet energy relative to the squared feature norm grows in every arm \(on average1\.131\.13\- to5\.455\.45\-fold\) except GRIT \+ virtual node \(PE rec\.\), where it falls to 0\.71–0\.92 of its input value, and the distance from consensus∥X−𝟏​x¯⊤∥F/∥X∥F\\lVert X\-\\mathbf\{1\}\\bar\{x\}^\{\\top\}\\rVert\_\{F\}/\\lVert X\\rVert\_\{F\}rises in every arm: at1010layers there is little over\-smoothing for a sink to prevent\.

Table 3:All ZINC arms \(mean±\\pm95%95\\%CI over three seeds, five for GraphGPS\)\.n​maxj​sjn\\max\_\{j\}s\_\{j\}: attention received by the most\-attended node, as a multiple of the uniform share\.*Ablation ratio*: test MAE after removing all attention onto the most\-attended node, divided by that for a random node\.Table 4:% of heads in which one node, whose input is scaled byρ\\rho, becomes a sink \(max over layers\)\.