What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

# What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation
Source: [https://arxiv.org/html/2607.22781](https://arxiv.org/html/2607.22781)
Minwoo Yu and Young\-guk HaM\. Yu and Y\.\-G\. Ha are with the Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea\. E\-mail: \{snowypainter, ygha\}@konkuk\.ac\.kr\. Y\.\-G\. Ha is the corresponding author\.

###### Abstract

High task performance alone does not establish whether a model retains the structural information it uses for prediction in its internal representation\. Temporal graph models, in particular, can achieve high future\-link AUC while basic graph statistics remain inconsistently recoverable from the same representation\. We identify one structural source of this gap in the weighted averaging performed by standard attention\. When an evidence pattern is repeated, the numerator and denominator of standard attention grow at the same rate; consequently, inputs with different amounts of accumulated evidence can produce the same aggregate\.

We address this limitation withMass\-Aware Attention \(MAA\), which generalizes standardL1L\_\{1\}normalization to anLpL\_\{p\}family of operators\. By making the numerator and denominator scale at different rates under repetition, MAA retains the effective number of contributing inputs in the magnitude of the representation\. It requires no additional supervision, parameters, hidden dimensions, or explicit count features, and recovers standard attention as the special casep=1p=1\.

Across four CTDG models and three datasets, MAA improves future\-link AUC in 11 of 12 model–dataset cells\. Linear recovery from the same hidden representation increases by 4\.49% on average, while preferential\-attachment recovery improves in all 12 cells and remains significant after family\-wise correction\. We confirm this effect in CTDG and observe consistent cross\-domain evidence in four additional settings: candidate\-mark counts in MTPP, pair recurrence in TKG, retained\-passage conditions in RAG, and local event counts in STPP\. Information accessibility and task utility remain distinct: NLL improves in MTPP and ranking is largely preserved in TKG, whereas the additional information in RAG does not improve the diagnostic head, and a downstream LayerNorm can erase the signal in STPP\. These results establish MAA as a general normalization principle for improving the informativeness of predictor\-facing representations by controlling the repetition invariance hard\-coded into standard attention\.

###### Index Terms:

Mass\-Aware Attention, attention normalization, representation auditing, temporal graph learning, evidence accumulation\.

## 1Introduction

High benchmark performance shows how accurately a model distinguishes future links, but it does not reveal how much pre\-query graph structure—such as pair frequency or degree accumulation—remains in the predictor\-facing representation\. This distinction is especially important for temporal graphs, where repeated interactions, node activity, recency, and periodicity can all support future\-link prediction\. Yet simple heuristics can achieve strong benchmark performance because of easy negative sampling and recurring edges\[[18](https://arxiv.org/html/2607.22781#bib.bib6),[8](https://arxiv.org/html/2607.22781#bib.bib8)\], and model predictions do not consistently reflect several graph characteristics\[[11](https://arxiv.org/html/2607.22781#bib.bib9)\]\. A model that learns a temporal graph effectively should therefore make prediction\-relevant graph statistics accessible in the representation supplied to its predictor\. In this work, statistic recovery is not a substitute for task performance; it is a*representation audit*of this condition\.

We study a small but pervasive blind spot shared by attention models across domains\. Standard softmax attention returns a weighted average of its input values\. When the same evidence pattern is repeated, its numerator andL1L\_\{1\}denominator can grow together, leaving the aggregate unchanged\. The operator preserves which evidence is relatively important while weakening how broadly that evidence has accumulated\. This effect can appear as low graph\-statistic recovery in CTDG, weak accessibility of cumulative candidate\-mark occurrences in MTPP, query\-specific pair recurrence in TKG, evidence completeness across passages in RAG, and event counts around a candidate location in STPP\.

We introduceMass\-Aware Attention \(MAA\)to address this blind spot\. The only modification is to generalize theL1L\_\{1\}norm in the attention denominator to anLpL\_\{p\}norm\. Standard attention becomes the pointp=1p=1; forp\>1p\>1, the numerator and denominator scale at different rates under repetition\. MAA thereby preserves the weighted direction while retaining an input\-dependent scale derived from the effective support of the attention distribution\. It uses no explicit count, auxiliary target, additional parameter, or hidden coordinate\. Instead, it exposes the exact repetition invariance hard\-coded by standard attention as a continuous control axis\.

Our evaluation follows a hierarchy of evidence\. The primary question is whether target information becomes more recoverable from the same predictor\-facing hidden representationZZ\. We then measure whether this change preserves or improves the original task\. Finally, a2×22\\times 2representation audit, an explicit\-cardinality control, output normalization, and power sensitivity distinguish MAA from coordinate exposure and uniform changes in representation norm\. Figure[1](https://arxiv.org/html/2607.22781#S1.F1)summarizes the central CTDG results, and Table[I](https://arxiv.org/html/2607.22781#S1.T1)shows how the common question is instantiated in each domain\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x1.png)Figure 1:Main CTDG results\. Linear recovery and context sensitivity increase on average\. Confirmatory evidence is strongest for preferential\-attachment recovery, which improves in 12/12 cells, and future\-link AUC, which improves in 11/12 cells\. Effect sizes vary across probes and statistics\.TABLE I:Evidence\-accumulation problem and evaluation target in each domain\. Every recovery result is measured from an equal\-dimensional predictor\-facing representationZZwithout auxiliary coordinates\.DomainInput distinctionInformation recovered fromZZTask\-utility metricCTDGDifferences in repeated interactions and temporal neighborhoodsPair frequency, recency, persistence, periodicity, granularity, density, homophily, and preferential attachmentFuture\-link ROC–AUC and context sensitivityMTPPAccumulation of preceding events supporting a candidate markGlobal and recent occurrence counts of the candidate markPoint\-process/mark NLL, mark accuracy, and time errorTKGRepeated historical facts connecting the current head and candidate tailPair recurrence, relation diversity, and two\-hop path countFiltered MRR, Hits@10, and hard\-negative AUCRAGComplete supporting passages versus inputs missing part of the evidenceRetained\-passage condition with literal\-duplicate cues removedDiagnostic AUC for complete versus partial evidenceSTPPPreceding events accumulated around a candidate locationLocal event counts at multiple radiiNELBO and spatial/temporal log\-likelihoodOur contributions are as follows\.

- •We reinterpret the repetition invariance of standard attention as thep=1p=1special case of anLpL\_\{p\}normalization family and introduceMAA, a parameter\-free mechanism for controlling it\.
- •We jointly evaluate recovery from the same representation, behavioral sensitivity, and task performance across four CTDG architectures, three datasets, and three seeds\. Preferential\-attachment recovery improves in all 12 model–dataset cells, and AUC improves in 11\.
- •We extend the evaluation to MTPP, TKG, RAG, and STPP, showing that one normalization principle improves access to distinct forms of accumulated evidence\.
- •We characterize the operational boundary ofMAAthrough information that becomes accessible but remains unused in RAG and information that a downstream normalization removes in STPP\.

## 2Related Work

This section reviews temporal representation auditing and attention normalization, and distinguishesMAAfrom explicit cardinality injection\.

Standard scaled dot\-product attention forms a weighted average of values using normalized coefficients\[[20](https://arxiv.org/html/2607.22781#bib.bib1)\]\. We study the repetition invariance imposed by the denominator of this basic operator\.

### 2\.1Temporal Graph Learning and Representation Auditing

TGAT constructs inductive node representations using time encoding and temporal attention\[[22](https://arxiv.org/html/2607.22781#bib.bib2)\]\. GraphMixer combines link and node encoders with an MLP mixer\[[6](https://arxiv.org/html/2607.22781#bib.bib3)\], while DyGFormer encodes interaction sequences as patches\[[26](https://arxiv.org/html/2607.22781#bib.bib4)\]\. CRAFT achieves strong future\-link performance using node identities and target\-aware cross\-attention\[[25](https://arxiv.org/html/2607.22781#bib.bib5)\]\.MAAdoes not replace these backbones; it modifies only the normalization at the attention boundary that forms the predictor\-facing representation\.

In dynamic link prediction, recurring edges and memorization can dominate under particular evaluation protocols\[[18](https://arxiv.org/html/2607.22781#bib.bib6)\]\. Hayes et al\. show through interventions that model predictions reflect temporal graph characteristics unevenly\[[11](https://arxiv.org/html/2607.22781#bib.bib9)\]\. Their analysis is behavioral, whereas ours examines frozen internal representations\. We measure the recoverability of query\-level graph statistics and test an operator that improves their accessibility without statistic supervision\.

### 2\.2Representation Probing

Linear probes are a standard tool for measuring whether a target is accessible through a simple readout from a frozen representation\[[1](https://arxiv.org/html/2607.22781#bib.bib27),[7](https://arxiv.org/html/2607.22781#bib.bib28),[12](https://arxiv.org/html/2607.22781#bib.bib29)\]\. Because a probe may itself learn the target or exploit surface correlations, its capacity and held\-out protocol must be interpreted separately\[[16](https://arxiv.org/html/2607.22781#bib.bib30)\]\. We report Ridge and ExtraTrees probes on chronological held\-out splits, never propagate probe gradients into the original model, and evaluate recovery together with behavioral sensitivity and task performance\. This design separates information that is readable from information that the predictor actually uses\.

### 2\.3Cardinality Preservation and Probability Normalization

Cardinality\-Preserved Attention \(CPA\) identifies the inability of graph attention to distinguish multisets with different multiplicities and augments the aggregate with an unweighted sum or explicit neighborhood cardinality\[[27](https://arxiv.org/html/2607.22781#bib.bib11)\]\. Its motivation is closely related, but its information source and operator differ from ours\. CPA injects the observed token count or an unweighted aggregate through a separate branch\.MAAinstead derives its scale from the effective support and concentration of the attention distribution already computed by the model\. It can therefore assign different scales to inputs with identical cardinality but different attention distributions, whileppcontinuously controls the strength of the invariance\.

Normalized Attention Without Probability Cage replaces softmax with normalization based on the mean and variance of logits and learnable gain and bias, motivated by the probability\-simplex and convex\-hull constraints\[[19](https://arxiv.org/html/2607.22781#bib.bib12)\]\. Affine\-Scaled Attention learns an input\-dependent affine transformation after softmax\[[3](https://arxiv.org/html/2607.22781#bib.bib13)\]\. In contrast,MAApreserves the standard weighted direction and learns no additional gate: its scale is deterministically computed from the normalized attentionα\\alpha\. The distinctive contribution ofMAAis thus to expose theL1L\_\{1\}repetition invariance hard\-coded into standard attention as an explicit control axis in anLpL\_\{p\}family\.

### 2\.4Evidence Accumulation Across Domains

In temporal point processes, event history determines mark\-specific conditional intensities and the distribution of the next event, which recurrent and attention\-based models represent in different ways\[[9](https://arxiv.org/html/2607.22781#bib.bib14),[17](https://arxiv.org/html/2607.22781#bib.bib15),[30](https://arxiv.org/html/2607.22781#bib.bib16),[23](https://arxiv.org/html/2607.22781#bib.bib17)\]\. In temporal knowledge graph forecasting, recurrent facts and relational paths support candidate\-entity ranking\[[14](https://arxiv.org/html/2607.22781#bib.bib18),[10](https://arxiv.org/html/2607.22781#bib.bib19),[29](https://arxiv.org/html/2607.22781#bib.bib20)\]\. Multi\-hop QA requires integrating supporting facts distributed across passages\[[24](https://arxiv.org/html/2607.22781#bib.bib21),[13](https://arxiv.org/html/2607.22781#bib.bib22)\], while spatio\-temporal point processes condition spatial intensity on local event density and revisit patterns\[[5](https://arxiv.org/html/2607.22781#bib.bib23),[28](https://arxiv.org/html/2607.22781#bib.bib24)\]\. Although these tasks and data structures differ, they share an attention boundary that conveys accumulated evidence to a predictor through a weighted average\. We therefore use domain\-specific targets that directly represent accumulation rather than imposing one statistic across domains\.

Layer normalization standardizes hidden\-state scale\[[2](https://arxiv.org/html/2607.22781#bib.bib25)\], and its placement before or after attention changes optimization and information flow in Transformers\[[21](https://arxiv.org/html/2607.22781#bib.bib26)\]\. Our STPP analysis identifies a concrete boundary at which this general architectural choice determines whether the input\-dependent amplitude introduced byMAAreaches the predictor\-facing representation\.

## 3Mass\-Aware Attention

This section formalizes the repetition invariance of standard weighted averaging and presents theLpL\_\{p\}normalization that relaxes it continuously\.

### 3\.1Repetition Invariance of Standard Attention

Let the inputs participating in the aggregation for one query beX=\{\(mi,vi\)\}i=1nX=\\\{\(m\_\{i\},v\_\{i\}\)\\\}\_\{i=1\}^\{n\}, whereviv\_\{i\}is a value andmi=exp⁡\(ai−maxj⁡aj\)m\_\{i\}=\\exp\(a\_\{i\}\-\\max\_\{j\}a\_\{j\}\)is a numerically stabilized positive attention weight\. Under this notation, standard softmax aggregation is defined by

A1​\(X\)=∑imi​vi∥m∥1\.A\_\{1\}\(X\)=\\frac\{\\sum\_\{i\}m\_\{i\}v\_\{i\}\}\{\\lVert m\\rVert\_\{1\}\}\.\(1\)Equation \([1](https://arxiv.org/html/2607.22781#S3.E1)\) divides the weighted sum of values by the totalL1L\_\{1\}mass, forcing the coefficients to sum exactly to one\. To examine how this operator treats multiplicity, letX\(r\)X^\{\(r\)\}containrrcopies of the same weighted\-value motif\. Replication multiplies both the numerator and denominator byrr, yielding

A1​\(X\(r\)\)=A1​\(X\)\.A\_\{1\}\(X^\{\(r\)\}\)=A\_\{1\}\(X\)\.\(2\)Equation \([2](https://arxiv.org/html/2607.22781#S3.E2)\) is a mathematical thought experiment rather than a training procedure that duplicates real observations\. It demonstrates that standard weighted averaging preserves the relative evidence pattern but may map different numbers of repetitions of that pattern to the same aggregate\.

Figure[2](https://arxiv.org/html/2607.22781#S3.F2)visualizes this invariance and the change introduced byMAA\. Standard attention collapses replicated inputs to the same aggregate, whereasMAApreserves the weighted direction while retaining an amplitude difference induced by repetition\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x2.png)Figure 2:Intuition behind Mass\-Aware Attention\. Standard attention can map repetitions of the same weighted\-value pattern to an identical aggregate\.MAAmakes the numerator and denominator grow at different rates under repetition, preserving the weighted direction while retaining an effective\-input\-dependent amplitude\.
### 3\.2TheLpL\_\{p\}Normalization Family

MAAextends the exactL1L\_\{1\}normalization hard\-coded into standard attention to a continuous family of operators\. It replaces only the denominator in \([1](https://arxiv.org/html/2607.22781#S3.E1)\) with anLpL\_\{p\}norm:

Ap​\(X\)=∑imi​vi∥m∥p,p≥1\.A\_\{p\}\(X\)=\\frac\{\\sum\_\{i\}m\_\{i\}v\_\{i\}\}\{\\lVert m\\rVert\_\{p\}\},\\qquad p\\geq 1\.\(3\)Standard attention is the special casep=1p=1\. The effect ofMAAbecomes clearer using normalized attentionα=m/∥m∥1\\alpha=m/\\lVert m\\rVert\_\{1\}\. Rearranging \([3](https://arxiv.org/html/2607.22781#S3.E3)\) gives

Ap​\(X\)=1∥α∥p⏟gp​\(α\)​A1​\(X\)\.A\_\{p\}\(X\)=\\underbrace\{\\frac\{1\}\{\\lVert\\alpha\\rVert\_\{p\}\}\}\_\{g\_\{p\}\(\\alpha\)\}A\_\{1\}\(X\)\.\(4\)Equation \([4](https://arxiv.org/html/2607.22781#S3.E4)\) shows thatMAApreserves the standard weighted directionA1​\(X\)A\_\{1\}\(X\)and retains only an input\-dependent scalegp​\(α\)g\_\{p\}\(\\alpha\)\. This scale is computed from the normalized attention distribution itself rather than from an injected token count\. Forp\>1p\>1, it satisfies

1≤gp​\(α\)≤n1−1/p\.1\\leq g\_\{p\}\(\\alpha\)\\leq n^\{1\-1/p\}\.\(5\)The scale approaches one when attention concentrates on a few inputs and grows as attention spreads over more inputs\. In terms of Rényi entropyHp​\(α\)H\_\{p\}\(\\alpha\), it can equivalently be written asgp​\(α\)=exp⁡\(\(1−1/p\)​Hp​\(α\)\)g\_\{p\}\(\\alpha\)=\\exp\(\(1\-1/p\)H\_\{p\}\(\\alpha\)\)\. Thus,gp​\(α\)g\_\{p\}\(\\alpha\)is a monotonic transformation of the attention\-weighted effective support\.

This generalization separates the growth rates of the numerator and denominator under repetition\. ForX\(r\)X^\{\(r\)\}, the numerator grows byrr, while theLpL\_\{p\}denominator grows byr1/pr^\{1/p\}\. The resulting response is

Ap​\(X\(r\)\)=r1−1/p​Ap​\(X\)\.A\_\{p\}\(X^\{\(r\)\}\)=r^\{1\-1/p\}A\_\{p\}\(X\)\.\(6\)Atp=1p=1, the two rates cancel exactly and recover \([2](https://arxiv.org/html/2607.22781#S3.E2)\)\. Forp\>1p\>1, the effective size of the replicated input remains as a controlled amplitude\. Importantly, this is not the arbitrary absolute mass of unnormalized scores\.MAAconveys a reproducible scale derived from the concentration and effective support of normalized attention\.

### 3\.3Implementation

MAAchanges only the computation of the normalization denominator in an existing attention pipeline\. Algorithm[1](https://arxiv.org/html/2607.22781#alg1)summarizes the operation for one query and one attention head\.

Algorithm 1MAAat One Attention Boundary1:logits

aa, values

VV, valid mask

MM, power

p≥1p\\geq 1
2:

s←maxi:Mi=1⁡ais\\leftarrow\\max\_\{i:M\_\{i\}=1\}a\_\{i\}
3:

mi←Mi​exp⁡\(ai−s\)m\_\{i\}\\leftarrow M\_\{i\}\\exp\(a\_\{i\}\-s\)
4:

dp←\(∑imip\)1/pd\_\{p\}\\leftarrow\(\\sum\_\{i\}m\_\{i\}^\{p\}\)^\{1/p\}
5:

βi←mi/max⁡\(dp,ϵ\)\\beta\_\{i\}\\leftarrow m\_\{i\}/\\max\(d\_\{p\},\\epsilon\)
6:return

∑iβi​Vi\\sum\_\{i\}\\beta\_\{i\}V\_\{i\}

The operation is applied independently to each query and head, with masked tokens excluded from the norm\. Logits, value projections, residual connections, and decoders remain unchanged, so neither parameter count nor hidden dimensionality increases\. Time and space complexity remain of the same order as standard attention\. This minimal change enables the same principle to be applied across domains, although a strong scale\-removing normalization after \([4](https://arxiv.org/html/2607.22781#S3.E4)\) can erase the signal\.

## 4Evaluation Principles and Protocol

The evaluation consistently separates information accessibility in the same representation, original\-task utility, and power selection\.

### 4\.1Representation Recovery

After training, we freeze the hidden representationZZsupplied to the predictor\. Probes are trained on a chronological held\-out split; neither recovery targets nor probe gradients participate in training the original model\. For CTDG, we use both linear Ridge and ExtraTrees\. The linear probe measures linear accessibility from the sameZZ, whereas ExtraTrees measures nonlinear accessibility\. Unless otherwise stated, the primary cross\-domain recovery measure uses a linear probe\. Continuous targets are evaluated with held\-outR2R^\{2\}and binary conditions with AUC\.

Shared recoverydirectly compares the equal\-dimensional hidden representationsZBZ\_\{B\}andZMAAZ\_\{\\textsc\{MAA\}\}of the baseline andMAA\.Beyond\-posthoc recoverysupplies both probes with the same type of auxiliary coordinatecccomputed during attention and measures the remaining difference\. The former tests whether the final representation itself changes; the latter tests whether the learned representation changes beyond simply exposing an already computed coordinate\.Context marginis the difference in prediction between the original query context and a condition with a substituted source context\. A larger margin indicates greater sensitivity of model output to contextual differences\.

### 4\.2Separating Power Selection from Evaluation

Becauseppdefines the operator family, we select it through a constrained model\-selection problem aligned with the purpose of MAA\. The primary objective is information accessibility in the predictor\-facing representation, while preservation of original\-task utility is a feasibility constraint\. The two outcomes are therefore not combined post hoc with equal status\. We use a pre\-specified lexicographic order: the selection split first identifies powers that preserve baseline task utility, and the pre\-specified recovery endpoint selects a power within this feasible set\. We then freezeppand evaluate final task performance and recovery once on held\-out conditions that did not participate in selection\.

For CTDG, we instantiate this rule through leave\-one\-model\-out selection\. Cells from three models selectpp, and the excluded model provides evaluation; all four folds selectp=1\.1p=1\.1\. Consequently, none of the 12 reported CTDG cells uses a power selected from results of its own model\. RAG applies the same task\-preserving recovery rule on an independent validation split and reports results on a separate test split\. MTPP fixesp=1\.3p=1\.3in an earlier pilot and evaluates it on new likelihood and time endpoints\. TKG separates an ICEWS14 pilot from ICEWS18 validation, fixingp=1\.2p=1\.2andp=1\.02p=1\.02, respectively, before test evaluation\. STPP evaluatesp=1\.05p=1\.05in a matched pre\-norm architecture after diagnosing the post\-norm boundary\.

#### Reusable decision rule\.

Algorithm[2](https://arxiv.org/html/2607.22781#alg2)operationalizes the procedure for a new model or domain\. The audit target, candidate grid, and task\-utility tolerance are fixed before opening the test set\. We use a small grid dense near the standard operator,𝒫=\{1,1\.02,1\.05,1\.1,1\.2,1\.3\}\\mathcal\{P\}=\\\{1,1\.02,1\.05,1\.1,1\.2,1\.3\\\}\. LetUsel​\(p\)U\_\{\\mathrm\{sel\}\}\(p\)denote the original\-task metric oriented so that larger is better, and letRsel​\(p\)R\_\{\\mathrm\{sel\}\}\(p\)denote recovery from the same predictor\-facingZZ\. In CTDG, a candidate is feasible only when its mean selection\-fold AUC is at least that of standard attention \(p=1p=1\)\. In RAG, feasibility analogously requires validation diagnostic AUC at least as high as thep=1p=1baseline\. Both studies therefore useϵU=0\\epsilon\_\{U\}=0and select the power with the highest pre\-specified recovery endpoint among feasible candidates\. Future applications may instead pre\-specify a domain\-accepted practical\-equivalence margin\.

Algorithm 2Validation\-Based Power Selection forMAA1:candidate grid

𝒫\\mathcal\{P\}, selection split, fixed audit target, task tolerance

ϵU\\epsilon\_\{U\}
2:Evaluate

Usel​\(p\)U\_\{\\mathrm\{sel\}\}\(p\)and

Rsel​\(p\)R\_\{\\mathrm\{sel\}\}\(p\)for every

p∈𝒫p\\in\\mathcal\{P\}
3:

ℱ←\{p:Usel​\(p\)≥Usel​\(1\)−ϵU\}\\mathcal\{F\}\\leftarrow\\\{p:U\_\{\\mathrm\{sel\}\}\(p\)\\geq U\_\{\\mathrm\{sel\}\}\(1\)\-\\epsilon\_\{U\}\\\}
4:if

ℱ=∅\\mathcal\{F\}=\\varnothingthen

5:return

p⋆=1p^\{\\star\}=1
6:endif

7:

p⋆←arg⁡maxp∈ℱ⁡Rsel​\(p\)p^\{\\star\}\\leftarrow\\arg\\max\_\{p\\in\\mathcal\{F\}\}R\_\{\\mathrm\{sel\}\}\(p\)
8:Break recovery ties toward the smaller

pp
9:returnfrozen

p⋆p^\{\\star\}; evaluate the held\-out test once

This lexicographic rule gives the two design goals explicit priority: the task constraint protects predictive utility, and the recovery objective selects the more informative representation within that constraint\. Candidate comparison, feasibility filtering, and tie breaking are completed entirely on the selection split, making held\-out recovery and task utility outcomes of evaluation rather than inputs to selection\.

This tunability is a design advantage of MAA\. Standard attention fixes exactL1L\_\{1\}normalization, and hencep=1p=1, for every input and task\. MAA exposes this implicit design decision as an explicit hyperparameter, allowing practitioners to calibrate the strength of repetition invariance through standard validation without observing test outcomes\. Because input length, attention concentration, and downstream normalization alter the distribution ofgp​\(α\)g\_\{p\}\(\\alpha\)induced by the samepp, an explicit and reproducible selection rule is more appropriate than a universal constant\. When no meaningful audit target is available,p=1\.1p=1\.1, the common CTDG value, can serve as a default, subject to the same validation check on task utility\.

### 4\.3Statistical Inference

Seeds within the same model–dataset pair are repeated measurements of one condition, not independent research settings\. Confirmatory CTDG inference therefore first averages the three seed\-level effects within each of the 12 model–dataset cells\. We apply two\-sided Wilcoxon signed\-rank tests and jointly adjust the six aggregate endpoints with Holm correction\. We additionally report 95% intervals from 20,000 crossed hierarchical bootstrap iterations that resample models and datasets independently and then resample seeds within each selected cell\. The 36 seed\-level results serve as a sensitivity analysis of direction and variability\.

The cross\-domain extension is exploratory, and all paired Wilcoxon tests are therefore two\-sided\. In small samples, conclusions jointly consider effect size, the number of improving repetitions, and original\-task performance rather than relying on significance alone\.

## 5Core Validation on CTDG

CTDG serves as the core validation domain because prior work has already identified a concrete gap between benchmark performance and the graph mechanisms captured by temporal models\. Dynamic link\-prediction results can depend strongly on negative sampling and recurring edges\[[18](https://arxiv.org/html/2607.22781#bib.bib6)\], and recent evidence shows that model predictions do not consistently reflect several temporal graph characteristics\[[11](https://arxiv.org/html/2607.22781#bib.bib9)\]\. We extend this behavioral question to recovery from predictor\-facing representations and conduct our broadest comparison and confirmatory inference in CTDG, using four models and three datasets\.

### 5\.1Setup and Representation Gap

TABLE II:Experimental setup for the core CTDG study\.ComponentSettingModelsCRAFT, TGAT, DyGFormer, GraphMixerDatasetsLastFM, MOOC, WikipediaRepetitionsSeeds 7, 17, and 29; 36 paired runsTraining dataMost recent 32,768 events in chronological order; 70/30 splitObjectiveOriginal future\-link BPR lossMethodBaselinep=1p=1; fixed\-power MAAp=1\.1p=1\.1ProbesChronological held\-out Ridge and ExtraTreesTask metricROC–AUC over positives and sampled negativesInference unit12 model–dataset cells after within\-cell seed averaging
We evaluate CRAFT, TGAT, DyGFormer, and GraphMixer on LastFM, MOOC, and Wikipedia\. These temporal interaction benchmarks were released with JODIE\[[15](https://arxiv.org/html/2607.22781#bib.bib7)\]\. Every model uses the same recent\-event budget and its original future\-link objective\. The eight recovery targets are pair frequency, recency, persistence, periodicity, granularity, density, homophily, and preferential attachment \(PA\)\. PA audits the count\-structured information left by repeated edge formation at the node level because it reflects accumulated endpoint degrees\[[4](https://arxiv.org/html/2607.22781#bib.bib10)\]\. Pair frequency directly counts repetitions at the pair level, whereas PA measures degree accumulation at the node level\.

Baseline recovery varies substantially across models and statistics\. CRAFT is relatively strong on PA and pair frequency, while TGAT recovers density and granularity more accurately\. This analysis motivates the central observation that high AUC does not determine the graph\-statistic content of a predictor\-facing representation\.

### 5\.22×22\\times 2Representation Audit

![Refer to caption](https://arxiv.org/html/2607.22781v1/x3.png)Figure 3:2×22\\times 2audit separating changes in the hidden representationZZlearned byMAAfrom gains obtained by exposing a normalization\-derived coordinate to the probe\. The shared comparison directly contrastsZBZ\_\{B\}andZMAAZ\_\{\\textsc\{MAA\}\}, while the post\-hoc comparison supplies the same type of coordinateccto both sides\.
The four conditions in Fig\.[5\.2](https://arxiv.org/html/2607.22781#S5.SS2)are the baseline representationZBZ\_\{B\}, its post\-hoc augmentation\(ZB,cB\)\(Z\_\{B\},c\_\{B\}\), the equal\-dimensionalMAArepresentationZMAAZ\_\{\\textsc\{MAA\}\}, and\(ZMAA,cMAA\)\(Z\_\{\\textsc\{MAA\}\},c\_\{\\textsc\{MAA\}\}\)\. The primary comparison isZMAAZ\_\{\\textsc\{MAA\}\}versusZBZ\_\{B\}, which rules out gains caused by providing a new feature to the probe\. The beyond\-posthoc comparison tests whether a difference remains after exposing an equivalent coordinate on both sides\.

### 5\.3Aggregate Results

TABLE III:Aggregate CTDG results\. Wilcoxon tests use 12 paired effects obtained by first averaging three seeds within each model–dataset cell\. Holm correction is applied jointly to the six aggregate endpoints\.MeasureBaselineMAAMean changeImproved cellsppHolmppAUC0\.859850\.86336\+0\.00350 \(\+0\.41%\)11/120\.0009770\.005859Shared ExtraTreesR2R^\{2\}0\.525260\.52628\+0\.00102 \(\+0\.19%\)4/120\.5185550\.518555Beyond ExtraTreesR2R^\{2\}0\.559280\.55761\-0\.00168 \(\-0\.30%\)5/120\.2333980\.466797Shared linearR2R^\{2\}0\.400200\.41818\+0\.01799 \(\+4\.49%\)9/120\.0341800\.136719Beyond linearR2R^\{2\}0\.442780\.45298\+0\.01020 \(\+2\.30%\)9/120\.0522460\.156738Context margin0\.541110\.57261\+0\.03150 \(\+5\.82%\)9/120\.0268550\.134277

Table[5\.3](https://arxiv.org/html/2607.22781#S5.SS3)shows that AUC rises by 0\.41% on average and improves in 11 of 12 cells\. The 12\-cell Wilcoxon test, Holm correction, and crossed\-bootstrap interval agree on the direction\. Shared linear recovery increases by 4\.49% and context margin by 5\.82%\. Both effects are positive in 9/12 cells, although uncertainty remains after aggregate family\-wise correction\. Mean ExtraTrees recovery is nearly unchanged\. The clearest representational effect ofMAAis therefore improved linear accessibility from the sameZZ, rather than a universal increase in nonlinear recoverability\.

TABLE IV:Key statistic\-wise changes in CTDG\. Holm correction over the eight statistics is performed separately for each probe\.ProbeStatisticΔ​R2\\Delta R^\{2\}CellHolmppLinearPA\+0\.0910812/120\.003906LinearPair frequency\+0\.013928/120\.187988LinearDensity\+0\.037549/120\.462891ExtraTreesDensity\-0\.021702/120\.039062
Statistic\-level analysis sharpens this result\. Linear PA recovery increases by0\.091080\.09108on average, improves in all 12 cells, and survives Holm correction across the eight statistics \(Table[5\.3](https://arxiv.org/html/2607.22781#S5.SS3)\)\. Pair frequency has a positive mean effect but remains uncertain after correction, whereas ExtraTrees density decreases significantly\. Figure[5\.3](https://arxiv.org/html/2607.22781#S5.SS3)shows both statistic\-level uncertainty and model heterogeneity\. The most stable gains occur along linear directions tied directly to repetition and degree accumulation, while some nonlinear structure is redistributed\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x4.png)Figure 4:Statistic\-wise changes in recovery and their boundary\. Linear recovery improves most strongly and consistently for preferential attachment, with pair frequency changing in the same direction\. The decrease in ExtraTrees density shows that accessibility varies across probes and statistics\.

### 5\.4Behavioral Sensitivity

Context margin measures the score difference between the observed source temporal neighborhood and a substituted neighborhood\. Its average increase of 5\.82% shows that the additional probe\-accessible information coincides with greater sensitivity of predictive behavior to contextual differences\. Because the margin alone does not establish directionally correct causal use, we interpret it together with original\-task AUC and statistic interventions\.

## 6Cross\-Domain Generalization

We extend the CTDG question to four additional domains\. For each domain, we first define a count\- or support\-based statistic thatMAAis expected to retain, and then separately measure its recovery from the sameZZ, the model’s behavioral response to evidence manipulation, and utility on the original task\. Figure[5](https://arxiv.org/html/2607.22781#S6.F5)and Table[6](https://arxiv.org/html/2607.22781#S6)place the results from all domains in a common framework\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x5.png)Figure 5:Changes in predictor\-facing representation recovery across domains\. Each value is an equally weighted average over the model–dataset conditions evaluated in that domain\. Targets and metrics follow the predictive structure of each domain; the common question is whether evidence\-accumulation information becomes more accessible from the sameZZ, without auxiliary coordinates\.TABLE V:Key cross\-domain results\. Recovery is the primary endpoint, and task metrics jointly report predictive utility\.DomainRepresentation improvementTask utilityCTDGLinearR2R^\{2\}\+4\.49%; PA12/12Link AUC\+0\.41%MTPPMark\-countR2R^\{2\}\+25\.05%Point\-process NLL2\.59% lowerTKGPair recurrence\+9\.09%Hits@10\+1\.07%RAGRetained\-passage AUC\+35\.03%Diagnostic AUC \-0\.73% \(n\.s\.\)STPPLocal\-countR2R^\{2\}\+9\.46%Matched NELBO4\.46% better
### 6\.1MTPP: Accumulated Occurrences of a Candidate Mark

AttNHP and THP encode event sequences with self\-attention to predict the mark and time of the next event\[[23](https://arxiv.org/html/2607.22781#bib.bib17),[30](https://arxiv.org/html/2607.22781#bib.bib16)\]\. The candidate mark of the next event is conditionally related to the occurrence pattern of the same mark in preceding events\. We use candidate\-mark counts over the full past and the most recent 32 and 8 events as recovery targets\. These quantities directly summarize inputs to the mark\-specific conditional intensity\.MAAimproves full\-history countR2R^\{2\}in all six aggregated model–dataset–seed conditions, yielding a mean improvement of 25\.05%\. Concurrently, next\-mark NLL decreases by 1\.29% and point\-process NLL by 2\.59%\. For AttNHP, next\-time MAE and RMSE also decrease by 2\.14% and 3\.96%, respectively\. The joint improvement in information accessibility and likelihood indicates that the information attenuated by standard weighted averaging has predictive value for this task\.

### 6\.2TKG: Query\-Specific Pair Recurrence

xERTE expands a query\-relevant temporal subgraph through attention to rank future entities\[[10](https://arxiv.org/html/2607.22781#bib.bib19)\]; this architecture differs from RE\-NET, which uses a recurrent event encoder\[[14](https://arxiv.org/html/2607.22781#bib.bib18)\]\. For a TKG query\(h​e​a​d,r​e​l​a​t​i​o​n,?,t​i​m​e\)\(head,relation,?,time\), the number of previous connections between the same head and candidate tail provides more query\-specific evidence than global tail popularity\. On ICEWS14 and ICEWS18, we recover pair recurrence, the number of relation types observed for the same pair, and the number of historical two\-hop paths\. The stand\-alone diagnostic AUC of pair recurrence exceeds that of global tail count on both datasets, establishing its relevance to the ranking problem\.

TABLE VI:Representative MTPP and TKG results\. A downward arrow marks a lower\-is\-better metric\.DomainSettingEndpointp=1p=1MAARelative changeImproved runsMTPPAll 6 settingsCandidate\-mark countR2R^\{2\}0\.107920\.13495\+25\.05%6/6Point\-process NLL↓\\downarrow2\.967492\.89063\-2\.59%5/6Next\-mark NLL↓\\downarrow1\.922801\.89791\-1\.29%6/6TKGICEWS14Pair\-recurrenceR2R^\{2\}0\.221490\.24773\+11\.85%3/3Hits@100\.683000\.69617\+1\.93%3/3ICEWS18Pair\-recurrenceR2R^\{2\}0\.130710\.13650\+4\.43%3/3MRR0\.228070\.22760\-0\.20%1/3

As shown in Table[6\.2](https://arxiv.org/html/2607.22781#S6.SS2), pair\-recurrence recovery increases by 11\.85% on ICEWS14 and by 4\.43% in the deterministic ICEWS18 rerun, with the same direction in all three seeds for both datasets\. Hits@10 improves by 1\.93% on ICEWS14\. On ICEWS18, MRR decreases by 0\.20% and Hits@10 by 0\.43%, preserving nearly equivalent ranking performance\. These results show thatMAAcan retain query\-specific recurrence more clearly while leaving the ranking function largely intact, even when every task metric does not improve simultaneously\.

### 6\.3RAG: Separating Accessibility from Utilization

In multi\-hop question answering tasks such as HotpotQA, success requires combining facts distributed across distinct supporting passages rather than merely repeating one passage\[[24](https://arxiv.org/html/2607.22781#bib.bib21)\]\. FiD encodes retrieved passages independently and combines them through decoder attention\[[13](https://arxiv.org/html/2607.22781#bib.bib22)\]\. We therefore define a retained\-passage condition that distinguishes complete evidence from inputs missing part of the supporting passages\. Our primary analysis masks duplicate\-token and duplicate\-passage indicators to prevent literal duplication from becoming a count shortcut\. Recovery AUC measures how well this binary condition can be separated from the sameZZ, whereas task AUC measures whether the original diagnostic head assigns higher scores to complete evidence\.

TABLE VII:Boundary results for RAG and STPP\. Arrows denote changes from baseline to MAA\.DomainEndpointBaseline→\\rightarrowMAAChangeRAGRetained\-passage recovery AUC0\.585→0\.7900\.585\\rightarrow 0\.790\+35\.03%Diagnostic task AUC0\.614→0\.6090\.614\\rightarrow 0\.609\-0\.73%STPPPredictor\-facing countR2R^\{2\}0\.324→0\.3550\.324\\rightarrow 0\.355\+9\.46%Encoder countR2R^\{2\}0\.452→0\.4930\.452\\rightarrow 0\.493\+9\.18%NELBO↓\\downarrow−1\.675→−1\.750\-1\.675\\rightarrow\-1\.7504\.46% better
MAAraises duplicate\-masked retained\-passage recovery AUC from 0\.58482 to 0\.78967 \(35\.03%, 5/5\)\. Diagnostic task AUC decreases by 0\.73%, with a two\-sidedp=1\.0p=1\.0\. As Fig\.[6\.3](https://arxiv.org/html/2607.22781#S6.SS3)illustrates, the presence of complete supporting evidence becomes substantially clearer in the representation, while the current task head does not automatically exploit that direction\. This separation delineates the role ofMAA—retaining information—from the role of the decoder and objective—using that information through an appropriate decision rule\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x6.png)Figure 6:Relationship between representation accessibility and task utility\. Recovery and representative task metrics improve together in CTDG and MTPP\. In TKG, ICEWS14 Hits@10 improves while ICEWS18 ranking remains nearly unchanged\. In RAG, the retained\-passage condition becomes more recoverable without a corresponding improvement in the diagnostic head’s AUC\.

### 6\.4STPP: The Boundary Imposed by Downstream Normalization

DeepSTPP constructs a nonparametric spatiotemporal intensity through a latent process\[[28](https://arxiv.org/html/2607.22781#bib.bib24)\], whereas Neural STPP conditions on preceding events through an attentive continuous\-time flow\[[5](https://arxiv.org/html/2607.22781#bib.bib23)\]\. For DeepSTPP, we use the numbers of preceding events within radii 0\.25, 0\.5, and 1\.0 of a candidate location as local\-count targets\. In the original architecture, recovery improves at the final attention output, but the effect disappears immediately after LayerNorm and becomes negative at the predictor\-facingZZ\. This trace shows thatMAAcreates the signal at the attention boundary and that a subsequent amplitude\-removing operation interrupts its propagation\.

In a matched pre\-norm architecture that places LayerNorm before attention\[[21](https://arxiv.org/html/2607.22781#bib.bib26)\], predictor\-facing recovery increases by 9\.46% and NELBO improves by 4\.46%\. Its absolute NELBO, however, remains below that of the original DeepSTPP architecture\. Figure[7](https://arxiv.org/html/2607.22781#S6.F7)summarizes this boundary\.MAAretains the effective\-support scale at the attention boundary; propagation to the final representation depends on whether subsequent architectural operations preserve that scale\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x7.png)Figure 7:Cross\-domain boundaries\. In RAG, increased information accessibility is separated from task utilization\. In STPP, post\-attention LayerNorm removes the scale created at the attention output; moving LayerNorm before attention in a matched architecture improves both predictor\-facing recovery and NELBO\.

## 7Mechanism Analysis and Alternative Explanations

This section tests whether the observed improvements are attributable to a training schedule, uniform norm expansion, or explicit cardinality injection\.

### 7\.1Power or Schedule?

TABLE VIII:Fixed\-power sensitivity\. Values are averaged over four representative conditions and characterize the effect of mildly relaxing exact repetition invariance atp=1p=1\.ppAUCΔ\\DeltaAUCShared linearR2R^\{2\}Δ​R2\\Delta R^\{2\}Context marginΔ\\Deltamargin1\.000\.84559–0\.38277–0\.48446–1\.020\.84809\+0\.30%0\.40403\+5\.56%0\.51870\+7\.07%1\.050\.84891\+0\.39%0\.40626\+6\.14%0\.52151\+7\.65%1\.100\.84935\+0\.45%0\.40891\+6\.83%0\.52465\+8\.29%1\.200\.84992\+0\.51%0\.41288\+7\.87%0\.53985\+11\.43%2\.000\.84553\-0\.01%0\.40940\+6\.96%0\.60774\+25\.45%

Our initial hypothesis assigned importance to a schedule that began with a largeppand strengthened invariance later in training\. The differences among fixedp=1\.1p=1\.1, forward2→1\.12\\\!\\rightarrow\\\!1\.1, and reverse1\.1→21\.1\\\!\\rightarrow\\\!2are small\. By contrast, the range 1\.02–1\.2, which mildly departs fromp=1p=1, consistently improves AUC, shared linear recovery, and context margin in Table[7\.1](https://arxiv.org/html/2607.22781#S7.SS1)\. Figure[8](https://arxiv.org/html/2607.22781#S7.F8)shows both the mean changes and variation across conditions\. Atp=2p=2, context margin increases substantially while the AUC gain disappears\. The principal effect therefore arises from mildly relaxing exactL1L\_\{1\}repetition invariance rather than from the order of training\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x8.png)Figure 8:Fixed\-power sensitivity\. Mild normalization relaxation improves recovery and context margin while maintaining or improving AUC\. Largerppfurther increases sensitivity but can weaken the balance with task utility\.
### 7\.2Uniform Scaling or an Input\-Dependent Effect?

Equation \([4](https://arxiv.org/html/2607.22781#S3.E4)\) shows thatMAAmultiplies the standard weighted direction by an input\-dependent amplitude\. To test the alternative explanation that a larger representation norm merely facilitates optimization, we use a matched control thatL2L\_\{2\}\-normalizes the aggregation output\. Removing the norm attenuates but does not eliminate the gains in AUC, shared linear recovery, and beyond\-posthoc recovery\. Figure[9](https://arxiv.org/html/2607.22781#S7.F9)indicates that amplitude is a principal transmission path and that training also induces some directional change in the representation\. The relevant mechanism is therefore the input\-varyinggp​\(α\)g\_\{p\}\(\\alpha\)and its associated optimization trajectory, rather than a uniform scale factor\.

![Refer to caption](https://arxiv.org/html/2607.22781v1/x9.png)Figure 9:Mechanism validation\. Removing theL2L\_\{2\}norm of theMAAoutput attenuates, but does not eliminate, the effect\. Input\-dependent amplitude is a principal signal path, accompanied by a smaller change in the learned hidden direction\.
### 7\.3Comparison with Explicit Cardinality

TABLE IX:Matched comparison of MAA with CPA, an explicit cardinality control\. Each cell reports a change from baseline; for CPA, the better result between its additive and scaled variants is shown for each endpoint\.Representative settingEndpointBest CPAΔ\\DeltaMAAΔ\\DeltaMAA–CPACRAFT–LastFMAUC\-0\.00258\+0\.00293\+0\.00550Shared linearR2R^\{2\}\-0\.00684\+0\.01527\+0\.02211TGAT–MOOCAUC\-0\.01582\+0\.00159\+0\.01741Shared linearR2R^\{2\}\+0\.12934\+0\.03166\-0\.09768AttNHP–StackOverflowNLL reduction\-0\.02067\+0\.04456\+0\.06523CountR2R^\{2\}\-0\.02380\+0\.00213\+0\.02594THP–StackOverflowNLL reduction\+0\.24629\+0\.18438\-0\.06190CountR2R^\{2\}\+0\.08140\+0\.08650\+0\.00511xERTE–ICEWS14MRR\-0\.00216\-0\.00341\-0\.00125Pair\-recurrenceR2R^\{2\}\-0\.06946\+0\.00128\+0\.07074xERTE–ICEWS18MRR\-0\.01280\-0\.00047\+0\.01233Pair\-recurrenceR2R^\{2\}\-0\.07276\+0\.00579\+0\.07855

CPA directly injects the number or unweighted sum of inputs and can substantially improve recovery or task performance in particular conditions, as observed for TGAT–MOOC recovery and THP NLL\. The same intervention, however, strongly perturbs performance or recovery for CRAFT, AttNHP, and xERTE\.MAAinstead reflects the effective support of the learned attention distribution without a separate count, producing a more stable representation–task trade\-off\. In Table[7\.3](https://arxiv.org/html/2607.22781#S7.SS3),MAAoutperforms the best CPA variant in recovery for five of six representative cases and in task utility for four of six\. This comparison empirically distinguishes the retention of attention\-weighted support from the direct reinjection of counts\.

### 7\.4What Is Learned, and Where Is It Useful?

The information made more accessible byMAAhas domain\-specific names but a common structure: the extent of evidence accumulated across multiple inputs\. In CTDG, PA and pair frequency describe pair\- and node\-level accumulation induced by repeated edge formation\. Candidate\-mark count in MTPP is a direct input to the intensity of the next event, and pair recurrence in TKG provides query\-specific support for a candidate tail\. The retained\-passage condition in RAG indicates whether all distinct evidence required for an answer is present, while local count in STPP represents conditional event density around a candidate location\.

This information supports three uses\. First, when the original task head exploits it, the improvement appears in metrics such as AUC, NLL, and ranking quality\. Second, linear accessibility from the sameZZenables model auditing, transfer heads, calibration, and downstream decisions even when the original head does not immediately convert it into task performance\. Third, tracing where accessibility disappears reveals architectural information bottlenecks\. RAG most clearly separates accessibility from direct utilization, while STPP exposes the downstream normalization boundary\.

### 7\.5Limitations

MAAis a general principle for improving representation informativeness rather than a universal task\-performance optimizer\. Information reaches the predictor\-facingZZwhen the path following attention preserves amplitude, and the task head converts it into utility when its objective is aligned with that information\. Recovery targets likewise follow the semantically meaningful evidence unit of each domain\. Because the cross\-domain samples are smaller than the CTDG study, we interpret them through effect sizes, replication counts, and task performance together\. Broader validation across architectures and large\-scale real\-world tasks remains valuable\. Although power selection is separated from evaluation, the validation protocols are not identical across domains; a transferable selection rule based on the distribution ofgp​\(α\)g\_\{p\}\(\\alpha\)is a promising direction\.

## 8Conclusion

Standard attention is a weighted average and can therefore map repetitions of the same evidence pattern to an identical aggregate\. We introduced Mass\-Aware Attention, which turns this implicitL1L\_\{1\}repetition invariance into an explicit control axis within anLpL\_\{p\}family\. The method changes a single denominator and adds no supervision, count feature, or parameter\.

In CTDG,MAAincreases the linear accessibility of graph statistics from the same hidden representation, improving preferential\-attachment recovery in all 12 model–dataset cells\. Future\-link AUC also increases in 11 of 12 cells\. Candidate\-mark counts and pair recurrence become more recoverable in MTPP and TKG, while RAG and STPP reveal how information retention interacts with task heads and downstream normalization\.

Together, these results connect representation gaps observed under different names to a shared blind spot of weighted averaging and show that a minimal normalization principle can make accumulated evidence more accessible\.MAAadvances attention design beyond judging learning solely through predictive performance by making the evidence transmitted to the final representation an explicit design consideration\.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper\.

## Acknowledgements

The authors have no acknowledgements to declare\.

## Funding

This research received no specific grant from any funding agency in the public, commercial, or not\-for\-profit sectors\.

## Data availability

All datasets used in this study are publicly available\. The exact data sources are listed in Table[Data availability](https://arxiv.org/html/2607.22781#Sx4)\.

## Code availability

## References

- \[1\]\(2017\)Understanding intermediate layers using linear classifier probes\.External Links:[Link](https://openreview.net/forum?id=ryF7rTqgl)Cited by:[§2\.2](https://arxiv.org/html/2607.22781#S2.SS2.p1.1)\.
- \[2\]J\. L\. Ba, J\. R\. Kiros, and G\. E\. Hinton\(2016\)Layer normalization\.arXiv preprint arXiv:1607\.06450\.Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p2.1)\.
- \[3\]J\. Bae, B\. Park, G\. Park, M\. Kim, J\. Lee, J\. Yoo, S\. Woo, J\. Ryu, S\. J\. Kwon, and D\. Lee\(2026\)Affine\-scaled attention: towards flexible and stable transformer attention\.arXiv preprint arXiv:2602\.23057\.Cited by:[§2\.3](https://arxiv.org/html/2607.22781#S2.SS3.p2.3)\.
- \[4\]A\. Barabási and R\. Albert\(1999\)Emergence of scaling in random networks\.Science286\(5439\),pp\. 509–512\.External Links:[Document](https://dx.doi.org/10.1126/science.286.5439.509)Cited by:[§5\.1](https://arxiv.org/html/2607.22781#S5.SS1.p2.1)\.
- \[5\]R\. T\. Q\. Chen, B\. Amos, and M\. Nickel\(2021\)Neural spatio\-temporal point processes\.InInternational Conference on Learning Representations,Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.4](https://arxiv.org/html/2607.22781#S6.SS4.p1.1)\.
- \[6\]W\. Cong, S\. Zhang, J\. Kang, B\. Yuan, H\. Wu, X\. Zhou, H\. Tong, and M\. Mahdavi\(2023\)Do we really need complicated model architectures for temporal networks?\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p1.1)\.
- \[7\]A\. Conneau, G\. Kruszewski, G\. Lample, L\. Barrault, and M\. Baroni\(2018\)What you can cram into a single $&\!\#\* vector: probing sentence embeddings for linguistic properties\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,pp\. 2126–2136\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1198)Cited by:[§2\.2](https://arxiv.org/html/2607.22781#S2.SS2.p1.1)\.
- \[8\]F\. Cornell, O\. Smirnov, G\. Z\. Gandler, and L\. Cao\(2025\)On the power of heuristics in temporal graphs\.InProceedings on “I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning” at ICLR 2025 Workshops,Proceedings of Machine Learning Research, Vol\.296,pp\. 37–46\.Cited by:[§1](https://arxiv.org/html/2607.22781#S1.p1.1)\.
- \[9\]N\. Du, H\. Dai, R\. Trivedi, U\. Upadhyay, M\. Gomez\-Rodriguez, and L\. Song\(2016\)Recurrent marked temporal point processes: embedding event history to vector\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1555–1564\.External Links:[Document](https://dx.doi.org/10.1145/2939672.2939875)Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1)\.
- \[10\]Z\. Han, P\. Chen, Y\. Ma, and V\. Tresp\(2021\)Explainable subgraph reasoning for forecasting on temporal knowledge graphs\.InInternational Conference on Learning Representations,Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.2](https://arxiv.org/html/2607.22781#S6.SS2.p1.1)\.
- \[11\]A\. J\. Hayes, T\. Schumacher, and M\. Strohmaier\(2025\)What do temporal graph learning models learn?\.arXiv preprint arXiv:2510\.09416\.Cited by:[§1](https://arxiv.org/html/2607.22781#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.22781#S5.p1.1)\.
- \[12\]J\. Hewitt and P\. Liang\(2019\)Designing and interpreting probes with control tasks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2733–2743\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1275)Cited by:[§2\.2](https://arxiv.org/html/2607.22781#S2.SS2.p1.1)\.
- \[13\]G\. Izacard and E\. Grave\(2021\)Leveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 874–880\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.74)Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.3](https://arxiv.org/html/2607.22781#S6.SS3.p1.1)\.
- \[14\]W\. Jin, M\. Qu, X\. Jin, and X\. Ren\(2020\)Recurrent event network: autoregressive structure inference over temporal knowledge graphs\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 6669–6683\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.541)Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.2](https://arxiv.org/html/2607.22781#S6.SS2.p1.1)\.
- \[15\]S\. Kumar, X\. Zhang, and J\. Leskovec\(2019\)Predicting dynamic embedding trajectory in temporal interaction networks\.InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 1269–1278\.External Links:[Document](https://dx.doi.org/10.1145/3292500.3330895)Cited by:[§5\.1](https://arxiv.org/html/2607.22781#S5.SS1.p2.1)\.
- \[16\]J\. Kunz and M\. Kuhlmann\(2020\)Classifier probes may just learn from linear context features\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 5136–5146\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.450)Cited by:[§2\.2](https://arxiv.org/html/2607.22781#S2.SS2.p1.1)\.
- \[17\]H\. Mei and J\. Eisner\(2017\)The neural hawkes process: a neurally self\-modulating multivariate point process\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1)\.
- \[18\]F\. Poursafaei, S\. Huang, K\. Pelrine, and R\. Rabbany\(2022\)Towards better evaluation for dynamic link prediction\.InAdvances in Neural Information Processing Systems,Vol\.35\.Note:Datasets and Benchmarks TrackCited by:[§1](https://arxiv.org/html/2607.22781#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.22781#S5.p1.1)\.
- \[19\]O\. Richter and R\. Wattenhofer\(2020\)Normalized attention without probability cage\.arXiv preprint arXiv:2005\.09561\.Cited by:[§2\.3](https://arxiv.org/html/2607.22781#S2.SS3.p2.3)\.
- \[20\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2607.22781#S2.p2.1)\.
- \[21\]R\. Xiong, Y\. Yang, D\. He, K\. Zheng, S\. Zheng, C\. Xing, H\. Zhang, Y\. Lan, L\. Wang, and T\. Liu\(2020\)On layer normalization in the transformer architecture\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 10524–10533\.Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p2.1),[§6\.4](https://arxiv.org/html/2607.22781#S6.SS4.p2.1)\.
- \[22\]D\. Xu, C\. Ruan, E\. Korpeoglu, S\. Kumar, and K\. Achan\(2020\)Inductive representation learning on temporal graphs\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p1.1)\.
- \[23\]C\. Yang, H\. Mei, and J\. Eisner\(2022\)Transformer embeddings of irregularly spaced events and their participants\.InInternational Conference on Learning Representations,Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.1](https://arxiv.org/html/2607.22781#S6.SS1.p1.1)\.
- \[24\]Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259)Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.3](https://arxiv.org/html/2607.22781#S6.SS3.p1.1)\.
- \[25\]L\. Yi, R\. Lei, F\. Mo, Y\. Zheng, Z\. Wei, and Y\. Ye\(2025\)Future link prediction without memory or aggregation\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p1.1)\.
- \[26\]L\. Yu, L\. Sun, B\. Du, and W\. Lv\(2023\)Towards better dynamic graph learning: new architecture and unified library\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2\.1](https://arxiv.org/html/2607.22781#S2.SS1.p1.1)\.
- \[27\]S\. Zhang and L\. Xie\(2020\)Improving attention mechanism in graph neural networks via cardinality preservation\.InInternational Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 1395–1402\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2020/194)Cited by:[§2\.3](https://arxiv.org/html/2607.22781#S2.SS3.p1.1)\.
- \[28\]Z\. Zhou, X\. Yang, R\. Rossi, H\. Zhao, and R\. Yu\(2022\)Neural point process for learning spatiotemporal event dynamics\.InProceedings of the 4th Annual Learning for Dynamics and Control Conference,Proceedings of Machine Learning Research, Vol\.168,pp\. 777–789\.Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.4](https://arxiv.org/html/2607.22781#S6.SS4.p1.1)\.
- \[29\]C\. Zhu, M\. Chen, C\. Fan, G\. Cheng, and Y\. Zhang\(2021\)Learning from history: modeling temporal knowledge graphs with sequential copy\-generation networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 4732–4740\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v35i5.16604)Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1)\.
- \[30\]S\. Zuo, H\. Jiang, Z\. Li, T\. Zhao, and H\. Zha\(2020\)Transformer hawkes process\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 11692–11702\.Cited by:[§2\.4](https://arxiv.org/html/2607.22781#S2.SS4.p1.1),[§6\.1](https://arxiv.org/html/2607.22781#S6.SS1.p1.1)\.

## Appendix ADefinitions and Application Details

This appendix specifies the temporal scope of the recovery targets and the attention boundaries at whichMAAis applied\.

### A\.1Recovery Targets

All recovery targets are computed solely from observations preceding the query time\. Table[XI](https://arxiv.org/html/2607.22781#A1.T11)reports the exact formulas, transformations, fallback values, and coverage used in the implementation\. Homophily is the continuous Jaccard similarity between endpoint neighborhoods, not a binary community label\. Explicit fallback values make every target, including recency and periodicity, numerically defined for every query\.

All eight CTDG targets are evaluated byR2R^\{2\}on the same chronological held\-out split, and broad recovery is their arithmetic mean\. Evidence support in the table is the proportion of queries with observed evidence for the statistic rather than its fallback value; it is not used as an evaluation mask\. Only the binary RAG evidence condition is evaluated separately with AUC\.

TABLE XI:Exact definitions of the CTDG recovery targets\. Every value is computed from observations preceding query timettand is numerically defined for every query\. Support reports the dataset\-wise proportion of queries with observed evidence rather than a fallback value\.TargetDefinitionTransformation and fallbackMetricQuery coverageEvidence supportPair frequencyCardinality of the preceding pair\-event setTu​vT\_\{uv\}log⁡\(1\+\|Tu​v\|\)\\log\(1\+\|T\_\{uv\}\|\); 0 for an unseen pairR2R^\{2\}100%28\.05–42\.82%Recencyt−max⁡Tu​vt\-\\max T\_\{uv\}log\(1\+⋅\)\\log\(1\+\\cdot\); source\-history age for an unseen pair, or 0 if the source also has no historyR2R^\{2\}100%28\.05–42\.82%PersistenceFraction ofBBbins in the source\-history span occupied by pair eventsB=min⁡\(10,max⁡\(2,⌊\|Tu\|\+1⌋\)\)B=\\min\(10,\\max\(2,\\lfloor\\sqrt\{\|T\_\{u\}\|\+1\}\\rfloor\)\); 0 with fewer than two pair eventsR2R^\{2\}100%16\.21–39\.32%PeriodicityRegularity of pair\-event intervalsG=Δ​Tu​vG=\\Delta T\_\{uv\}1/\(1\+std⁡\(G\)/\(mean⁡\(G\)\+10−12\)\)1/\(1\+\\operatorname\{std\}\(G\)/\(\\operatorname\{mean\}\(G\)\+10^\{\-12\}\)\); 0 with fewer than three events or a zero mean intervalR2R^\{2\}100%9\.76–36\.86%GranularityMedian positive interval between source eventslog⁡\(1\+median⁡\(Δ​Tu\+\)\)\\log\(1\+\\operatorname\{median\}\(\\Delta T\_\{u\}^\{\+\}\)\); 0 if no positive interval existsR2R^\{2\}100%89\.18–99\.22%DensityFraction of observed source neighbors\|N​\(u\)\|/\(\|V\|−1\)\|N\(u\)\|/\(\|V\|\-1\)No additional transformationR2R^\{2\}100%100%HomophilyEndpoint\-neighborhood Jaccard similarity\|N​\(u\)∩N​\(v\)\|/\|N​\(u\)∪N​\(v\)\|\|N\(u\)\\cap N\(v\)\|/\|N\(u\)\\cup N\(v\)\|0 if the union is emptyR2R^\{2\}100%97\.87–99\.97%Preferential attachmentEndpoint\-degree productdu​dvd\_\{u\}d\_\{v\}immediately before the querylog⁡\(1\+du​dv\)\\log\(1\+d\_\{u\}d\_\{v\}\)R2R^\{2\}100%100%
### A\.2Attention Boundaries

Within CTDG,MAAis applied only to the temporal aggregation that forms the predictor\-facing representation: CRAFT’s target\-aware cross\-attention, TGAT’s query\-to\-neighbor attention, DyGFormer’s final query attention over encoded patches, and GraphMixer’s final query attention over mixed tokens\. Each model’s internal Transformer, token mixer, feature encoder, and predictor remain unchanged\.

The same rule governs the other domains\. We first identify an attention boundary of the form∑iαi​vi\\sum\_\{i\}\\alpha\_\{i\}v\_\{i\}that forms a predictor\-facing representation, retain the score and value computations, and replace only the normalization denominator fromL1L\_\{1\}toLpL\_\{p\}\. This rule standardizes the scope of modification across models and ensures that recovery changes do not arise from additional hidden dimensions or features\.