Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenders

# Residual Dominance as a Structural Account ofLast-Item Reliance in Causal Self-Attention Recommenders
Source: [https://arxiv.org/html/2608.14021](https://arxiv.org/html/2608.14021)
## Residual Dominance as a Structural Account of Last\-Item Reliance in Causal Self\-Attention RecommendersConference:20th ACM Conference on Recommender Systems; September 27\-October 02, 2026; Minneapolis, MN, USA20th ACM Conference on Recommender Systems \(RecSys ’26\), September 27\-October 02, 2026, Minneapolis, MN, USADOI:[10\.1145/3773078\.3831798](https://doi.org/10.1145/3773078.3831798)ISBN:979\-8\-4007\-2284\-4/2026/09CCS:Information systems Recommender systems

Keito Kozaki,Keigo SakuraiAffiliation:Hokkaido University,Sapporo,Hokkaido,Japanemail:[sakurai@lmd\.ist\.hokudai\.ac\.jp](mailto:[email protected]),Ren TogoAffiliation:Hokkaido University,Sapporo,Hokkaido,Japanemail:[togo@lmd\.ist\.hokudai\.ac\.jp](mailto:[email protected]),Takahiro OgawaAffiliation:Hokkaido University,Sapporo,Hokkaido,Japanemail:[ogawa@lmd\.ist\.hokudai\.ac\.jp](mailto:[email protected])andMiki HaseyamaAffiliation:Hokkaido University,Sapporo,Hokkaido,Japanemail:[mhaseyama@lmd\.ist\.hokudai\.ac\.jp](mailto:[email protected])

2026; © cc

###### Abstract\.

Transformer\-based sequential recommenders with causal self\-attention often rely heavily on the most recent interaction at inference time, but how this behavior is structurally expressed in the representation used for prediction remains unclear\. We combine prediction\-time diagnostics with norm\-based analysis of the full attention block\. First, we show that SASRec\-style models exhibit highly localized last\-item reliance\. We then find that, although self\-attention aggregates contextual information, residual addition sharply shifts the full\-block representation toward same\-position contributions, which we term residual dominance\. To probe this interpretation, we use inference\-time residual scaling as a controlled diagnostic intervention\. Changing the residual strength induces a monotonic trade\-off between structural mixing and last\-item reliance, while reducing residual strength recovers a subset of final\-position misses for which representations at non\-final positions already rank the ground\-truth item correctly\. Our results provide a structural account linking extreme last\-item reliance to residual dominance at inference time\. The code is publicly available\.111[https://github\.com/keito0329/Residual](https://github.com/keito0329/Residual)

###### Keywords:

Sequential Recommendation; Causal Self\-Attention; Last\-Item Reliance

††cc\-license:by## 1\.Introduction

![The figure consists of a sequence-shuffling diagram on the left and nine bar charts on the right. In the partial-shuffle diagram, the final item remains fixed at the final position while all preceding items are randomly permuted. In the full-shuffle diagram, all items, including the original final item, are permuted. The bar charts are arranged in three rows for GRU4Rec, BERT4Rec, and SASRec, and three columns for Toys, ML-1M, and Steam. Each chart compares NDCG at 10 for the original chronological sequence, partial shuffle, and full shuffle. For SASRec on Toys and ML-1M, partial shuffling produces a substantially smaller performance change than full shuffling. On Steam, both perturbations produce relatively small changes. GRU4Rec and BERT4Rec generally exhibit weaker contrasts between partial and full shuffling than SASRec.](https://arxiv.org/html/2608.14021v1/shuffle_merge.png)Figure 1\.Impact of sequence ordering at inference time\. Empty bars: chronologically ordered sequences; light diagonal hatch patterns:*partial shuffle*where only the final positionvtv\_\{t\}is kept fixed and the remaining items are permuted; heavy diagonal hatch patterns:*full shuffle*where all positions, includingvtv\_\{t\}, are permuted\.Key takeaway:on the sparse and dense datasets, preserving onlyvtv\_\{t\}causes a substantially smaller change in SASRec’s performance than full shuffling, indicating a discontinuous last\-position privilege; this contrast is weak on the non\-sequential dataset\.The figure consists of a sequence\-shuffling diagram on the left and nine bar charts on the right\. In the partial\-shuffle diagram, the final item remains fixed at the final position while all preceding items are randomly permuted\. In the full\-shuffle diagram, all items, including the original final item, are permuted\. The bar charts are arranged in three rows for GRU4Rec, BERT4Rec, and SASRec, and three columns for Toys, ML\-1M, and Steam\. Each chart compares NDCG at 10 for the original chronological sequence, partial shuffle, and full shuffle\. For SASRec on Toys and ML\-1M, partial shuffling produces a substantially smaller performance change than full shuffling\. On Steam, both perturbations produce relatively small changes\. GRU4Rec and BERT4Rec generally exhibit weaker contrasts between partial and full shuffling than SASRec\.Sequential recommendation predicts\([4](https://arxiv.org/html/2608.14021#bib.bib1);[37](https://arxiv.org/html/2608.14021#bib.bib2);[24](https://arxiv.org/html/2608.14021#bib.bib44)\)a user’s next interaction from an ordered sequence of past interactions\. Among many approaches\([28](https://arxiv.org/html/2608.14021#bib.bib6);[44](https://arxiv.org/html/2608.14021#bib.bib7);[10](https://arxiv.org/html/2608.14021#bib.bib18);[33](https://arxiv.org/html/2608.14021#bib.bib17)\), Transformer\([36](https://arxiv.org/html/2608.14021#bib.bib20)\)\-based models with causal self\-attention\([39](https://arxiv.org/html/2608.14021#bib.bib16);[31](https://arxiv.org/html/2608.14021#bib.bib43)\), exemplified by SASRec\([12](https://arxiv.org/html/2608.14021#bib.bib3)\), have become a dominant paradigm due to their strong empirical performance\([13](https://arxiv.org/html/2608.14021#bib.bib14);[17](https://arxiv.org/html/2608.14021#bib.bib15)\)\. These models typically generate recommendations from the final\-position representation, implicitly assuming that it aggregates useful signals from the full interaction history\. In this work, we revisit this assumption at inference time:*how is information from different positions actually expressed in the representation used for prediction?*Importantly, our goal is not to determine why a dataset or training process makes the last item predictive, but to examine how a trained causal self\-attention recommender structurally expresses last\-item reliance at inference time\.

Recent studies\([41](https://arxiv.org/html/2608.14021#bib.bib8);[23](https://arxiv.org/html/2608.14021#bib.bib9)\)suggest that causal self\-attention recommenders often exhibit highly localized inference\-time behavior, where predictions are strongly anchored to the most recent interaction\. As illustrated in Figure[1](https://arxiv.org/html/2608.14021#acmlabel1), preserving only the last item while perturbing earlier positions causes only minor changes, whereas also perturbing the final position leads to a sharp degradation\. Complementary ranking diagnostics\([23](https://arxiv.org/html/2608.14021#bib.bib9)\)show that causal self\-attention models frequently rank the last interacted item first, while doing so far less often for immediately preceding items\. Although these diagnostics capture different aspects of model behavior, together they indicate a discontinuous privilege of the final position\. While this behavior is consistent with the intuition that recent interactions are often informative, its extreme localization remains insufficiently explained\.

If positional influence were determined mainly by attention weights, one might expect a gradual decay from the final position to nearby preceding positions\([38](https://arxiv.org/html/2608.14021#bib.bib12)\)\. Instead, we observe a discontinuous pattern: predictive influence collapses onto the final item, while immediately preceding positions contribute little\. Prior explanations based on recency bias or attention patterns therefore remain largely descriptive and do not fully account for how learned signals are actually utilized at prediction time\([38](https://arxiv.org/html/2608.14021#bib.bib12);[41](https://arxiv.org/html/2608.14021#bib.bib8)\)\. This suggests that components beyond attention weights play an important role in shaping inference behavior\.

To study this question, we analyze the full attention block, including residual connections and normalization, using norm\-based decomposition\([15](https://arxiv.org/html/2608.14021#bib.bib10);[16](https://arxiv.org/html/2608.14021#bib.bib11)\)\. Our results show that, although the attention output aggregates contextual information, adding the residual pathway sharply reduces the relative contribution of preceding positions and strongly preserves same\-position information\. Because SASRec forms its prediction from the final\-position representation, this self\-information\-retention structure provides a direct pathway through which the most recent item can dominate the prediction representation\. We refer to this structural tendency as*residual dominance*\. We do not interpret residual dominance as the sole origin of last\-item reliance, but as an important inference\-time mechanism through which learned last\-item reliance is expressed\.

To probe this interpretation, we employ inference\-time residual scaling as a controlled diagnostic intervention\. Residual scaling provides a one\-dimensional modification of the residual contribution while keeping the trained parameters fixed, allowing us to examine whether representation structure and prediction behavior vary systematically with residual strength without retraining\. This intervention is intended as a mechanistic sensitivity probe rather than strict causal identification or a new recommendation method\. Our results show that reducing residual strength weakens the direct preservation of the final\-position input, while recovering a subset of previously missed predictions for which representations at non\-final positions already rank the ground\-truth item correctly under standard inference\. These findings suggest that prediction errors can depend not only on what information is encoded, but also on how that information is expressed at inference time\.

Overall, we provide a structural account of how residual dominance is associated with last\-item reliance and use residual scaling to probe prediction\-time information utilization\.

In summary, the contributions of this work are as follows:

- •We systematically characterize last\-item reliance in causal sequential recommenders using complementary prediction\-time diagnostics across diverse datasets\.
- •We analyze the complete attention block and show that the residual pathway sharply reduces contextual mixing while preserving same\-position information, providing a structural account of how the final item can dominate the representation used for prediction\.
- •We use inference\-time residual scaling as a controlled diagnostic probe and show that residual strength is systematically associated with last\-item reliance, contextual mixing, and the recovery of final\-position misses with correct non\-final predictions\.

## 2\.Preliminaries and Overall Setup

This section defines the task notation and summarizes the datasets, baselines, and evaluation protocol\. Unless otherwise specified, the settings described here are applied to all experiments and diagnostics in Sections[3](https://arxiv.org/html/2608.14021#S3)–[5](https://arxiv.org/html/2608.14021#S5)\.

### 2\.1\.Preliminaries

Consider a user set𝒰=\{u1,u2,…,u\|𝒰\|\}\\mathcal\{U\}=\\\{u\_\{1\},u\_\{2\},\\dots,u\_\{\|\\mathcal\{U\}\|\}\\\}and an item set𝒱=\{v1,v2,…,v\|𝒱\|\}\\mathcal\{V\}=\\\{v\_\{1\},v\_\{2\},\\dots,v\_\{\|\\mathcal\{V\}\|\}\\\}\. For a useruu, we denote the chronological interaction sequence asSu=\(v1,v2,…,vt\)S\_\{u\}=\(v\_\{1\},v\_\{2\},\\dots,v\_\{t\}\), wheret=\|Su\|t=\|S\_\{u\}\|andvtv\_\{t\}is the most recent item\. The next\-item prediction task is to estimate a probability distribution over𝒱\\mathcal\{V\}for the next itemvt\+1v\_\{t\+1\}:

\(1\)p⁡\(vt\+1=v∣Su;Θ\),v∈𝒱,p\(v\_\{t\+1\}=v\\mid S\_\{u\};\\Theta\),\\quad v\\in\\mathcal\{V\},whereΘ\\Thetarepresents the model parameters\. Throughout the remainder of the paper, we denote the final position byLL\(i\.e\.,L=tL=t\), and refer to preceding positions asL−1L\-1,L−2L\-2, etc\., when discussing position\-wise diagnostics\.

### 2\.2\.Overall Setup

Table 1\.Statistics of the processed datasets afterpp\-core filtering \(p=5p=5\) and consecutive\-repeat removal\.#### 2\.2\.1\.Datasets

We conduct our analysis on nine publicly available datasets that are widely used in sequential recommendation research and collectively cover a broad range of domains, sparsity levels, sequence lengths, and temporal characteristics\.

As summarized in Table[1](https://arxiv.org/html/2608.14021#S2.T1), these datasets vary in scale and structure, spanning both short session\-based interactions and long\-term user histories\. All interactions are treated as implicit feedback, following the standard next\-item prediction setting\([13](https://arxiv.org/html/2608.14021#bib.bib14);[35](https://arxiv.org/html/2608.14021#bib.bib5)\)\.

Importantly, the datasets used in this study are deliberately chosen to reflect different degrees of sequential structure discussed in recent literature\([14](https://arxiv.org/html/2608.14021#bib.bib29)\)\. Specifically, our benchmark includes datasets that are widely regarded as exhibiting clear sequential dependencies and being well suited for sequential recommendation, such as the Amazon Beauty, Sports, and Zvuk datasets, as well as datasets for which prior work\([14](https://arxiv.org/html/2608.14021#bib.bib29)\)has reported weaker or more limited sequential structure, including Diginetica and Steam\. By covering datasets with substantially different degrees of inherent sequentiality, our experimental setting enables us to examine whether prediction\-time information utilization, and in particular last\-item reliance, consistently emerges across datasets, rather than being an artifact of a specific dataset choice\.

To ensure data quality and comparability across datasets, we apply55\-core filtering\([29](https://arxiv.org/html/2608.14021#bib.bib24);[5](https://arxiv.org/html/2608.14021#bib.bib25);[34](https://arxiv.org/html/2608.14021#bib.bib26)\)\. In addition, consecutive repeated items within user sequences are removed\([9](https://arxiv.org/html/2608.14021#bib.bib23)\), as such repetitions do not convey additional sequential dependency information and may artificially inflate recency effects\.

#### 2\.2\.2\.Baseline Models

We consider three representative baseline models for sequential recommendation: GRU4Rec, SASRec, and BERT4Rec\. These models are widely used benchmarks and represent three distinct modeling paradigms\. These three models, as employed in analytical studies such as\([13](https://arxiv.org/html/2608.14021#bib.bib14);[1](https://arxiv.org/html/2608.14021#bib.bib13)\), generate predictions from a single sequence\-level representation\. This shared prediction interface allows us to directly compare their prediction\-time behaviors across different inductive biases\. In addition to these standard baselines, we incorporate two models to extend our analysis:

- •SASRec\([12](https://arxiv.org/html/2608.14021#bib.bib3)\)represents causal self\-attention–based Transformers, in which next\-item prediction relies on the final\-position representation under a strictly autoregressive information flow\.
- •GRU4Rec\([10](https://arxiv.org/html/2608.14021#bib.bib18)\)represents recurrent architectures, where user preferences are incrementally aggregated through sequential hidden\-state transitions\.
- •BERT4Rec\([33](https://arxiv.org/html/2608.14021#bib.bib17)\)represents bidirectional self\-attention–based Transformers, which leverage both past and future context through an item\-masking objective\. At inference time, we predict by masking the target position\.
- •DuoRec\([26](https://arxiv.org/html/2608.14021#bib.bib4)\)is a representative model that builds upon SASRec by incorporating a contrastive learning loss\. We utilize this model to analyze how modifications to the loss function affect the model’s dependency on the final item\.
- •BSARec\([31](https://arxiv.org/html/2608.14021#bib.bib43)\)is a recently proposed model that, while utilizing a self\-attention mechanism similar to SASRec, operates within the frequency domain\.

Implementation details of each model and the training environment are provided in our repository\.

#### 2\.2\.3\.Evaluation Metrics

Following\([6](https://arxiv.org/html/2608.14021#bib.bib19)\), we evaluate all models under a Global Time Split \(GTS\) protocol and report NDCG@K and HR@K on the test set; we setK=10K=10\. Prior work has shown that leave\-one\-out evaluation allows future interactions to be included in the training set, resulting in temporal information leakage and overly optimistic performance estimates\([32](https://arxiv.org/html/2608.14021#bib.bib27);[9](https://arxiv.org/html/2608.14021#bib.bib23);[6](https://arxiv.org/html/2608.14021#bib.bib19)\)\. To avoid this issue, we adopt a GTS protocol, which strictly separates training and testing interactions in time\. Specifically, we employ the*GTS\-Last*target strategy together with a GT\-based validation split, where the training and validation sets jointly account for 90% of the data and the remaining portion is reserved for testing\.

Finally, we report results using*full\-catalog ranking*, in which each model ranks the ground\-truth item against the entire item set\. This choice avoids sampling\-based metrics, which have been shown to yield inconsistent results and potentially misleading model comparisons\([3](https://arxiv.org/html/2608.14021#bib.bib22);[18](https://arxiv.org/html/2608.14021#bib.bib21);[9](https://arxiv.org/html/2608.14021#bib.bib23)\)\. To ensure a fair evaluation of next\-item prediction, we exclude previously interacted items from the top\-K recommendation lists \(i\.e\., filter\-seen\)\([11](https://arxiv.org/html/2608.14021#bib.bib28)\)\. However, we bypass this filtering process for Diginetica and Zvuk, as repeat consumption is a characteristic feature of user behavior in these specific datasets\([14](https://arxiv.org/html/2608.14021#bib.bib29)\)\.

## 3\.Empirical Characterization of Last\-Item Reliance

In this section, we empirically characterize the inference\-time behavior of SASRec\([12](https://arxiv.org/html/2608.14021#bib.bib3)\), with a particular focus on its tendency to rely on the last item in the input sequence\. Using a diverse set of datasets, we characterize the structure of inference\-time behavior through two complementary diagnostic tools: inference\-time position shuffling and a recency\-localized hit\-rate metric\.

Figure 2\.HRLI@1 and HRL2I@1 across models and datasets \(computed without filter\-seen\)\. HRLI@1 measures whether the last interacted itemvtv\_\{t\}appears in the top\-1 list; HRL2I@1 is defined analogously forvt−1v\_\{t\-1\}\.A grouped bar chart compares HRLI at 1 and HRL2I at 1 for five models across nine datasets: Beauty, Sports, Video, Diginetica, Toys, Steam, BeerAdvocate, ML\-1M, and Zvuk\. For each model and dataset, a solid bar represents the frequency with which the last interacted item is ranked first, and a hatched bar represents the corresponding frequency for the second\-to\-last item\. SASRec, DuoRec, and BSARec generally show tall solid bars and very small hatched bars, producing a large gap between HRLI and HRL2I\. The largest solid bars occur for several sparse datasets, including Beauty, Sports, and Toys, as well as Zvuk\. BERT4Rec and GRU4Rec generally show lower HRLI values and smaller differences between the two positional metrics\. Diginetica and Steam exhibit comparatively smaller last\-item localization than several other datasets, although the solid bars for the causal self\-attention models remain higher than their hatched counterparts\.### 3\.1\.Inference\-Time Position Shuffling

We first verify position sensitivity via controlled input perturbations, which serve as a counterfactual probe of positional influence\. Specifically, given an interaction sequence, we construct variants where \(i\) only the final item is preserved while earlier positions are randomly permuted, and \(ii\) the entire sequence is permuted\. Comparing prediction performance across these variants isolates the effect of the final position\.

##### Observations\.

Figure[1](https://arxiv.org/html/2608.14021#acmlabel1)illustrates representative results for sparse, dense, and weakly sequential datasets\. Across the complete nine\-dataset results, SASRec exhibits three distinct behavioral regimes characterized by data density and sequentiality\.

On sparse datasets, SASRec is largely insensitive to partial shuffling but shows a sharp performance drop under full shuffling\. This indicates that the final item contains useful signals and that SASRec’s predictions are heavily dependent on it\. On denser datasets such as ML\-1M, while partial shuffling leads to a moderate decrease in performance, a much more substantial drop is still observed under full shuffling\.

For datasets with inherently weak sequential signals, such as Steam and Diginetica\([14](https://arxiv.org/html/2608.14021#bib.bib29)\), shuffling has a negligible effect on all models\. These observations align with prior findings\([41](https://arxiv.org/html/2608.14021#bib.bib8)\); full results for all datasets and additional model variants, including DuoRec and BSARec, are available in our repository\. DuoRec and BSARec also exhibit trends similar to SASRec\.

### 3\.2\.Quantifying Last\-Item Reliance via HRLI

While inference\-time shuffling probes a model’s sensitivity to perturbations in input order, it does not directly characterize how the predicted ranking is structured under standard inference\. To more directly quantify positional bias in the predicted ranking, we adopt the*Hit Rate of the Last Item*\(HRLI@K\)\([23](https://arxiv.org/html/2608.14021#bib.bib9)\)as a diagnostic measure\. This metric enables a systematic and quantitative characterization of positional bias, and will be used throughout the subsequent analysis\.

HRLI@K is defined as

\(2\)HRLI​@​K=1\|𝒟\|​∑Su∈𝒟𝕀⁡\(vt∈Top​\-​K​\(Su\)\),\\mathrm\{HRLI\}@K=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{S\_\{u\}\\in\\mathcal\{D\}\}\\mathbb\{I\}\\bigl\(v\_\{t\}\\in\\mathrm\{Top\}\\text\{\-\}K\(S\_\{u\}\)\\bigr\),where𝒟\\mathcal\{D\}denotes the evaluation set of user sequences,Top​\-​K​\(Su\)\\mathrm\{Top\}\\text\{\-\}K\(S\_\{u\}\)is the recommendation list induced by the predicted scoresp⁡\(vt\+1=v∣Su;Θ\)p\(v\_\{t\+1\}=v\\mid S\_\{u\};\\Theta\), and𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function that equals11if the condition is satisfied and00otherwise\. Intuitively, HRLI@K measures the fraction of user sequences for which the last item in the input sequence reappears in the Top\-KKrecommendation list\.

Similarly, we define the*Hit Rate of the Second\-to\-Last Item*\(HRL2I@K\) by replacingvtv\_\{t\}with the second\-to\-last itemvt−1v\_\{t\-1\}\. Together, HRLI and HRL2I quantify how frequently items at specific positions in the input sequence are recovered in the predicted ranking\.

In contrast to the standard ranking metrics used in Section[2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3), HRLI and HRL2I are computed without applying filter\-seen across all datasets\. This choice follows directly from the definition of these metrics, which are designed to measure how frequently a specific item appearing at a given position in the input sequence is ranked by the model\. Applying filter\-seen would trivially prevent the target item from appearing in the recommendation list, rendering HRLI and HRL2I ill\-defined or uninformative\. Although this evaluation setting differs from typical deployment scenarios, it is necessary to faithfully capture the intended quantity measured by these diagnostics\. HRLI should not be interpreted as a direct counterfactual measure of dependence on the final item\. Rather, it measures how frequently the model ranks an item from a specified input position within the recommendation list under standard inference\. We therefore use HRLI as a recency\-localized ranking diagnostic that complements the perturbation\-based analysis in Section[3\.1](https://arxiv.org/html/2608.14021#S3.SS1)\. Its interpretation requires additional care in repeat\-consumption domains\.

Throughout our experiments, we focus on the caseK=1K=1\. Under this setting, HRLI@1 captures the most extreme form of last\-item localization, namely the case in which the last item is ranked first\. We adopt this setting for clarity, as it provides a sharp and easily interpretable signal of positional dominance\. Results for larger values ofKKexhibit similar qualitative trends and are reported in our repository\.

##### Observations\.

Figure[2](https://arxiv.org/html/2608.14021#acmlabel2)shows HRLI@1 and HRL2I@1 values for each model\. Our results across nine datasets with varying domains and properties reveal that causal self\-attention models, including SASRec, DuoRec, and BSARec, generally yield substantially higher HRLI@1 than HRL2I@1\. Conversely, BERT4Rec and GRU4Rec display a much smaller gap between these metrics, despite a slight inclination toward HRLI@1\. These results reveal a highly localized ranking tendency toward the last interacted item in SASRec\-based models\. Even in datasets such as Steam and Diginetica, HRLI consistently exceeds HRL2I in relative terms\. This provides a complementary perspective to our shuffling experiments, revealing a degree of last\-item reliance that remained undetected by sensitivity\-based diagnostics\.

Taken together, these complementary diagnostics provide consistent evidence of highly localized last\-item behavior in SASRec, but remain agnostic to the mechanisms responsible for this behavior\. To move beyond descriptive evidence, we next analyze how information from different positions is structurally composed within the attention block\.

![Four heatmaps visualizing position-wise contribution patterns for SASRec on the Toys and ML-1M datasets. From left to right, the figure shows attention-only outputs (Attn) and full attention-block representations including residual connections and layer normalization (AttnResLN) for Toys and ML-1M. Compared to attention-only outputs, the full attention block exhibits stronger diagonal components, indicating increased preservation of self-information.](https://arxiv.org/html/2608.14021v1/section4_heatmap.png)Figure 3\.Input\-wise contribution patterns in SASRec for the representative sparse and dense datasets, Toys and ML\-1M\. From left to right: Toys \(Attn\), Toys \(AttnResLN\), ML\-1M \(Attn\), and ML\-1M \(AttnResLN\)\. Compared with the attention\-only contributions \(Attn\), the complete attention block outputs after residual addition and layer normalization \(AttnResLN\) exhibit markedly stronger diagonal components, indicating stronger preservation of same\-position information\.Four heatmaps visualizing position\-wise contribution patterns for SASRec on the Toys and ML\-1M datasets\. From left to right, the figure shows attention\-only outputs \(Attn\) and full attention\-block representations including residual connections and layer normalization \(AttnResLN\) for Toys and ML\-1M\. Compared to attention\-only outputs, the full attention block exhibits stronger diagonal components, indicating increased preservation of self\-information\.

## 4\.Structural Analysis of Last\-Item Reliance

The empirical analyses in Section[3](https://arxiv.org/html/2608.14021#S3)reveal that SASRec exhibits last\-item reliance, as evidenced by both inference\-time shuffling and HRLI diagnostics\. To examine how this behavior is structurally expressed in the representation used for prediction, we analyze the contributions of different input positions to the attention block output\.

### 4\.1\.Analysis of Attention Block

A common approach to analyzing self\-attention models is to inspect attention weights\([38](https://arxiv.org/html/2608.14021#bib.bib12)\)\. However, attention weights alone are insufficient to explain the observed behavior\. In particular, attention weights typically vary smoothly across positions, whereas the last\-item reliance observed in Section[3](https://arxiv.org/html/2608.14021#S3)emerges as a highly discontinuous phenomenon\. This discrepancy suggests that “where the model’s attention is placed” does not necessarily reflect “which information is actually used”\.

This limitation arises because attention weights alone do not indicate which information is actually utilized in the final representation\. While attention weights describe how strongly each position is attended to, they do not directly quantify how much information from that position survives in the representation used for prediction\. In practice, information aggregation within a sequence is influenced not only by the attention mechanism, but also by other architectural components, most notably residual connections\. To address this issue, we analyze the full attention block, explicitly accounting for all its constituent components\.

Specifically, we adopt a norm\-based analysis\([15](https://arxiv.org/html/2608.14021#bib.bib10);[16](https://arxiv.org/html/2608.14021#bib.bib11)\)\. This approach quantifies each position’s contribution using the norm of its input\-wise component in the block output, rather than relying on attention weights alone\. It thereby accounts for variations in the norms of transformed value vectors, residual addition, and layer normalization\.

Formally, letxix\_\{i\}denote the input representation at positionii\. The output of a causal self\-attention block is:

x~i=LN​\(Attn​\(xi,X\)\+xi\)\.\\tilde\{x\}\_\{i\}=\\text\{LN\}\(\\text\{Attn\}\(x\_\{i\},X\)\+x\_\{i\}\)\.
Following\([15](https://arxiv.org/html/2608.14021#bib.bib10);[16](https://arxiv.org/html/2608.14021#bib.bib11)\), this representation can be decomposed into input\-wise contributions:

x~i=∑jFi​\(xj\),\\tilde\{x\}\_\{i\}=\\sum\_\{j\}F\_\{i\}\(x\_\{j\}\),whereFi​\(xj\)F\_\{i\}\(x\_\{j\}\)denotes the contribution of positionjjto positionii\.

We quantify contextual aggregation using the mixing ratio:

ri=∑j≠i‖Fi​\(xj\)‖∑j‖Fi​\(xj\)‖\.r\_\{i\}=\\frac\{\\sum\_\{j\\neq i\}\\\|F\_\{i\}\(x\_\{j\}\)\\\|\}\{\\sum\_\{j\}\\\|F\_\{i\}\(x\_\{j\}\)\\\|\}\.
Lower values indicate dominance of self\-information retention, while higher values suggest stronger aggregation of contextual information through the attention mechanism\. Mixing ratios are computed using norm\-based decomposition of the full attention block, in which multi\-head attention is already integrated into a single representation and thus requires no explicit head\-wise averaging\. Table[2](https://arxiv.org/html/2608.14021#S4.T2)reports values averaged across positions, layers, and test sequences to provide a compact cross\-dataset comparison\. To examine whether residual dominance is specific to the final position or concentrated in a particular layer, we additionally report position\- and layer\-specific AttnResLN mixing ratios for representative sparse and dense datasets in Table[3](https://arxiv.org/html/2608.14021#S4.T3)\. Because representations at deeper layers incorporate transformations from preceding layers, these values characterize the representation after each layer rather than the isolated incremental effect of that layer\. Since this approach is applicable to attention\-based models with a compatible block structure regardless of their learning objective, we extend our analysis to DuoRec as well\. Details of the derivation are provided in the supplementary material in our repository\.222https://github\.com/keito0329/Residual/blob/main/supplement\.pdf

Table 2\.Mixing ratios of SASRec and DuoRec\. Attn denotes the attention\-only output, \+Res denotes the output after residual addition, and \+LN denotes the subsequent output after layer normalization\.
### 4\.2\.Results and Structural Interpretation

Table[2](https://arxiv.org/html/2608.14021#S4.T2)reports the mixing ratios computed at different stages of the attention block: the attention output alone \(Attn\), after adding the residual connection \(AttnRes\), and after applying layer normalization \(AttnResLN\)\.

Across all datasets, the Attn values are consistently high, indicating that the attention output aggregates substantial information from other positions\. Once the residual connection is added, however, the mixing ratio drops sharply\. This pattern reveals a consistent structural tendency: although contextual information is aggregated by the attention operation, the residual addition sharply shifts the balance of the block output toward the same\-position contribution\.

The additional effect of layer normalization is relatively minor compared with the effect of residual addition\. Across datasets, the AttnRes and AttnResLN values are nearly identical, suggesting that residual addition is the primary structural factor associated with the reduction in contextual mixing\. While the absolute values of the mixing ratio vary across datasets, the qualitative trend is consistent\.

To illustrate this tendency, Figure[3](https://arxiv.org/html/2608.14021#S3.F3)visualizes the input\-wise contribution patterns in SASRec for the Toys and ML\-1M datasets, comparing the attention output alone with the complete attention block output after residual addition and layer normalization\. Compared with the attention\-only contribution patterns, the full block exhibits markedly stronger diagonal components, indicating strong preservation of same\-position information\.

Table 3\.Position\- and layer\-specific AttnResLN mixing ratios of SASRec on Toys and ML\-1M, representing sparse and dense settings, respectively\. Values are averaged over test sequences\.Table[3](https://arxiv.org/html/2608.14021#S4.T3)further examines whether residual dominance is specific to the final position or concentrated in a particular layer\. Within each layer, the mixing ratios vary only modestly acrossLL,L−1L\-1, andL−2L\-2\. The maximum absolute position\-wise difference is 0\.0032 in layer 1 and 0\.0047 in layer 2 on Toys, and 0\.0186 and 0\.0054, respectively, on ML\-1M\. Notably, the final\-position mixing ratio is not consistently lower than those of the preceding positions; on ML\-1M, it is slightly higher\. Together with the sharp reduction in mixing observed after residual addition in Table[2](https://arxiv.org/html/2608.14021#S4.T2), these results indicate that residual dominance is broadly expressed across the examined positions and in both Transformer layers, rather than being specifically concentrated at the final position\.

These results refine the structural account of last\-item reliance\. The attention block exhibits a broadly self\-preserving tendency across the examined positions\. In SASRec, however, next\-item prediction is formed exclusively from the final\-position representation\. The preserved same\-position component at this prediction interface is therefore anchored to the most recently interacted item\. Consequently, the discontinuous last\-position privilege observed in Section[3](https://arxiv.org/html/2608.14021#S3)can be understood as the interaction between this broadly self\-preserving tendency and the final\-position\-only prediction interface\.

### 4\.3\.Discussion

In principle, the norm\-based analysis employed in this section can be applied to other Transformer\-based sequential recommendation models, including bidirectional models such as BERT4Rec\. However, we deliberately refrain from conducting a structural comparison with BERT4Rec in this work\. In SASRec\-style causal models, predictions are made from the encoder output at the final item position, whose residual input is associated with an actual interacted item\. Consequently, the mixing ratio reflects a structural trade\-off between preserving the final\-position input, which is anchored to the last interacted item, and aggregating information from preceding positions\. In contrast, BERT4Rec predicts the next item from the encoder output at a\[MASK\]position, whose residual input corresponds to a virtual placeholder rather than an observed item\. As a result, the decomposition into preserving versus mixing effects does not align directly with last\-item reliance in sequential recommendation, making the mixing ratio less task\-aligned for structural interpretation\. For this reason, we restrict our structural analysis to SASRec\-style causal models, where the interpretation of information preservation and aggregation is clearer and more directly connected to the prediction process\.

![Effect of inference-time residual scaling on structural mixing, recency bias, and accuracy.](https://arxiv.org/html/2608.14021v1/section5.png)Figure 4\.Effect of inference\-time residual scaling \(α\\alpha\) applied to the residual branch at all sequence positions in SASRec\. Left: structural mixing averaged over positions and layers after the full attention block \(AttnResLN; mixing ratio\) and recency\-localized reliance \(HRLI@1\) as functions ofα\\alpha; the red dashed line indicates the standard settingα=1\\alpha=1\. Right: accuracy–reliance trade\-off, plotting HR@10 and NDCG@10 against HRLI@1; each colored point corresponds to a value ofα\\alpha\(colorbar\), and baseline markers indicate BERT4Rec and GRU4Rec\.Effect of inference\-time residual scaling on structural mixing, recency bias, and accuracy\.

## 5\.Inference\-Time Probing of Residual Contribution

While Section[4](https://arxiv.org/html/2608.14021#S4)provides a structural account of last\-item reliance, it remains unclear whether prediction\-time representations and rankings vary systematically with residual strength\. We therefore apply inference\-time residual scaling as a controlled mechanistic probe and examine its relationship to contextual mixing, last\-item reliance, and ranking outcomes\. This intervention is intended as a sensitivity analysis rather than strict causal identification\.

### 5\.1\.Residual Scaling as a Mechanistic Probe

In a SASRec\-style Transformer block, the representation after self\-attention and residual addition is

\(3\)hi=xi\+Attn​\(x\)i\.h\_\{i\}=x\_\{i\}\+\\mathrm\{Attn\}\(x\)\_\{i\}\.
We modify this computation at inference time:

\(4\)hi\(α\)=α​xi\+Attn​\(x\)i,h\_\{i\}^\{\(\\alpha\)\}=\\alpha x\_\{i\}\+\\mathrm\{Attn\}\(x\)\_\{i\},whereα\\alphamodulates the strength of residual preservation\. Reducingα\\alphaweakens same\-position residual preservation at all sequence positions in every Transformer layer and increases the relative contribution of contextual aggregation\. At the final\-position prediction interface, this reduces the direct preservation of information anchored to the last item\.

As shown in Figure[4](https://arxiv.org/html/2608.14021#S4.F4), decreasingα\\alphaconsistently increases the AttnResLN mixing ratio and decreases HRLI@1\. Residual strength is therefore monotonically associated with both contextual mixing and last\-item reliance, supporting the structural account in Section[4](https://arxiv.org/html/2608.14021#S4)\.

We further analyze how ranking performance changes withα\\alpha\. Asα\\alphadecreases, HR@10 and NDCG@10 consistently degrade, while last\-item reliance is reduced\. This reveals a trade\-off between preserving self\-information at the final position and incorporating contextual information from preceding positions\.

Importantly, this trade\-off does not by itself indicate whether contextual information contributes meaningful predictive signals or merely reflects degradation of the final representation\. These possibilities cannot be distinguished using standard ranking metrics alone\. Therefore, at this stage, residual scaling establishes that prediction behavior varies systematically with residual strength, but does not determine whether additional predictive signals are effectively utilized\. To address this question, we next examine whether useful predictive signals exist beyond the final position and whether reducing residual strength can recover some of these final\-position misses\. Full residual\-scaling curves for all datasets are available in our repository\.

Table 4\.Correct non\-final predictions among final\-position misses\. The first row reports the standard HR@10 obtained from the final\-position representation\. For each preceding positionL−kL\-k, the remaining rows report the joint proportion of all test sequences for which the final\-position prediction misses the ground\-truth item, while the representation at positionL−kL\-kranks it within the top 10\. These values indicate the prevalence of potentially useful predictive signals at non\-final positions\.

### 5\.2\.Correct Predictions at Non\-Final Positions

Here, a non\-final hit means that the representation at positionL−kL\-kranks the final test targetvt\+1v\_\{t\+1\}within the top 10; it does not refer to the position\-specific next\-item target used during training\.

Table[4](https://arxiv.org/html/2608.14021#S5.T4)reports the joint proportion of all test sequences for which the final\-position prediction misses the ground\-truth item, while the representation at a preceding position ranks it within the top 10\. The results show that correct predictions can occur at non\-final positions even when the final\-position prediction fails\.

Across datasets, the largest single\-position joint rate corresponds to approximately 24–46% of the standard HR@10 value, indicating that such cases occur at a non\-negligible scale\. These results establish the presence of potentially useful predictions at non\-final positions\. In the next subsection, we examine whether reducing residual strength can recover the corresponding final\-position misses\.

### 5\.3\.Recovery of Final\-Position Misses

To test whether reducing residual strength can recover final\-position misses with correct non\-final hits, we consider sequences satisfying the following conditions under the standard settingα=1\\alpha=1: \(i\) the final\-position representation fails to rank the final test targetvt\+1v\_\{t\+1\}within the top 10, and \(ii\) at least one representation atL−1L\-1,L−2L\-2, orL−3L\-3ranksvt\+1v\_\{t\+1\}within the top 10\. This conditioning isolates final\-position misses for which a correct non\-final hit is already present under standard inference\.

For each value ofα\\alpha, we measure the fraction of these sequences for which the final\-position representation ranksvt\+1v\_\{t\+1\}within the top 10\. As shown in Figure[5](https://arxiv.org/html/2608.14021#acmlabel5), decreasingα\\alpharecovers a subset of final\-position misses, with peak recovery observed aroundα≈0\.1\\alpha\\approx 0\.1–0\.30\.3\.

Because every evaluated sequence contains a correct non\-final hit under standard inference, the observed recovery is consistent with the hypothesis that reducing residual preservation increases the contribution of non\-final predictive information to the final\-position ranking\.

Figure 5\.Recovery of final\-position misses under inference\-time residual scaling\. For each dataset, we consider sequences for which the standard final\-position inference \(α=1\\alpha=1\) fails to rank the ground\-truth item within the top 10, while at least one representation atL−1L\-1,L−2L\-2, orL−3L\-3ranks it within the top 10\. The curve reports the fraction of such sequences for which the final\-position representation ranks the ground\-truth item within the top 10 after applying residual scaling with factorα\\alphaat all sequence positions\. The recovery rate is zero atα=1\\alpha=1by definition\.A multi\-line plot shows the recovery rate of final\-position misses as a function of the residual scaling factor alpha for nine datasets\. The horizontal axis ranges from complete removal of the residual contribution at alpha equal to zero to the standard setting at alpha equal to one\. The vertical axis shows the fraction of conditioned sequences that are recovered at the final position\. All curves reach zero at alpha equal to one by construction\. For most datasets, recovery increases as alpha is reduced from one, reaches its maximum at an intermediate value around 0\.1 to 0\.3, and then remains stable or decreases slightly as alpha approaches zero\. Diginetica shows the highest peak recovery, exceeding 0\.3, while the remaining datasets exhibit lower but consistently positive peaks\. The common non\-monotonic pattern indicates that moderate residual reduction generally recovers more cases than either the standard residual strength or complete residual removal\.
### 5\.4\.Validation\-Based Selection

While our primary goal is analysis rather than optimization, we include this experiment to demonstrate that the observed controllability is reproducible without access to test data\. We adopt a simple validation\-based selection rule:

\(5\)α∗=arg⁡maxα\\displaystyle\\alpha^\{\*\}=\\arg\\max\_\{\\alpha\}Recoveryval​\(α\)\\displaystyle\\mathrm\{Recovery\}\_\{\\text\{val\}\}\(\\alpha\)s\.t\.\\displaystyle\\text\{s\.t\. \}HR​@​10val​\(α\)≥\(1−ε\)​HR​@​10val​\(1\)\\displaystyle\\mathrm\{HR@10\}\_\{\\text\{val\}\}\(\\alpha\)\\geq\(1\-\\varepsilon\)\\mathrm\{HR@10\}\_\{\\text\{val\}\}\(1\)
This formulation prioritizes recovery while constraining accuracy degradation\. Figure[6](https://arxiv.org/html/2608.14021#acmlabel6)shows, on Toys, that validation\-selectedα\\alphaclosely matches oracle\-selectedα\\alphaover a wide range ofϵ\\epsilonvalues, reproducing the same accuracy–recovery trade\-off observed on the test set\.

These results indicate that the controllability revealed by residual scaling is stable and can be reproduced using standard validation procedures\. However, this experiment is intended only as a demonstration of reproducibility and does not constitute a separate modeling contribution\. Results for additional datasets are available in our repository\.

Figure 6\.Validation\-based reproduction of the accuracy–recovery trade\-off\. \(Top\) Comparison between validation\-selected and oracle\-selectedα\\alphaacross different values ofε\\varepsilon\. \(Bottom\) Corresponding test performance, showing the trade\-off betweenΔ\\DeltaHR@10 relative to theα=1\\alpha=1baseline and recovery under validation\-based selection\. On Toys, validation\-based selection reproduces the same accuracy–recovery trade\-off observed under oracle tuning\.Two vertically stacked plots summarize validation\-based residual\-scaling selection on Toys\. The top plot shows the selected residual scaling factor alpha as a function of the allowed accuracy\-loss parameter epsilon\. The validation\-selected curve and the oracle\-selected curve follow similar downward trajectories: selected alpha is high under a strict accuracy constraint, decreases as epsilon increases, and stabilizes near a small value under looser constraints\. The bottom plot shows the resulting test\-set trade\-off\. The left vertical axis reports the change in HR at 10 relative to standard inference, and the right vertical axis reports recovery\. As epsilon increases, the HR at 10 change becomes more negative, while recovery increases\. Both quantities stabilize once the selected alpha becomes approximately constant\. The two panels together show that validation\-based selection reproduces a trade\-off similar to oracle selection without selecting alpha directly on the test set\.Summary\.Taken together, the results in this section establish a consistent picture: \(1\) prediction behavior varies systematically with residual strength, \(2\) correct predictions can occur at non\-final positions when the final position misses, and \(3\) reducing residual strength recovers a subset of these cases\.

## 6\.Related Work

### 6\.1\.Sequential Recommendation

Sequential recommender systems were initially dominated by RNN\-\([10](https://arxiv.org/html/2608.14021#bib.bib18);[19](https://arxiv.org/html/2608.14021#bib.bib30)\)and CNN\-based models\([35](https://arxiv.org/html/2608.14021#bib.bib5);[43](https://arxiv.org/html/2608.14021#bib.bib31)\), which improved upon Markov chain–based models\([44](https://arxiv.org/html/2608.14021#bib.bib7);[28](https://arxiv.org/html/2608.14021#bib.bib6)\)by capturing longer\-term dependencies\. Transformer\-based architectures later advanced the field by enabling flexible sequence modeling via self\-attention\([33](https://arxiv.org/html/2608.14021#bib.bib17);[12](https://arxiv.org/html/2608.14021#bib.bib3);[39](https://arxiv.org/html/2608.14021#bib.bib16)\), with causal self\-attention models such as SASRec becoming widely adopted baselines\. Despite recent developments including graph\-based methods\([27](https://arxiv.org/html/2608.14021#bib.bib34);[40](https://arxiv.org/html/2608.14021#bib.bib33);[25](https://arxiv.org/html/2608.14021#bib.bib32)\), generative models\([20](https://arxiv.org/html/2608.14021#bib.bib36);[42](https://arxiv.org/html/2608.14021#bib.bib35);[22](https://arxiv.org/html/2608.14021#bib.bib37)\), and LLM\-based approaches\([21](https://arxiv.org/html/2608.14021#bib.bib40);[30](https://arxiv.org/html/2608.14021#bib.bib38);[7](https://arxiv.org/html/2608.14021#bib.bib39)\), causal self\-attention remains a core architectural choice in many sequential recommenders\. Understanding the inference behavior of such foundational models is therefore important for clarifying how sequential information is actually utilized\.

### 6\.2\.Attention Analysis

Prior work has investigated which dependencies self\-attention should emphasize in sequential recommendation\([31](https://arxiv.org/html/2608.14021#bib.bib43);[38](https://arxiv.org/html/2608.14021#bib.bib12)\)\. Several studies\([38](https://arxiv.org/html/2608.14021#bib.bib12);[8](https://arxiv.org/html/2608.14021#bib.bib42)\)highlight the importance of local and short\-term dependencies, showing that unconstrained global attention can be noisy or suboptimal, especially under sparse interaction regimes\. While these works analyze or modify attention mechanisms to improve performance, they largely focus on attention weights or attention modules in isolation\. In contrast, how attention outputs are utilized at prediction time remains underexplored, motivating our analysis of the full attention block with residual connections\.

### 6\.3\.Recency Bias

Recency bias refers to the tendency of sequential recommender systems to rely disproportionately on recent interactions\([2](https://arxiv.org/html/2608.14021#bib.bib41)\)\. Prior work proposed both regularization\-based mitigation strategies\([2](https://arxiv.org/html/2608.14021#bib.bib41)\)and evaluation metrics such as HRLI\([23](https://arxiv.org/html/2608.14021#bib.bib9)\), revealing that causal self\-attention models often exhibit extreme last\-item reliance\. More recent studies\([41](https://arxiv.org/html/2608.14021#bib.bib8)\)further demonstrated that this behavior persists across architectures, embedding sizes, and loss functions, and argued that strong last\-item reliance is essential for achieving high next\-item prediction accuracy\.

In contrast to these studies, which primarily characterize or justify last\-item reliance as an empirical or task\-level property, our work investigates how such extreme dependence is structurally expressed in the prediction representation of SASRec\.

## 7\.Conclusion

We showed that causal self\-attention recommenders often exhibit highly localized last\-item reliance at inference time\. Through norm\-based analysis of the full attention block, we identified residual dominance as a broadly self\-preserving tendency that, together with the final\-position\-only prediction interface, provides a structural account of this behavior\. Inference\-time residual scaling further revealed systematic relationships among residual strength, contextual mixing, and last\-item reliance, and recovered a subset of final\-position misses with correct non\-final hits\. These findings establish prediction\-time information utilization as a useful analytical lens for understanding causal self\-attention recommenders beyond attention weights and ranking accuracy alone\.

Limitations and future work\.Residual dominance should not be interpreted as the sole cause of last\-item reliance\. Other factors, such as causal masking, positional information, the prediction interface, training objectives, data characteristics, and optimization may also contribute\. Our analysis and intervention are restricted to causal self\-attention models that predict from the final\-position representation, and extending this characterization to other prediction heads and training objectives remains an important direction for future work\. In addition, HRLI and HRL2I are designed as diagnostic metrics rather than replacements for standard accuracy metrics, and establishing their connection to online utility is an open problem\. Finally, understanding how these findings translate to real\-world recommendation settings remains an important direction for future work\.

###### Acknowledgements\.

This work was partly supported by JSPS KAKENHI Grant Numbers JP24K02942 and JP23K11141\.

## References

- Betelloet al\.\(2024\)F\. Betello, F\. Siciliano, P\. Mishra, and F\. SilvestriInvestigating the robustness of sequential recommender systems against training data perturbations\.InEuropean Conference on Information Retrieval,pp\. 205–220\.Cited by:[§2\.2\.2](https://arxiv.org/html/2608.14021#S2.SS2.SSS2.p1.1)\.
- Changet al\.\(2022\)B\. Chang, C\. Xu, M\. Lê, J\. Feng, Y\. Le, S\. Badam, E\. Chi, and M\. ChenRecency dropout for recurrent recommender systems\.arXiv preprint arXiv:2201\.11016\.Cited by:[§6\.3](https://arxiv.org/html/2608.14021#S6.SS3.p1.1)\.
- Dallmannet al\.\(2021\)A\. Dallmann, D\. Zoller, and A\. HothoA case study on sampling strategies for evaluating neural sequential item recommendation models\.InProceedings of the ACM Conference on Recommender Systems,pp\. 505–514\.Cited by:[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p2.1)\.
- Fanget al\.\(2020\)H\. Fang, D\. Zhang, Y\. Shu, and G\. GuoDeep learning for sequential recommendation: algorithms, influential factors, and evaluations\.ACM Transactions on Information Systems39\(1\),pp\. 1–42\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1)\.
- Ferrari Dacremaet al\.\(2019\)M\. Ferrari Dacrema, P\. Cremonesi, and D\. JannachAre we really making much progress? a worrying analysis of recent neural recommendation approaches\.InProceedings of the ACM Conference on Recommender Systems,pp\. 101–109\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p4.1)\.
- Gusaket al\.\(2025\)D\. Gusak, A\. Volodkevich, A\. Klenitskiy, A\. Vasilev, and E\. FrolovTime to split: exploring data splitting strategies for offline evaluation of sequential recommenders\.InProceedings of the ACM Conference on Recommender Systems,pp\. 874–883\.Cited by:[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p1.1)\.
- Harteet al\.\(2023\)J\. Harte, W\. Zorgdrager, P\. Louridas, A\. Katsifodimos, D\. Jannach, and M\. FragkoulisLeveraging large language models for sequential recommendation\.InProceedings of the ACM Conference on Recommender Systems,pp\. 1096–1102\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Heet al\.\(2021\)Z\. He, H\. Zhao, Z\. Lin, Z\. Wang, A\. Kale, and J\. McAuleyLocker: locally constrained self\-attentive sequential recommendation\.InProceedings of the ACM International Conference on Information and Knowledge Management,pp\. 3088–3092\.Cited by:[§6\.2](https://arxiv.org/html/2608.14021#S6.SS2.p1.1)\.
- Hidasi and Czapp \(2023\)B\. Hidasi and Á\. T\. CzappWidespread flaws in offline evaluation of recommender systems\.InProceedings of the ACM Conference on Recommender Systems,pp\. 848–855\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p4.1),[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p1.1),[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p2.1)\.
- Hidasiet al\.\(2015\)B\. Hidasi, A\. Karatzoglou, L\. Baltrunas, and D\. TikkSession\-based recommendations with recurrent neural networks\.arXiv preprint arXiv:1511\.06939\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[2nd item](https://arxiv.org/html/2608.14021#S2.I1.i2.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Higleyet al\.\(2022\)K\. Higley, E\. Oldridge, R\. Ak, S\. Rabhi, and G\. de Souza Pereira MoreiraBuilding and deploying a multi\-stage recommender system with merlin\.InProceedings of the ACM Conference on Recommender Systems,pp\. 632–635\.Cited by:[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p2.1)\.
- Kang and McAuley \(2018\)W\. Kang and J\. McAuleySelf\-attentive sequential recommendation\.InProceedings of the IEEE International Conference on Data Mining,pp\. 197–206\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[1st item](https://arxiv.org/html/2608.14021#S2.I1.i1.p1.1),[§3](https://arxiv.org/html/2608.14021#S3.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Klenitskiy and Vasilev \(2023\)A\. Klenitskiy and A\. VasilevTurning dross into gold loss: is bert4rec really better than sasrec?\.InProceedings of the ACM Conference on Recommender Systems,pp\. 1120–1125\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p2.1),[§2\.2\.2](https://arxiv.org/html/2608.14021#S2.SS2.SSS2.p1.1)\.
- Klenitskiyet al\.\(2024\)A\. Klenitskiy, A\. Volodkevich, A\. Pembek, and A\. VasilevDoes it look sequential? an analysis of datasets for evaluation of sequential recommendations\.InProceedings of the ACM Conference on Recommender Systems,pp\. 1067–1072\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p3.1),[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p2.1),[§3\.1](https://arxiv.org/html/2608.14021#S3.SS1.SSS0.Px1.p3.1)\.
- Kobayashiet al\.\(2020\)G\. Kobayashi, T\. Kuribayashi, S\. Yokoi, and K\. InuiAttention is not only a weight: analyzing transformers with vector norms\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 7057–7075\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.14021#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2608.14021#S4.SS1.p5.1)\.
- Kobayashiet al\.\(2021\)G\. Kobayashi, T\. Kuribayashi, S\. Yokoi, and K\. InuiIncorporating residual and normalization layers into analysis of masked language models\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 4547–4568\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.14021#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2608.14021#S4.SS1.p5.1)\.
- Koneruet al\.\(2025\)V\. H\. Koneru, X\. Neufeld, S\. Loth, and A\. GrünSASRec in action: real\-world adaptations for zdf streaming service\.InProceedings of the ACM Conference on Recommender Systems,pp\. 1054–1057\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1)\.
- Krichene and Rendle \(2020\)W\. Krichene and S\. RendleOn sampled metrics for item recommendation\.InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1748–1757\.Cited by:[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p2.1)\.
- Liet al\.\(2017\)J\. Li, P\. Ren, Z\. Chen, Z\. Ren, T\. Lian, and J\. MaNeural attentive session\-based recommendation\.InProceedings of the ACM International Conference on Information and Knowledge Management,pp\. 1419–1428\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Liet al\.\(2023\)Z\. Li, A\. Sun, and C\. LiDiffurec: a diffusion model for sequential recommendation\.ACM Transactions on Information Systems42\(3\),pp\. 1–28\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Liuet al\.\(2024\)Q\. Liu, X\. Wu, Y\. Wang, Z\. Zhang, F\. Tian, Y\. Zheng, and X\. ZhaoLLM\-esr: large language models enhancement for long\-tailed sequential recommendation\.Advances in Neural Information Processing Systems37,pp\. 26701–26727\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Maoet al\.\(2025\)W\. Mao, S\. Liu, H\. Liu, H\. Liu, X\. Li, and L\. HuDistinguished quantized guidance for diffusion\-based sequence recommendation\.InProceedings of the ACM on Web Conference,pp\. 425–435\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Oh and Cho \(2024\)J\. Oh and S\. ChoMeasuring recency bias in sequential recommendation systems\.arXiv preprint arXiv:2409\.09722\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.14021#S3.SS2.p1.1),[§6\.3](https://arxiv.org/html/2608.14021#S6.SS3.p1.1)\.
- Panet al\.\(2026\)L\. Pan, W\. Pan, M\. Wei, H\. Yin, and Z\. MingA survey on sequential recommendation\.Frontiers of Computer Science20\(3\),pp\. 2003606\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1)\.
- Qiuet al\.\(2021\)R\. Qiu, Z\. Huang, T\. Chen, and H\. YinExploiting positional information for session\-based recommendation\.ACM Transactions on Information Systems40\(2\),pp\. 1–24\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Qiuet al\.\(2022\)R\. Qiu, Z\. Huang, H\. Yin, and Z\. WangContrastive learning for representation degeneration problem in sequential recommendation\.InProceedings of the ACM International Conference on Web Search and Data Mining,pp\. 813–823\.Cited by:[4th item](https://arxiv.org/html/2608.14021#S2.I1.i4.p1.1)\.
- Qiuet al\.\(2020\)R\. Qiu, H\. Yin, Z\. Huang, and T\. ChenGAG: global attributed graph neural network for streaming session\-based recommendation\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 669–678\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Rendleet al\.\(2010\)S\. Rendle, C\. Freudenthaler, and L\. Schmidt\-ThiemeFactorizing personalized markov chains for next\-basket recommendation\.InProceedings of the ACM International Conference on World Wide Web,pp\. 811–820\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Sachdeva and McAuley \(2020\)N\. Sachdeva and J\. McAuleyHow useful are reviews for recommendation? a critical review and potential improvements\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1845–1848\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p4.1)\.
- Sakuraiet al\.\(2025\)K\. Sakurai, R\. Togo, T\. Ogawa, and M\. HaseyamaLLM is knowledge graph reasoner: llm’s intuition\-aware knowledge graph reasoning for cold\-start sequential recommendation\.InProceedings of the European Conference on Information Retrieval,pp\. 263–278\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Shinet al\.\(2024\)Y\. Shin, J\. Choi, H\. Wi, and N\. ParkAn attentive inductive bias for sequential recommendation beyond the self\-attention\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 8984–8992\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[5th item](https://arxiv.org/html/2608.14021#S2.I1.i5.p1.1),[§6\.2](https://arxiv.org/html/2608.14021#S6.SS2.p1.1)\.
- Sun \(2023\)A\. SunTake a fresh look at recommender systems from an evaluation standpoint\.InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2629–2638\.Cited by:[§2\.2\.3](https://arxiv.org/html/2608.14021#S2.SS2.SSS3.p1.1)\.
- Sunet al\.\(2019\)F\. Sun, J\. Liu, J\. Wu, C\. Pei, X\. Lin, W\. Ou, and P\. JiangBERT4Rec: sequential recommendation with bidirectional encoder representations from transformer\.InProceedings of the ACM International Conference on Information and Knowledge Management,pp\. 1441–1450\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[3rd item](https://arxiv.org/html/2608.14021#S2.I1.i3.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Sunet al\.\(2020\)Z\. Sun, D\. Yu, H\. Fang, J\. Yang, X\. Qu, J\. Zhang, and C\. GengAre we evaluating rigorously? benchmarking recommendation for reproducible evaluation and fair comparison\.InProceedings of the ACM Conference on Recommender Systems,pp\. 23–32\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p4.1)\.
- Tang and Wang \(2018\)J\. Tang and K\. WangPersonalized top\-n sequential recommendation via convolutional sequence embedding\.InProceedings of the ACM International Conference on Web Search and Data Mining,pp\. 565–573\.Cited by:[§2\.2\.1](https://arxiv.org/html/2608.14021#S2.SS2.SSS1.p2.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1)\.
- Wanget al\.\(2019\)S\. Wang, L\. Hu, Y\. Wang, L\. Cao, Q\. Z\. Sheng, and M\. OrgunSequential recommender systems: challenges, progress and prospects\.arXiv preprint arXiv:2001\.04830\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1)\.
- Wanget al\.\(2025\)Y\. Wang, Z\. He, Z\. Yue, J\. McAuley, and D\. WangYour causal self\-attentive recommender hosts a lonely neighborhood\.InProceedings of the ACM International Conference on Web Search and Data Mining,pp\. 688–696\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.14021#S4.SS1.p1.1),[§6\.2](https://arxiv.org/html/2608.14021#S6.SS2.p1.1)\.
- Wuet al\.\(2020\)L\. Wu, S\. Li, C\. Hsieh, and J\. SharpnackSSE\-pt: sequential recommendation via personalized transformer\.InProceedings of the ACM Conference on Recommender Systems,pp\. 328–337\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Xuet al\.\(2019\)C\. Xu, P\. Zhao, Y\. Liu, V\. S\. Sheng, J\. Xu, F\. Zhuang, J\. Fang, and X\. ZhouGraph contextualized self\-attention network for session\-based recommendation\.InProceedings of the International Joint Conference on Artificial Intelligence,pp\. 3940–3946\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Xuet al\.\(2026\)C\. Xu, G\. Li, J\. Wang, and W\. ZhangMarkovian pre\-trained transformer for next\-item recommendation\.arXiv preprint arXiv:2601\.08275\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p2.1),[§1](https://arxiv.org/html/2608.14021#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.14021#S3.SS1.SSS0.Px1.p3.1),[§6\.3](https://arxiv.org/html/2608.14021#S6.SS3.p1.1)\.
- Yanget al\.\(2023\)Z\. Yang, J\. Wu, Z\. Wang, X\. Wang, Y\. Yuan, and X\. HeGenerate what you prefer: reshaping sequential recommendation via guided diffusion\.Advances in Neural Information Processing Systems36,pp\. 24247–24261\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Yuanet al\.\(2019\)F\. Yuan, A\. Karatzoglou, I\. Arapakis, J\. M\. Jose, and X\. HeA simple convolutional generative network for next item recommendation\.InProceedings of the ACM International Conference on Web Search and Data Mining,pp\. 582–590\.Cited by:[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.
- Zimdarset al\.\(2013\)A\. Zimdars, D\. M\. Chickering, and C\. MeekUsing temporal data for making recommendations\.arXiv preprint arXiv:1301\.2320\.Cited by:[§1](https://arxiv.org/html/2608.14021#S1.p1.1),[§6\.1](https://arxiv.org/html/2608.14021#S6.SS1.p1.1)\.