ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

# ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
Source: [https://arxiv.org/html/2607.26369](https://arxiv.org/html/2607.26369)
###### Abstract\.

Rotary Position Embedding \(RoPE\) has been widely adopted in transformer\-based large language models\. However, its log\-linear frequency schedule, originally designed to produce long\-term attention decay, limits its adoption in domains with more complex distance\-correlation patterns, such as temporal periodicity in sequential recommendation\. We investigate the expressiveness of general query/key rotations and find that any normalized continuous positive\-definite attention modulation function can be approximated by random rotations induced by its own Fourier transform, which we term Random Fourier Rotations\. Building on this theory, we propose ClockRoPE for routine modeling in sequential recommendation, where rotation frequencies are derived from periodic attention modulation functions\. In online A/B tests, ClockRoPE demonstrates consistent improvements in valued engagement metrics, and has been successfully deployed in production\-scale generative retrieval system at a major video\-sharing platform\.

## 1\.Introduction

As transformers\(Vaswani et al\.,[2017](https://arxiv.org/html/2607.26369#bib.bib34)\)and the scaling law\(Kaplan et al\.,[2020](https://arxiv.org/html/2607.26369#bib.bib17); Hoffmann et al\.,[2022](https://arxiv.org/html/2607.26369#bib.bib12)\)have demonstrated great success in language modeling\(Devlin et al\.,[2018](https://arxiv.org/html/2607.26369#bib.bib9); Brown et al\.,[2020](https://arxiv.org/html/2607.26369#bib.bib3)\)and image generation\(Peebles and Xie,[2023](https://arxiv.org/html/2607.26369#bib.bib26)\), the generative recommendation field is converging towards transformer\-style architectures for sequence modeling\(Kang and McAuley,[2018](https://arxiv.org/html/2607.26369#bib.bib16); Zhai et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib37)\)\. However different from language modeling, time is a crucial and native signal in sequential recommendation\. Various methods have been explored to incorporate temporal signals in transformer\-based sequential recommendation, including time difference input feature\(Chai et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib4)\), additive temporal attention bias\(Zhai et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib37)\)and relative time embedding within attention\(Li et al\.,[2020](https://arxiv.org/html/2607.26369#bib.bib22)\)\.

On the other hand, RoPE\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32)\)has shown superior performance in transformer\-based language models, featuring properties such as translation\-invariance and long\-term attention decay\. Despite being originally designed for 1D discrete token positions, RoPE has quickly seen adoption in domains with multi\-dimensional continuous coordinates\(Li et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib21); Schenck et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib30); Heo et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib10)\)\.

Naturally, several recent work have argued for extending RoPE to encode time in sequential recommendation\(Wei et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib35); Hou et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib13)\):\(Wei et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib35)\)adopts similar log\-linear frequency spacing as in standard RoPE and replaces position with timestamp delta from an anchor timestamp; ROTE\(Hou et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib13)\)first computes log\-scaled timestamp gaps and constructs rotation angles based on them to capture temporal proximity\. Both work emphasize on using timestamp together with order to encode recency\.

However, an important characteristic of user behavior in recommender systems is periodicity\. Interactions occurring at the same hour of the day or the same day of the week often exhibit high correlations\. Yet, the log\-linear frequency schedule in standard RoPE produces long\-term attention decay and struggles to capture more complex distance\-correlation patterns including periodicity\.

To address this limitation, we leverage Fourier analysis to investigate the expressiveness of general query/key rotations\. We prove that any continuous normalized positive\-definite modulation function can be unbiasedly approximated by sampling RoPE frequencies from their own Fourier transform distribution\. In summary, the contributions of our work include:

- •Random Fourier Rotations \(RFR\):a general sampling method for approximating arbitrary normalized positive\-definite attention modulation functions using query/key rotations\.
- •ClockRoPE:A temporal periodicity encoding approach for user routine modeling, derived from applying Random Fourier Rotations to approximate periodic attention modulation functions\.
- •Production\-Scale Evaluation:Empirical evidence of ClockRoPE’s effectiveness in improving model’s routine\-awareness and overall viewer value within a production\-scale generative retrieval system\.

## 2\.Related Work

### 2\.1\.Relative Positional Encoding

Relative Positional Encoding\(RPE\) techniques\(Shaw et al\.,[2018](https://arxiv.org/html/2607.26369#bib.bib31); Huang et al\.,[2019](https://arxiv.org/html/2607.26369#bib.bib15); Li et al\.,[2023](https://arxiv.org/html/2607.26369#bib.bib23); Choromanski et al\.,[2022](https://arxiv.org/html/2607.26369#bib.bib7); Raffel et al\.,[2020](https://arxiv.org/html/2607.26369#bib.bib27)\)is a powerful class of positional encoding methods used in Transformer models\. In contrast toabsolute positional encodings\(APEs\)\(Vaswani et al\.,[2017](https://arxiv.org/html/2607.26369#bib.bib34); Kazemi et al\.,[2019](https://arxiv.org/html/2607.26369#bib.bib18)\)that add / concatenate sinusoidal\-based embeddings of the tokens’ absolute positions with content embeddings, they more explicitly modulate attention matrix values via \(learnable\) functions of the relative positions / distances between the tokens in the corresponding metric space \(e\.g\. 1D line as it is the case for language, 2D grids for images or 3D grids for videos\)\. RPEs is a rich class of techniques and is often taxonomized via two branches: \(1\) theadditive mechanisms\(Choromanski et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib6),[2022](https://arxiv.org/html/2607.26369#bib.bib7); Li et al\.,[2023](https://arxiv.org/html/2607.26369#bib.bib23)\)that introduce additive terms in the logits space \(numerically equivalent to Hadamard multiplication obtained after element\-wise logits exponentiation\) and \(2\) themultiplicative mechanisms\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32); Cheng et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib5); Heo et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib10); Schenck et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib30)\)that modulate query\-key dot\-products by introducing multiplicative terms \(those two strategies can also be combined into hybrid methods\)\. RPEs are used across different modalities: language\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32)\), images\(Choromanski et al\.,[2022](https://arxiv.org/html/2607.26369#bib.bib7); Heo et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib10); Schenck et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib30)\)and more recently, even point clouds\(Kim et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib19)\)\.

RPEs can be often efficiently implemented, in time near\-linear in the input sequence length \(rather than quadratic\)\. For instance, it is well known\(Choromanski et al\.,[2022](https://arxiv.org/html/2607.26369#bib.bib7); Luo et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib25)\)that as long as the mask matrix𝐌\\mathbf\{M\}used in the Hadamard multiplication of the additive RPE mechanism supports sub\-quadratic \(in sequence length\) matrix\-vector multiplication, the corresponding RPE mechanism can be applied to linear low\-rank attention Transformers\(Choromanski et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib8)\)in sub\-quadratic time\. Similarly, multiplicative RPE mechanisms often support efficient implementations \(see: Sec\.[2\.2](https://arxiv.org/html/2607.26369#S2.SS2)\)\.

### 2\.2\.RoPE & STRING

Rotary Position Encodings, or: RoPE\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32); Heo et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib10); Reid et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib29); Li et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib21); Hua et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib14)\), is an instantiation of the multiplicative RPE mechanism, where the queries and keys are post\-processed by rotations matrices that depend on the positions of their corresponding tokens\. More specifically, a rotation matrix corresponding to the token is defined as a product of disjoint two\-dimensional \(Givens\) rotations with rotations angles defined as linear functions of tokens’s positions \(1D or higher\-dimensional\) with potentially trainable parameters \(different variations include axial\-RoPe, mixed\-RoPE and more\)\. This particular design choice for the rotation matrices corresponding to the tokens is a gateway to re\-writing the modulated attention scores as:𝐪⊤​𝐑​\(δ​𝐫\)​𝐤\\mathbf\{q\}^\{\\top\}\\mathbf\{R\}\(\\delta\\mathbf\{r\}\)\\mathbf\{k\}, where𝐑​\(δ​𝐫\)\\mathbf\{R\}\(\\delta\\mathbf\{r\}\)is a rotation matrix defined as a product of disjoint22\-dimensional rotations with angles given as linear functions of the relative positions\.

The rigid structure of the rotations matrices used to rotate queries and keys in the RoPE mechanism leads totranslation invariance\(the mechanism depends only on the relative positions\), but in practice can be significantly relaxed without sacrificing this property\. The recently introduced STRING method\(Schenck et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib30)\)is a super\-set of RoPEs, where the rotation matrix is obtained by exponentiating \(in the matrix\-exponentiation rather than element\-wise sense\)skew\-symmetric\(anti\-symmetric\) matrices\. Each skew\-symmetric matrix is defined as a linear combination of the skew\-symmetricgenerators\(often learnable\) with coefficients given by the coordinates of the tokens corresponding to queries/keys\. As it is shown in\(Schenck et al\.,[2025](https://arxiv.org/html/2607.26369#bib.bib30)\), as long as the generators commute, the resulting mechanism remains translation\-invariant, with RoPE providing its special instantiation\. Furthermore, it is the most general extension of the RoPE mechanism in the setting, where queries and keys are post\-processed via rotation matrices and that keeps translation\-invariance\.

Both STRING and RoPE trivially support fast computations with efficient attention methods, since they do not require the explicit materialization of the logits matrix, acting independently on the set of queries and keys\.

### 2\.3\.Fourier Methods in Sequence Modeling

Fourier methods are a common tool in sequence modeling\. One early example for Transformer models in particular is FNet\(Lee\-Thorp et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib20)\), which replaced the self\-attention mechanism in Transformer layers with the discrete Fourier transform\. Despite lacking learnable parameters, the discrete Fourier transform served as an efficient token ”mixing” mechanism to replace attention while retaining much of the quality in BERT\(Devlin et al\.,[2018](https://arxiv.org/html/2607.26369#bib.bib9)\)models\.

Fourier analysis has also been used to improve RoPE itself\. FoPE\(Hua et al\.,[2024](https://arxiv.org/html/2607.26369#bib.bib14)\)leverages Discrete Signal Processing theory to improve length generalization, while nD\-RoPE\(Li et al\.,[2026](https://arxiv.org/html/2607.26369#bib.bib21)\)derives a spectral isotropy condition to generalize RoPE to higher\-dimensional spaces\. Both refine RoPE’s existing frequency structure, whereas ClockRoPE derives a new frequency\-sampling distribution targeting periodic attention modulation\.

### 2\.4\.Fourier Features

Fourier features have a broad history in kernel approximation and function representation\. Random Fourier Features \(RFF\)\(Rahimi and Recht,[2007](https://arxiv.org/html/2607.26369#bib.bib28)\)approximate shift\-invariant kernels via random sinusoidal projections sampled from the kernel’s Fourier transform\. These features have since been shown to help networks learn high\-frequency functions\(Tancik et al\.,[2020](https://arxiv.org/html/2607.26369#bib.bib33)\), encode continuous time in self\-attention\(Xu et al\.,[2019](https://arxiv.org/html/2607.26369#bib.bib36)\), and support learnable frequency bases for spatial positional encoding\(Li et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib24)\)\. Unlike these methods, which construct Fourier\-based input features, our Random Fourier Rotations builds on RFF and extend from feature maps to rotation operators on queries and keys, preserving RoPE’s multiplicative structure and efficient implementation\.

## 3\.Methodology

### 3\.1\.Motivation

![Refer to caption](https://arxiv.org/html/2607.26369v1/decay.png)Figure 1\.Long\-term decay of RoPERoPE\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32)\)is built with 2d rotation per feature pair, which is inherently periodic\. However as proved in the original paper\(Su et al\.,[2021](https://arxiv.org/html/2607.26369#bib.bib32)\), when combining these 2d rotations with the standard log\-linear spaced frequency schedule :θi=10000−2​i/d,i∈\[0,2,…,d/2−1\]\\theta\_\{i\}=10000^\{\-2i/d\},\\quad i\\in\[0,2,\\ldots,d/2\-1\], RoPE enforces attention score upper bound decay with relative distance, exhibiting a monotonically decreasing trend with random oscillations as shown in Figure[1](https://arxiv.org/html/2607.26369#S3.F1)\. This property is termed as long\-term decay in the original paper\. Although the long\-term decay property suits the nature of language modeling, it risks oversimplifying how distances affects correlation in domains with natural periodicity, such as generative recommendation\. We empirically confirm this effect in §[4\.4\.1](https://arxiv.org/html/2607.26369#S4.SS4.SSS1)Figure[2](https://arxiv.org/html/2607.26369#S4.F2), where standard RoPE is shown to concentrate attention on recent interactions while obscuring the daily periodicity pattern\.

In sequential generative recommendation, interactions at similar hour of the day or day of the week exhibit high correlations\. An ideal frequency schedule for periodicity modeling provides higher attention score upper bound for interaction pairs at similar hour\-of\-the\-day or day\-of\-the\-week than others\.

### 3\.2\.Modulating Attention Logits via Random Fourier Rotations

Before jumping into our specific approach for periodic attention modulation, we first introduce the theoretic foundation for modulating attention scores based on continuous token positions\.

Suppose we want to modulate the standard attention logits such that the similarity between a query𝒒m\\boldsymbol\{q\}\_\{m\}at positionpmp\_\{m\}and a key𝒌n\\boldsymbol\{k\}\_\{n\}at positionpnp\_\{n\}is weighted by a positive definite kernelf:ℝ→ℝf:\\mathbb\{R\}\\to\\mathbb\{R\}withf​\(0\)=1f\(0\)=1:

\(1\)𝒒m⊤​𝒌n↝𝒒m⊤​𝒌n​f​\(pm−pn\)\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}\\leadsto\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)wherepm,pn∈ℝp\_\{m\},p\_\{n\}\\in\\mathbb\{R\}\. For example,ffcould be Gaussian:f​\(x\)=e−x2f\(x\)=e^\{\-x^\{2\}\}, Laplace:f​\(x\)=e−\|x\|f\(x\)=e^\{\-\|x\|\}or trigonometricf​\(x\)=cos⁡\(x\)f\(x\)=\\cos\(x\)\.

We claim in the following proposition that any normalized continuous positive\-definite modulation function can be unbiasedly approximated by sampling RoPE frequencies from their own Fourier transform distribution\.

###### Proposition 3\.1 \(Random Fourier Rotation Estimator\)\.

Letξ0,ξ1,…,ξd/2−1​∼i\.i\.d\.​τ​\(ξ\)\\xi\_\{0\},\\xi\_\{1\},\\ldots,\\xi\_\{d/2\-1\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\tau\(\\xi\), whereτ​\(ξ\)\\tau\(\\xi\)is the Fourier transform of a continuous positive definite kernelf:ℝ→ℝf:\\mathbb\{R\}\\to\\mathbb\{R\}withf​\(0\)=1f\(0\)=1111f​\(0\)=1f\(0\)=1is required to satisfy the normalized condition in Bochner’s theorem\(Bochner,[1933](https://arxiv.org/html/2607.26369#bib.bib2)\)\. Let𝐪m\(j\)=\[𝐪m,2​j,𝐪m,2​j\+1\]⊤∈ℝ2\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}=\[\\boldsymbol\{q\}\_\{m,2j\},\\boldsymbol\{q\}\_\{m,2j\+1\}\]^\{\\top\}\\in\\mathbb\{R\}^\{2\}and𝐤n\(j\)=\[𝐤n,2​j,𝐤n,2​j\+1\]⊤∈ℝ2\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}=\[\\boldsymbol\{k\}\_\{n,2j\},\\boldsymbol\{k\}\_\{n,2j\+1\}\]^\{\\top\}\\in\\mathbb\{R\}^\{2\}denote thejj\-th consecutive feature pair of query𝐪m∈ℝd\\boldsymbol\{q\}\_\{m\}\\in\\mathbb\{R\}^\{d\}and key𝐤n∈ℝd\\boldsymbol\{k\}\_\{n\}\\in\\mathbb\{R\}^\{d\}respectively, forj=0,…,d/2−1j=0,\\ldots,d/2\-1\. Then the Random Fourier Rotation estimator:

\(2\)g^​\(𝒒m,𝒌n,pm,pn\)=∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\\hat\{g\}\(\\boldsymbol\{q\}\_\{m\},\\boldsymbol\{k\}\_\{n\},p\_\{m\},p\_\{n\}\)=\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)
where

\(3\)𝑹m\(j\)=\[cos⁡\(2​π​ξj​pm\)−sin⁡\(2​π​ξj​pm\)sin⁡\(2​π​ξj​pm\)cos⁡\(2​π​ξj​pm\)\]\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}=\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\end\{bmatrix\}
\(4\)𝑹n\(j\)=\[cos⁡\(2​π​ξj​pn\)−sin⁡\(2​π​ξj​pn\)sin⁡\(2​π​ξj​pn\)cos⁡\(2​π​ξj​pn\)\]\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}=\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\end\{bmatrix\}
is an unbiased estimator of𝐪m⊤​𝐤n​f​\(pm−pn\)\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\), i\.e\.:

\(5\)𝔼ξ0,…,ξd/2−1∼τ​\(ξ\)​\[∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\]=𝒒m⊤​𝒌n​f​\(pm−pn\)\\mathbb\{E\}\_\{\\xi\_\{0\},\\ldots,\\xi\_\{d/2\-1\}\\sim\\tau\(\\xi\)\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\\right\]=\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)

###### Proof\.

By Bochner’s theorem\(Bochner,[1933](https://arxiv.org/html/2607.26369#bib.bib2)\), sinceffis continuous positive definite withf​\(0\)=1f\(0\)=1, its Fourier transformτ​\(ξ\)=∫ℝf​\(x\)​e−i​2​π​ξ​x​𝑑x\\tau\(\\xi\)=\\int\_\{\\mathbb\{R\}\}f\(x\)e^\{\-i2\\pi\\xi x\}dxis a proper probability density function satisfyingτ​\(ξ\)≥0\\tau\(\\xi\)\\geq 0and∫ℝτ​\(ξ\)​𝑑ξ=f​\(0\)=1\\int\_\{\\mathbb\{R\}\}\\tau\(\\xi\)d\\xi=f\(0\)=1\. This allows us to expressffas an expectation overξ\\xisimilar to the derivation in RFF\(Rahimi and Recht,[2007](https://arxiv.org/html/2607.26369#bib.bib28)\):

f​\(pm−pn\)\\displaystyle f\(p\_\{m\}\-p\_\{n\}\)=∫ℝei​2​π​ξ​\(pm−pn\)​τ​\(ξ\)​𝑑ξ\\displaystyle=\\int\_\{\\mathbb\{R\}\}e^\{i2\\pi\\xi\(p\_\{m\}\-p\_\{n\}\)\}\\tau\(\\xi\)d\\xi\(6\)=𝔼ξ∼τ​\(ξ\)​\[ei​2​π​ξ​\(pm−pn\)\]\\displaystyle=\\mathbb\{E\}\_\{\\xi\\sim\\tau\(\\xi\)\}\\left\[e^\{i2\\pi\\xi\(p\_\{m\}\-p\_\{n\}\)\}\\right\]
Grouping query and key features intod/2d/2pairs𝒒m\(j\)=\[𝒒m,2​j,𝒒m,2​j\+1\]⊤\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}=\[\\boldsymbol\{q\}\_\{m,2j\},\\boldsymbol\{q\}\_\{m,2j\+1\}\]^\{\\top\}and𝒌n\(j\)=\[𝒌n,2​j,𝒌n,2​j\+1\]⊤\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}=\[\\boldsymbol\{k\}\_\{n,2j\},\\boldsymbol\{k\}\_\{n,2j\+1\}\]^\{\\top\}, we have:

\(7\)𝒒m⊤​𝒌n​f​\(pm−pn\)=∑j=0d/2−1\(𝒒m\(j\)\)⊤​𝒌n\(j\)​f​\(pm−pn\)\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)=\\sum\_\{j=0\}^\{d/2\-1\}\\left\(\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\)^\{\\top\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}f\(p\_\{m\}\-p\_\{n\}\)
Using the complex representation of 2d vector𝒒m\(j\)↦𝒒m,2​j\+i​𝒒m,2​j\+1\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\mapsto\\boldsymbol\{q\}\_\{m,2j\}\+i\\boldsymbol\{q\}\_\{m,2j\+1\},𝒌n\(j\)↦𝒌n,2​j\+i​𝒌n,2​j\+1\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\mapsto\\boldsymbol\{k\}\_\{n,2j\}\+i\\boldsymbol\{k\}\_\{n,2j\+1\}, and the property

\(8\)𝒒m\(j\)⊤​𝒌n\(j\)=Re​\[𝒒m\(j\)​𝒌n\(j\)∗\]\\boldsymbol\{q\}\_\{m\}^\{\(j\)\\top\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}=\\mathrm\{Re\}\[\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\], where𝒌n\(j\)∗\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}is the complex conjugate of𝒌n\(j\)\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}, we arrive at:

𝒒m⊤​𝒌n​f​\(pm−pn\)\\displaystyle\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)=Re​\[∑j=0d/2−1𝒒m\(j\)​𝒌n\(j\)∗​f​\(pm−pn\)\]\\displaystyle=\\mathrm\{Re\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}f\(p\_\{m\}\-p\_\{n\}\)\\right\]\(9\)=Re​\[∑j=0d/2−1𝒒m\(j\)​𝒌n\(j\)∗​𝔼ξ∼τ​\(ξ\)​\[ei​2​π​ξ​\(pm−pn\)\]\]\\displaystyle=\\mathrm\{Re\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\\mathbb\{E\}\_\{\\xi\\sim\\tau\(\\xi\)\}\\left\[e^\{i2\\pi\\xi\(p\_\{m\}\-p\_\{n\}\)\}\\right\]\\right\]
On the other hand,

𝔼ξ0,…,ξd/2−1∼τ​\(ξ\)​\[∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\]\\displaystyle\\mathbb\{E\}\_\{\\xi\_\{0\},\\ldots,\\xi\_\{d/2\-1\}\\sim\\tau\(\\xi\)\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\\right\]=𝔼ξ0,…,ξd/2−1∼τ​\(ξ\)​\[∑j=0d/2−1Re​\[\(𝒒m\(j\)​ei​2​π​ξj​pm\)​\(𝒌n\(j\)​ei​2​π​ξj​pn\)∗\]\]\\displaystyle=\\mathbb\{E\}\_\{\\xi\_\{0\},\\ldots,\\xi\_\{d/2\-1\}\\sim\\tau\(\\xi\)\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\\mathrm\{Re\}\\left\[\\left\(\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}e^\{i2\\pi\\xi\_\{j\}p\_\{m\}\}\\right\)\\left\(\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}e^\{i2\\pi\\xi\_\{j\}p\_\{n\}\}\\right\)^\{\*\}\\right\]\\right\]\(10\)=𝔼ξ0,…,ξd/2−1∼τ​\(ξ\)​Re​\[∑j=0d/2−1𝒒m\(j\)​𝒌n\(j\)∗​ei​2​π​ξj​pm​e−i​2​π​ξj​pn\]\\displaystyle=\\mathbb\{E\}\_\{\\xi\_\{0\},\\ldots,\\xi\_\{d/2\-1\}\\sim\\tau\(\\xi\)\}\\mathrm\{Re\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}e^\{i2\\pi\\xi\_\{j\}p\_\{m\}\}e^\{\-i2\\pi\\xi\_\{j\}p\_\{n\}\}\\right\]\(11\)=Re​\[∑j=0d/2−1𝒒m\(j\)​𝒌n\(j\)∗​𝔼ξ∼τ​\(ξ\)​\[ei​2​π​ξj​pm​e−i​2​π​ξj​pn\]\]\\displaystyle=\\mathrm\{Re\}\\left\[\\sum\_\{j=0\}^\{d/2\-1\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\\mathbb\{E\}\_\{\\xi\\sim\\tau\(\\xi\)\}\\left\[e^\{i2\\pi\\xi\_\{j\}p\_\{m\}\}e^\{\-i2\\pi\\xi\_\{j\}p\_\{n\}\}\\right\]\\right\]\(12\)=𝒒m⊤​𝒌n​f​\(pm−pn\)\\displaystyle=\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)
which completes the proof\. ∎

Next, we investigate how fast Random Fourier Rotations converges as the number of feature pairs increases \.

###### Proposition 3\.2 \(Convergence of Random Fourier Rotation Estimator\)\.

Under the same setting as Proposition 3\.1, for anyϵ\>0\\epsilon\>0:

P​\(\|1d/2​∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)−1d/2​𝒒m⊤​𝒌n​f​\(pm−pn\)\|≥ϵ\)\\displaystyle P\\left\(\\left\|\\frac\{1\}\{d/2\}\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\-\\frac\{1\}\{d/2\}\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)\\right\|\\geq\\epsilon\\right\)\(13\)≤2​exp⁡\(−ϵ2​d28​∑j=0d/2−1\(‖𝒒m\(j\)‖​‖𝒌n\(j\)‖\)2\)\\displaystyle\\leq 2\\exp\\left\(\-\\frac\{\\epsilon^\{2\}d^\{2\}\}\{8\\sum\_\{j=0\}^\{d/2\-1\}\(\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|\)^\{2\}\}\\right\)

The proof of Proposition 3\.2 can be found in Appendix A\.1

In particular, when∑j=0d/2−1\(\(𝒒m\(j\)\)⊤​𝒌n\(j\)\)2\\sum\_\{j=0\}^\{d/2\-1\}\(\(\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)^\{2\}scales linearly withdd, e\.g\. at the beginning of training, Proposition 3\.2 guarantees convergence of

1d/2​∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\\frac\{1\}\{d/2\}\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)to

1d/2​𝒒m⊤​𝒌n​f​\(pm−pn\)\\frac\{1\}\{d/2\}\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)exponentially fast indd, enforcing the attention modulation priorfffrom the start of training\. See Appendix A\.2 for more details

The Random Fourier Rotations algorithm is summarized in Algorithm[1](https://arxiv.org/html/2607.26369#alg1)\.

#### 3\.2\.1\.Random Fourier Rotations for Periodic Functions

We highlight a special case for periodic functions here as their Fourier transform result in discrete fourier series, forming a probability mass function instead of a density function in the general case\.

###### Corollary 3\.3 \(Periodic Case via Herglotz’s Theorem\)\.

Letffbe a periodic continuous positive definite kernel with periodTT,f​\(0\)=1f\(0\)=1\. The complex Fourier series offfis

∑k=−∞∞αk​ei​2​π​k​x/T\\sum\_\{k=\-\\infty\}^\{\\infty\}\\alpha\_\{k\}e^\{i2\\pi kx/T\}
where

\(14\)αk=1T​∫0Tf​\(x\)​e−i​2​π​k​x/T​𝑑x=1T​∫0Tf​\(x\)​cos⁡\(2​π​k​x/T\)​𝑑x\\alpha\_\{k\}=\\frac\{1\}\{T\}\\int\_\{0\}^\{T\}f\(x\)e^\{\-i2\\pi kx/T\}\\,dx=\\frac\{1\}\{T\}\\int\_\{0\}^\{T\}f\(x\)\\cos\(2\\pi kx/T\)\\,dx
By Herglotz’s theorem\(Herglotz,[1911](https://arxiv.org/html/2607.26369#bib.bib11)\), the Fourier coefficients\{αk\}k=−∞∞\\\{\\alpha\_\{k\}\\\}\_\{k=\-\\infty\}^\{\\infty\}satisfyαk≥0\\alpha\_\{k\}\\geq 0and∑k=−∞∞αk=f​\(0\)=1\\sum\_\{k=\-\\infty\}^\{\\infty\}\\alpha\_\{k\}=f\(0\)=1, forming a valid probability mass function over the discrete harmonics\{ξk=k/T\}k=−∞∞\\\{\\xi\_\{k\}=k/T\\\}\_\{k=\-\\infty\}^\{\\infty\}\. Consequently, Propositions 3\.1 and 3\.2 apply directly with frequencies sampled from\{ξk=k/T\}k=−∞∞\\\{\\xi\_\{k\}=k/T\\\}\_\{k=\-\\infty\}^\{\\infty\}with PMFP​\(ξ\)=∑k=−∞∞αk​δ​\(ξ−k/T\)P\(\\xi\)=\\sum\_\{k=\-\\infty\}^\{\\infty\}\\alpha\_\{k\}\\delta\(\\xi\-k/T\)\.

Algorithm 1Random Fourier Rotations for Attention Modulation0:A continuous positive definite attention modulating prior

ffwith

f​\(0\)=1f\(0\)=1, embedding dimension

dd, continuous positions

pm,pn∈ℝp\_\{m\},p\_\{n\}\\in\\mathbb\{R\}
0:Rotation matrice

𝑹md\\boldsymbol\{R\}\_\{m\}^\{d\}and

𝑹nd\\boldsymbol\{R\}\_\{n\}^\{d\}such that

\(𝑹md​𝒒m\)⊤​\(𝑹nd​𝒌n\)≈𝒒m⊤​𝒌n​f​\(pm−pn\)\(\\boldsymbol\{R\}\_\{m\}^\{d\}\\boldsymbol\{q\}\_\{m\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{d\}\\boldsymbol\{k\}\_\{n\}\)\\approx\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)
1:Compute the Fourier transform of

ff:

τ​\(ξ\)=∫ℝf​\(x\)​e−i​2​π​ξ​x​𝑑x\\tau\(\\xi\)=\\int\_\{\\mathbb\{R\}\}f\(x\)e^\{\-i2\\pi\\xi x\}dx
2:Draw

d/2d/2i\.i\.d\. samples

ξ0,ξ1,…,ξd/2−1∼τ​\(ξ\)\\xi\_\{0\},\\xi\_\{1\},\\ldots,\\xi\_\{d/2\-1\}\\sim\\tau\(\\xi\)
3:For each feature pair

j=0,…,d/2−1j=0,\\ldots,d/2\-1, construct rotation matrices:

𝑹m\(j\)=\[cos⁡\(2​π​ξj​pm\)−sin⁡\(2​π​ξj​pm\)sin⁡\(2​π​ξj​pm\)cos⁡\(2​π​ξj​pm\)\]\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}=\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\end\{bmatrix\}𝑹n\(j\)=\[cos⁡\(2​π​ξj​pn\)−sin⁡\(2​π​ξj​pn\)sin⁡\(2​π​ξj​pn\)cos⁡\(2​π​ξj​pn\)\]\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}=\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\end\{bmatrix\}stack

\{𝑹m\(j\)\}j=0d/2−1\\\{\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\\}\_\{j=0\}^\{d/2\-1\}and

\{𝑹n\(j\)\}j=0d/2−1\\\{\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\\}\_\{j=0\}^\{d/2\-1\}in a block diagonal fashion to form

𝑹md\\boldsymbol\{R\}\_\{m\}^\{d\}and

𝑹nd\\boldsymbol\{R\}\_\{n\}^\{d\}respectively\.

### 3\.3\.ClockRoPE: Application in Capturing User Routines

Table 1\.ClockRoPE Sampling DistributionsDistributionsFormParameterscosine\-sym12​δ​\(ξ−1T\)\+12​δ​\(ξ\+1T\)\\frac\{1\}\{2\}\\delta\\left\(\\xi\-\\frac\{1\}\{T\}\\right\)\+\\frac\{1\}\{2\}\\delta\\left\(\\xi\+\\frac\{1\}\{T\}\\right\)no parametergaussian\-symCategorical​\(αk∑u=−ssαu\)k=−ss\\text\{Categorical\}\\left\(\\frac\{\\alpha\_\{k\}\}\{\\sum\_\{u=\-s\}^\{s\}\\alpha\_\{u\}\}\\right\)\_\{k=\-s\}^\{s\}truncation sizess, gaussian stdσ\\sigmacosine\-foldδ​\(ξ−1T\)\\delta\\left\(\\xi\-\\frac\{1\}\{T\}\\right\)no parametergaussian\-foldCategorical​\(βk∑u=0sβu\)k=0s\\text\{Categorical\}\\left\(\\frac\{\\beta\_\{k\}\}\{\\sum\_\{u=0\}^\{s\}\\beta\_\{u\}\}\\right\)\_\{k=0\}^\{s\},βk=2​αk\\beta\_\{k\}=2\\alpha\_\{k\}whenk\>0k\>0;βk=αk\\beta\_\{k\}=\\alpha\_\{k\}whenk=0k=0truncation sizess, gaussian stdσ\\sigmaTo capture user routines with period T, we consider the following two attention modulation kernel: cosine prior and periodic gaussian prior\.

#### 3\.3\.1\.Cosine Prior\.

The simplest symmetric periodic function we can come up with is cosine with periodTT\.

\(15\)f​\(t\)=c​o​s​\(2​π​tT\)\\displaystyle f\(t\)=cos\(\\frac\{2\\pi t\}\{T\}\)
Appling Corollary 3\.3 yields the binary distribution for RFR frequencies:

\(16\)P​\(ξ\)=12​δ​\(ξ−1T\)\+12​δ​\(ξ\+1T\)P\(\\xi\)=\\frac\{1\}\{2\}\\delta\\left\(\\xi\-\\frac\{1\}\{T\}\\right\)\+\\frac\{1\}\{2\}\\delta\\left\(\\xi\+\\frac\{1\}\{T\}\\right\)whereδ​\(⋅\)\\delta\(\\cdot\)denotes the Dirac delta function\.

#### 3\.3\.2\.Periodic Gaussian Prior\.

To provide more control over the temporal receptive field, we consider a periodic gaussian prior\.

\(17\)f​\(t\)=e−12​\(tσ\)2,t∈\[−T2,T2\]\\displaystyle f\(t\)=e^\{\-\\frac\{1\}\{2\}\\left\(\\frac\{t\}\{\\sigma\}\\right\)^\{2\}\},\\quad t\\in\[\-\\frac\{T\}\{2\},\\frac\{T\}\{2\}\]f​\(t\)=f​\(t\+T\)\\displaystyle f\(t\)=f\(t\+T\)
A largerσ\\sigmaproduces a broader attention peak, allowing interactions further apart in time to remain correlated, while a smallerσ\\sigmaenforces sharper, more localized attention\.

Whenσ≪T/2\\sigma\\ll T/2,ffis positive definite and we can then apply Corollary 3\.3222The periodic Gaussian priorf​\(t\)=exp⁡\(−t2/2​σ2\)f\(t\)=\\exp\(\-t^\{2\}/2\\sigma^\{2\}\)is positive definite whenσ≪T/2\\sigma\\ll T/2\. Positive definiteness requires all Fourier coefficientsαk≥0\\alpha\_\{k\}\\geq 0, which holds sinceαk=σ​2​πT​exp⁡\(−2​π2​σ2​k2/T2\)≥0\\alpha\_\{k\}=\\frac\{\\sigma\\sqrt\{2\\pi\}\}\{T\}\\exp\(\-2\\pi^\{2\}\\sigma^\{2\}k^\{2\}/T^\{2\}\)\\geq 0for allk∈ℤk\\in\\mathbb\{Z\}\. The conditionσ≪T/2\\sigma\\ll T/2ensures the periodic extension offfis smooth, i\.e\.f​\(±T/2\)=exp⁡\(−T2/8​σ2\)≈0f\(\\pm T/2\)=\\exp\(\-T^\{2\}/8\\sigma^\{2\}\)\\approx 0, so thatffbehaves as a proper Gaussian on each period\. For our choiceσ≤2\\sigma\\leq 2hr andT=24T=24hr,f​\(±12​hr\)=exp⁡\(−18\)≈10−8f\(\\pm 12\\text\{hr\}\)=\\exp\(\-18\)\\approx 10^\{\-8\}, confirming positive definiteness to numerical precision\.\. The Fourier Random Rotation matrices can be formed by sampling discrete harmonics\{k/T\}k=−∞∞\{\\\{k/T\\\}\}\_\{k=\-\\infty\}^\{\\infty\}as the rotation frequencies from probability mass functionP​\(ξ=k/T\)=αkP\(\\xi=k/T\)=\\alpha\_\{k\}, whereαk\\alpha\_\{k\}is the Fourier coefficient of harmonick/Tk/T:

αk\\displaystyle\\alpha\_\{k\}=1T​∫−T/2T/2e−12​\(tσ\)2​cos⁡\(2​π​k​t/T\)​𝑑t\\displaystyle=\\dfrac\{1\}\{T\}\\int\_\{\-T/2\}^\{T/2\}e^\{\-\\frac\{1\}\{2\}\\left\(\\frac\{t\}\{\\sigma\}\\right\)^\{2\}\}\\cos\\left\(2\\pi kt/T\\right\)dt\(18\)≈σ​2​πT​e−2​π2​σ2​k2T2\\displaystyle\\approx\\frac\{\\sigma\\sqrt\{2\\pi\}\}\{T\}e^\{\-\\frac\{2\\pi^\{2\}\\sigma^\{2\}k^\{2\}\}\{T^\{2\}\}\}=1σ∗​2​πe−12​\(kσ∗\)2,σ∗=T2​π​σ\\displaystyle=\\frac\{1\}\{\\sigma^\{\*\}\\sqrt\{2\\pi\}\}e^\{\-\\frac\{1\}\{2\}\\left\(\\frac\{k\}\{\\sigma^\{\*\}\}\\right\)^\{2\}\},\\quad\\sigma\*=\\frac\{T\}\{2\\pi\\sigma\}
The approximation in \([3\.3\.2](https://arxiv.org/html/2607.26369#S3.Ex12)\) comes from extending the integration limits\[−T/2,T/2\]\[\-T/2,T/2\]to\(−∞,∞\)\(\-\\infty,\\infty\), valid whenσ≪T/2\\sigma\\ll T/2\.333Forσ=2\\sigma=2hr andT=24T=24hr, the boundary valuef​\(±T/2\)=exp⁡\(−18\)≈10−8f\(\\pm T/2\)=\\exp\(\-18\)\\approx 10^\{\-8\}confirms this approximation is exact to numerical precision\.The coefficients decay exponentially ink2k^\{2\}, guaranteeing rapid convergence of the truncated series and justifying the use of a small truncation sizes≪∞s\\ll\\inftyin practice\.

With truncation to\{k/T\}k=−ss\{\\\{k/T\\\}\}\_\{k=\-s\}^\{s\}, we re\-normalize the probability mass function, sample:

\(19\)k0\(ℓ\),…,kd/2−1\(ℓ\)∼Categorical​\(αk∑u=−ssαu\)k=−ss,ℓ=1,…,Lk\_\{0\}^\{\(\\ell\)\},\\ldots,k\_\{d/2\-1\}^\{\(\\ell\)\}\\sim\\text\{Categorical\}\\left\(\\frac\{\\alpha\_\{k\}\}\{\\sum\_\{u=\-s\}^\{s\}\\alpha\_\{u\}\}\\right\)\_\{k=\-s\}^\{s\},\\quad\\ell=1,\\ldots,Land setξj\(ℓ\)←kj\(ℓ\)/T\\xi\_\{j\}^\{\(\\ell\)\}\\leftarrow k\_\{j\}^\{\(\\ell\)\}/Tforj=0,…,d/2−1j=0,\\ldots,d/2\-1in each attention layerll\.

We compare different truncation sizessand varianceσ\\sigmain ablation study §[4\.3\.1](https://arxiv.org/html/2607.26369#S4.SS3.SSS1)\.

#### 3\.3\.3\.Direction Awareness with Folded Frequency Distribution

The frequency distributions we have derived so far from cosine prior and periodic gaussian prior are all symmetric\. In this section, we argue that sampling from folded frequency distributions maintains the modulation effect as their symmetric counterparts and has additional benefit of directional awareness\. To fold a distribution, we perform

\(20\)Pfold​\(x\)=\{2​P​\(x\)for​x\>0P​\(x\)for​x=00otherwiseP\_\{\\text\{fold\}\}\(x\)=\\begin\{cases\}2P\(x\)&\\text\{for \}x\>0\\\\ P\(x\)&\\text\{for \}x=0\\\\ 0&\\text\{otherwise\}\\end\{cases\}Table[1](https://arxiv.org/html/2607.26369#S3.T1)shows all the distributions we have discussed so far and their folded versions\.

We first show that folded distributions maintain the modulation effect of the symmetric ones\. We have the following expression for attention logit modulation:

\(21\)\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\\displaystyle\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\(22\)=\(\[cos⁡\(2​π​ξj​pm\)−sin⁡\(2​π​ξj​pm\)sin⁡\(2​π​ξj​pm\)cos⁡\(2​π​ξj​pm\)\]​\[qm,2​jqm,2​j\+1\]\)T\\displaystyle=\(\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{m\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{m\}\)\\end\{bmatrix\}\\begin\{bmatrix\}q\_\{m,2j\}\\\\ q\_\{m,2j\+1\}\\end\{bmatrix\}\)^\{T\}\(23\)\[cos⁡\(2​π​ξj​pn\)−sin⁡\(2​π​ξj​pn\)sin⁡\(2​π​ξj​pn\)cos⁡\(2​π​ξj​pn\)\]​\[kn,2​jkn,2​j\+1\]\\displaystyle\\quad\\quad\\begin\{bmatrix\}\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\-\\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\\\ \\sin\(2\\pi\\xi\_\{j\}p\_\{n\}\)&\\cos\(2\\pi\\xi\_\{j\}p\_\{n\}\)\\end\{bmatrix\}\\begin\{bmatrix\}k\_\{n,2j\}\\\\ k\_\{n,2j\+1\}\\end\{bmatrix\}\(24\)=\(qm,2​j​kn,2​j\+qm,2​j\+1​kn,2​j\+1\)​cos⁡\(2​π​ξj​\(pm−pn\)\)\\displaystyle=\(q\_\{m,2j\}k\_\{n,2j\}\+q\_\{m,2j\+1\}k\_\{n,2j\+1\}\)\\cos\(2\\pi\\xi\_\{j\}\(p\_\{m\}\-p\_\{n\}\)\)\(25\)\+\(qm,2​j​kn,2​j\+1−qm,2​j\+1​kn,2​j\)​sin⁡\(2​π​ξj​\(pm−pn\)\)\\displaystyle\\quad\+\(q\_\{m,2j\}k\_\{n,2j\+1\}\-q\_\{m,2j\+1\}k\_\{n,2j\}\)\\sin\(2\\pi\\xi\_\{j\}\(p\_\{m\}\-p\_\{n\}\)\)
Similar to RoPE, ClockRoPE is designed to modulate the upper bound of attention logits: lowering the attention logit upper bound at other times so that through softmax, more attention can be concentrated to tokens during similar hour of the day or day of the week\. Attention logits get close to the upper bound when query and key are roughly aligned\.

In this case, we have\|qm,2​j​kn,2​j\+1−qm,2​j\+1​kn,2​j\|≪\|qm,2​j​kn,2​j\+qm,2​j\+1​kn,2​j\+1\|\|q\_\{m,2j\}k\_\{n,2j\+1\}\-q\_\{m,2j\+1\}k\_\{n,2j\}\|\\ll\|q\_\{m,2j\}k\_\{n,2j\}\+q\_\{m,2j\+1\}k\_\{n,2j\+1\}\|in \(24\), the dominating cosine term does not depend on the sign of the frequencies\. We can thus replace all negative frequencies with its reverse and maintain the signal modulating effect\.

Sampling from folded non\-negative frequencies has additional benefits of allowing the model to be directional aware and ensuring all feature pairs share a consistent rotational orientation\. This consistency could help the projection matrices learn a unified representation of temporal causality\.

Algorithm 2ClockRoPE for Routine Modeling1:// Network initialization: Sample per\-layer frequencies

2:foreach attention layer

ℓ=1,…,L\\ell=1,\\ldots,Ldo

3:Set

s​e​e​dℓ←s​e​e​db​a​s​e\+ℓseed\_\{\\ell\}\\leftarrow seed\_\{base\}\+\\ell
4:Draw

d/2d/2i\.i\.d\. samples with

s​e​e​dℓseed\_\{\\ell\}:

ξ1\(ℓ\),…,ξd/2\(ℓ\)∼ClockRoPE Sampling Distribution\\xi\_\{1\}^\{\(\\ell\)\},\\ldots,\\xi\_\{d/2\}^\{\(\\ell\)\}\\sim\\text\{ClockRoPE Sampling Distribution\}
5:Form rotation frequency vector

Θ\(ℓ\)←\(ξ1\(ℓ\),ξ2\(ℓ\),…,ξd/2\(ℓ\)\)\\Theta^\{\(\\ell\)\}\\leftarrow\\left\(\\xi\_\{1\}^\{\(\\ell\)\},\\xi\_\{2\}^\{\(\\ell\)\},\\ldots,\\xi\_\{d/2\}^\{\(\\ell\)\}\\right\)
6:endfor

7:// Forward pass: apply ClockRoPE rotation \(RoPE\-style efficient realization\)

8:foreach layer

ℓ=1,…,L\\ell=1,\\ldots,Ldo

9:Compute rotation angles

2​π​Θ\(ℓ\)​pm,2​π​Θ\(ℓ\)​pn2\\pi\\Theta^\{\(\\ell\)\}p\_\{m\},2\\pi\\Theta^\{\(\\ell\)\}p\_\{n\}and expand them to

rm\(ℓ\),rn\(ℓ\)∈ℝdr^\{\(\\ell\)\}\_\{m\},r^\{\(\\ell\)\}\_\{n\}\\in\\mathbb\{R\}^\{d\}by repeating each element twice\.

10:Define

rotate\_pair​\(𝒗\)←\(−v2,v1,−v4,v3,…,−vd,vd−1\)\\text\{rotate\\\_pair\}\(\\boldsymbol\{v\}\)\\leftarrow\(\-v\_\{2\},v\_\{1\},\-v\_\{4\},v\_\{3\},\\ldots,\-v\_\{d\},v\_\{d\-1\}\)
11:

𝒒m\(ℓ\)←𝒒m\(ℓ\)⊗cos⁡\(rm\(ℓ\)\)\+rotate\_pair​\(𝒒m\(ℓ\)\)⊗sin⁡\(rm\(ℓ\)\)\\boldsymbol\{q\}\_\{m\}^\{\(\\ell\)\}\\leftarrow\\boldsymbol\{q\}\_\{m\}^\{\(\\ell\)\}\\otimes\\cos\\left\(r^\{\(\\ell\)\}\_\{m\}\\right\)\+\\text\{rotate\\\_pair\}\\left\(\\boldsymbol\{q\}\_\{m\}^\{\(\\ell\)\}\\right\)\\otimes\\sin\\left\(r^\{\(\\ell\)\}\_\{m\}\\right\)
12:

𝒌n\(ℓ\)←𝒌n\(ℓ\)⊗cos⁡\(rn\(ℓ\)\)\+rotate\_pair​\(𝒌n\(ℓ\)\)⊗sin⁡\(rn\(ℓ\)\)\\boldsymbol\{k\}\_\{n\}^\{\(\\ell\)\}\\leftarrow\\boldsymbol\{k\}\_\{n\}^\{\(\\ell\)\}\\otimes\\cos\\left\(r^\{\(\\ell\)\}\_\{n\}\\right\)\+\\text\{rotate\\\_pair\}\\left\(\\boldsymbol\{k\}\_\{n\}^\{\(\\ell\)\}\\right\)\\otimes\\sin\\left\(r^\{\(\\ell\)\}\_\{n\}\\right\)
13:endfor

#### 3\.3\.4\.Combine Multiple Distributions

To capture multiple periods e\.g\. daily and weekly, we can \(1\) combine ClockRoPE sampling distributions with a weighted sum of respective pmf or \(2\) shard features based on the importance of each periodicity and apply corresponding sampling distribution to each feature group\. For simplicity, we took the second approach in our experiments\.

The ClockRoPE algorithm is summarized in Algorithm[2](https://arxiv.org/html/2607.26369#alg2)\. In practice, we sample ClockRoPE frequencies separately for each head in multi\-head attention\.

## 4\.Experiments

In this section, we evaluate ClockRoPE on production\-scale generative retrieval task at a major video\-sharing platform\. We experiment with ClockRoPE and other baseline methods on top of a transformer\-style base model, which predicts users’ next interaction given their historic interaction sequences\. This foundation model powers top\-k retrieval from a massive video corpus on both watch page and home page, and serves as the upstream of the more fine\-grained ranking model\.

We aim to answer the following questions through experiments:

1. \(1\)How does ClockRoPE affect attention distribution ?
2. \(2\)Does the attention distribution change benefit viewer satisfaction?
3. \(3\)How does ClockRoPE interact with RoPE ?

### 4\.1\.Offline Evaluation

Table 2\.Experiment group 1: no position embedding\. Percentage MAP improvement over control\.MethodMAP@1MAP@50Arm 1 \(Fourier Features\)\+0\.43%\+0\.29%Arm 2 \(ClockRoPE\-cosine\-sym\)\+1\.89%\+1\.25%Arm 3 \(ClockRoPE\-cosine\-fold\)\+1\.96%\+1\.37%Arm 4 \(ClockRoPE\-gaussian\-sym\)\+3\.19%\+2\.12%Arm 5 \(ClockRoPE\-gaussian\-fold\)\+3\.61%\+2\.25%Table 3\.Experiment group 2: combining with RoPE\. Percentage MAP improvement over RoPE only control\.MethodMAP@1MAP@50Arm 1 \(ClockRoPE\-cosine\-fold\)\+1\.02%\+1\.17%Arm 2 \(ClockRoPE\-gaussian\-sym\)\+2\.18%\+1\.91%Arm 3 \(ClockRoPE\-gaussian\-fold\)\+2\.39%\+2\.00%We conducted two groups of experiments to investigate ClockRoPE’s effectiveness with and without RoPE\.

#### 4\.1\.1\.Experiment Group 1

All the variants in this group use absolute timestamp embedding and no position embedding\. For ClockRoPE\-gaussian methods, we used parameter sweep to select truncation size and variance\.

- •Control:Hour\-of\-the\-day and day\-of\-the\-week input features :\[tmodTd,⌊tTd⌋modTwTd\]\[t\\bmod T\_\{d\},\\quad\\left\\lfloor\\frac\{t\}\{T\_\{d\}\}\\right\\rfloor\\bmod\\frac\{T\_\{w\}\}\{T\_\{d\}\}\]\. This is the production baseline\.
- •Arm 1 \(Fourier Features\):Daily and weekly Fourier input features \[cos⁡\(2​π​tTd\),sin⁡\(2​π​tTd\),cos⁡\(2​π​tTw\),sin⁡\(2​π​tTw\)\]\[\\cos\(\\frac\{2\\pi t\}\{T\_\{d\}\}\),\\sin\(\\frac\{2\\pi t\}\{T\_\{d\}\}\),\\cos\(\\frac\{2\\pi t\}\{T\_\{w\}\}\),\\sin\(\\frac\{2\\pi t\}\{T\_\{w\}\}\)\]\. This periodic encoding has the nice property of translation invariance as the cosine difference formula ensures that dot product between two such feature vectors depends only on the relative temporal distance\.
- •Arm 2 \(ClockRoPE\-cosine\-sym\):We divide all features into two halves and apply ClockRoPE\-cosine\-sym with daily period and weekly period to each half respectively\.
- •Arm 3 \(ClockRoPE\-cosine\-fold\):Same as Arm 2 except for using cosine\-fold sampling distribution\.
- •Arm 4 \(ClockRoPE\-gaussian\-sym\):Same as Arm 2 except for using gaussian\-sym sampling distribution\.
- •Arm 5 \(ClockRoPE\-gaussian\-fold\):Same as Arm 2 except for using gaussian\-fold sampling distribution\.

As shown in Table[2](https://arxiv.org/html/2607.26369#S4.T2), ClockRoPE\-gaussian variants consistently outperform ClockRoPE\-cosine variants\. A potential explanation lies in the range of the two priors: the gaussian prior is non\-negative and decays smoothly toward0as temporal distancetmodTt\\bmod TapproachesT/2T/2\. The cosine prior, in contrast, ranges over\[−1,1\]\[\-1,1\]and reaches−1\-1att=T/2t=T/2, actively inverting the sign of the attention logit for maximally out\-of\-phase pairs rather than merely suppressing it\. This sign inversion can inject additional noise into the attention distribution\.

We selected the best performing variants from group 1 and experimented with combining them with RoPE in the next section\.

#### 4\.1\.2\.Experiment Group 2

In the second group, we evaluated how ClockRoPE interacts with RoPE\. All the variants in this group use both absolute timestamp embedding and standard RoPE for position embedding\. We combined ClockRoPE with RoPE by applying RoPE to half of the hidden features and ClockRoPE to the other half\. Other ways of combining RoPE and ClockRoPE are discussed in Ablation §[4\.3\.1](https://arxiv.org/html/2607.26369#S4.SS3.SSS1)\.

- •Control:Hour\-of\-the\-day and day\-of\-the\-week input features :\[tmodTd,⌊tTd⌋modTwTd\]\[t\\bmod T\_\{d\},\\quad\\left\\lfloor\\frac\{t\}\{T\_\{d\}\}\\right\\rfloor\\bmod\\frac\{T\_\{w\}\}\{T\_\{d\}\}\]with standard RoPE \.
- •Arm 1 \(ClockRoPE\-cosine\-fold\):We divide all features into two halves and apply RoPE to the first half\. We further divide the second half evenly into two feature groups and apply ClockRoPE\-cosine\-fold with daily period and weekly period to each group respectively\.
- •Arm 2 \(ClockRoPE\-gaussian\-sym\):Same as Arm 1 except for using gaussian\-sym sampling distribution\.
- •Arm 3 \(ClockRoPE\-gaussian\-fold\):Same as Arm 1 except for using gaussian\-fold sampling distribution\.

Table[2](https://arxiv.org/html/2607.26369#S4.T2)and Table[3](https://arxiv.org/html/2607.26369#S4.T3)show that ClockRoPE\-gaussian\-fold performs the best both with and without RoPE\.

### 4\.2\.Online A/B Tests

Table 4\.RoPE and ClockRoPE each independently improves sitewide valued engagement\. The best result is achieved by combining RoPE and ClockRoPE\.MethodEngagementValued EngagementRoPE Only\+0\.04%\+0\.06%ClockRoPE Only\+0\.03%\+0\.05%RoPE \+ ClockRoPE\+0\.08%\+0\.08%We conducted an online A/B test for 14 days, assigning1%1\\%total traffic to each variant\. In this study, we used the best config from offline evaluations: ClockRoPE\-gaussian\-fold, where ClockRoPE and RoPE are combined in the same way as in §[4\.1\.2](https://arxiv.org/html/2607.26369#S4.SS1.SSS2), each applying to half of the features\.

As shown in Table[4](https://arxiv.org/html/2607.26369#S4.T4), RoPE and ClockRoPE are complementary to each other:

- •Applying RoPE or ClockRoPE alone each helps improve sitewide valued engagement\.
- •The best result is achieved by combining RoPE and ClockRoPE, leveraging both long\-term attention decay of RoPE and periodicity\-awareness of ClockRoPE\.

More analysis on how ClockRoPE interacts with RoPE in terms of attention distribution can be found in §[4\.4\.1](https://arxiv.org/html/2607.26369#S4.SS4.SSS1)\.

### 4\.3\.Ablation Studies

#### 4\.3\.1\.Truncation Size and Variance in ClockRoPE Gaussian

We ablated ClockRoPE\-gaussian\-fold with single periodT=24T=24hr on both truncation sizessand variance\. The results are shown in Table[5](https://arxiv.org/html/2607.26369#S4.T5)\. We see that tuning variance has a larger impact than truncation size\. When the truncation size is large enough e\.g\. 6 in our case, keep increasing it does not yield extra performance gain\.

Table 5\.Percentage MAP improvement over prod control with varying truncation size and variance\.MethodMAP@1MAP@50s=6s=6,σ=1\.0\\sigma=1\.0hr\+0\.95%\+0\.65%s=6s=6,σ=1\.5\\sigma=1\.5hr\+1\.51%\+0\.85%𝒔=𝟔\\boldsymbol\{s=6\},σ=2\.0\\boldsymbol\{\\sigma=2\.0\}hr\+2\.12%\+2\.05%s=6s=6,σ=2\.5\\sigma=2\.5hr\+1\.95%\+1\.84%s=3s=3,σ=2\.0\\sigma=2\.0hr\+1\.59%\+1\.12%s=9s=9,σ=2\.0\\sigma=2\.0hr\+1\.88%\+1\.41%s=12s=12,σ=2\.0\\sigma=2\.0hr\+2\.01%\+1\.94%
#### 4\.3\.2\.Different Ways of Combining RoPE and ClockRoPE

We compared two approaches:

- •Feature\-wise division: rotate half hidden features with RoPE and the other half with ClockRoPE\.
- •Head\-wise division: rotate all features in half heads with RoPE and features in the other half heads with ClockRoPE\.

Table[6](https://arxiv.org/html/2607.26369#S4.T6)shows that combining RoPE and ClockRoPE feature\-wise performs slightly better in offline evaluation\.

Table 6\.Percentage MAP improvement over RoPE only control\. Feature\-wise division performs slightly better than Head\-wise division\.MethodMAP@1MAP@50Feature\-wise division\+2\.39%\+2\.00%Head\-wise division\+1\.94%\+1\.87%

### 4\.4\.Visualization of Temporal Attention Distribution

#### 4\.4\.1\.How ClockRoPE Interacts with RoPE

![Refer to caption](https://arxiv.org/html/2607.26369v1/attn.png)Figure 2\.Visualization of average attention probability over a week\. Key observations: \(2\) Standard RoPE allocates more attention to the most recent history and obscures daily period\. \(3\) Applying ClockRoPE alone sharpens attention peak around every 24 hrs\. \(4\) The combination of RoPE and ClockRoPE retains both long\-term decay and clear daily period\.Figure[2](https://arxiv.org/html/2607.26369#S4.F2)shows the attention distribution over temporal distances ranging from 0 to 168hr, averaging attention probabilities for every hour bucket\. In this comparison, we used ClockRoPE\-gaussian\-fold with daily period only\.

Figure[2](https://arxiv.org/html/2607.26369#S4.F2)subgraph 2 confirms the motivation in §[3\.1](https://arxiv.org/html/2607.26369#S3.SS1): the log\-linear frequency schedule’s long\-term decay dominates the attention pattern, suppressing the periodic signal that ClockRoPE is designed to recover\. Subgraph 4 visually verifies that both long\-term decay and clear daily period are retained with the combination of RoPE and ClockRoPE, which explains why the combination achieves the best online performance gain in §[4\.2](https://arxiv.org/html/2607.26369#S4.SS2)\.

We do not highlight the difference between Gaussian\-symmetric and Gaussian\-fold here as they produce similar pattern at the hourly level\.

#### 4\.4\.2\.Symmetric vs\. Folded ClockRoPE Frequency Distribution

![Refer to caption](https://arxiv.org/html/2607.26369v1/week.png)Figure 3\.Visualization of average attention probability over six weeks\. Key observations: \(1\) ClockRoPE introduces weekly attention peak that is not explicit in the baseline\. \(2\) With ClockRoPE\-gaussian\-fold, more attention is allocated to the day immediate before the weekly cycle mark than the day immediate after \(3\) The model with ClockRoPE\-gaussian\-symmetric allocates attention much more evenly between the two days surrounding the weekly cycle mark\.When averaging attention probabilities for every daily bucket as in Figure[3](https://arxiv.org/html/2607.26369#S4.F3), we start to observe difference between folded frequency distribution and its symmetric counterpart\. As discussed in §[3\.3\.3](https://arxiv.org/html/2607.26369#S3.SS3.SSS3), the folded frequency distribution allows the model to distinguish between interactions one day before the weekly period mark and interactions one day after by performing ClockRoPE rotations in different directions\. In contrast, interactions before and after the weekly period mark result in the same rotations with ClockRoPE\-gaussian\-symmetric as long as their temporal distance to the weekly period mark is the same\. Figure[3](https://arxiv.org/html/2607.26369#S4.F3)subgraph 2 suggests that with the advantage of being directional\-aware, the model learns to allocate more attention to interactions before it than those after it when projecting all interactions onto the same weekly circle e\.g\. an interaction on Monday distributes more attention to interactions on the past Sunday \(8 days ago\) than those on the past Tuesday \(6 days ago\)\. This explains that a potential reason for the performance improvement of ClockRoPE\-gaussian\-fold over its symmetric counterpart is its capability of capturing routines with causal relationships\.

### 4\.5\.Production Deployment Results

Table 7\.Core Deployment MetricsCore Deployment MetricsResultsValued Engagement Time\+0\.08%Homepage Triggered Engagement Time\+0\.11%Homepage Views\-0\.51%Daily Unique Impressed Videos\-0\.19%TPU Serving Cost\-0\.63%Serving LatencyNeutralTable 8\.Routine\-related MetricsRoutine\-related MetricsOnline LiftEngaged Vod Topics\+0\.05%Repeat Engaged Topics\+0\.03%Gaming\-Related Content Valued Engagement\+0\.08%News Valued Engagement\+0\.17%Learning Content Valued Engagement\+0\.15%ClockRoPE has been successfully deployed together with RoPE across homepage and watch page at a major video\-sharing platform with neutral serving latency impact\. As shown in Table[7](https://arxiv.org/html/2607.26369#S4.T7), we observe fewer daily unique impressed videos and homepage views, alongside more homepage triggered engagement time and overall valued engagement time, suggesting that it now takes users fewer scrolls and clicks to find satisfying contents, which in turn reduces serving volume and saves overall TPU serving cost by0\.63%0\.63\\%\.

In addition to the core deployment metrics, Table[8](https://arxiv.org/html/2607.26369#S4.T8)demonstrates evidence that ClockRoPE helps improve sitewide valued engagement of routine\-related watch behavior, such as learning, gaming and news watching, tying back to our initial motivation of improving routine\-awareness in sequential recommendation\.

## 5\.Conclusion

In this work, we introduced ClockRoPE, a mathematically grounded periodic encoding for modeling user routines\. By leveraging the framework of Random Fourier Rotations, ClockRoPE achieves performance gain without requiring additional input features or increasing model dimensionality\. Crucially, ClockRoPE serves as a powerful complement to standard RoPE, enabling transformers to capture natural behavioral periodicity alongside monotonic recency decay\. The successful deployment of ClockRoPE in a major video\-sharing platform, yielding consistent valued engagement gains with resource savings, confirms its effectiveness and efficiency for large\-scale industrial applications\.

## References

- \(1\)
- Bochner \(1933\)Salomon Bochner\. 1933\.*Monotone Funktionen, Stieltjessche Integrale und harmonische Analyse*\.Springer\.
- Brown et al\.\(2020\)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al\.2020\.Language models are few\-shot learners\.*Advances in neural information processing systems*33 \(2020\), 1877–1901\.
- Chai et al\.\(2025\)Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al\.2025\.Longer: Scaling up long sequence modeling in industrial recommenders\. In*Proceedings of the Nineteenth ACM Conference on Recommender Systems*\. 247–256\.
- Cheng et al\.\(2026\)Hailing Cheng, Daqi Sun, and Xinyu Lu\. 2026\.Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling\.*arXiv preprint arXiv:2604\.24717*\(2026\)\.
- Choromanski et al\.\(2024\)Krzysztof Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamás Sarlós, Thomas Weingarten, and Adrian Weller\. 2024\.Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers\. In*International Conference on Artificial Intelligence and Statistics, 2\-4 May 2024, Palau de Congressos, Valencia, Spain**\(Proceedings of Machine Learning Research, Vol\. 238\)*, Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li \(Eds\.\)\. PMLR, 2278–2286\.[https://proceedings\.mlr\.press/v238/choromanski24a\.html](https://proceedings.mlr.press/v238/choromanski24a.html)
- Choromanski et al\.\(2022\)Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Sehanobish, Valerii Likhosherstov, Jack Parker\-Holder, Tamás Sarlós, Adrian Weller, and Thomas Weingarten\. 2022\.From block\-Toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked Transformers\. In*International Conference on Machine Learning, ICML 2022, 17\-23 July 2022, Baltimore, Maryland, USA**\(Proceedings of Machine Learning Research, Vol\. 162\)*, Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato \(Eds\.\)\. PMLR, 3962–3983\.[https://proceedings\.mlr\.press/v162/choromanski22a\.html](https://proceedings.mlr.press/v162/choromanski22a.html)
- Choromanski et al\.\(2021\)Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J\. Colwell, and Adrian Weller\. 2021\.Rethinking Attention with Performers\. In*9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\-7, 2021*\. OpenReview\.net\.[https://openreview\.net/forum?id=Ua6zuk0WRH](https://openreview.net/forum?id=Ua6zuk0WRH)
- Devlin et al\.\(2018\)Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\. 2018\.BERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding\.*arXiv preprint arXiv:1810\.04805*\(2018\)\.
- Heo et al\.\(2024\)Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun\. 2024\.Rotary Position Embedding for Vision Transformer\. In*Computer Vision \- ECCV 2024 \- 18th European Conference, Milan, Italy, September 29\-October 4, 2024, Proceedings, Part X**\(Lecture Notes in Computer Science, Vol\. 15068\)*, Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol \(Eds\.\)\. Springer, 289–305\.[doi:10\.1007/978\-3\-031\-72684\-2\_17](https://doi.org/10.1007/978-3-031-72684-2_17)
- Herglotz \(1911\)Gustav Herglotz\. 1911\.Über Potenzreihen mit positivem, reellem Teil im Einheitskreis\.*Berichte über die Verhandlungen der Königlich Sächsischen Gesellschaft der Wissenschaften zu Leipzig*63 \(1911\), 501–511\.
- Hoffmann et al\.\(2022\)Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al\.2022\.Training compute\-optimal large language models\.*arXiv preprint arXiv:2203\.15556*\(2022\)\.
- Hou et al\.\(2026\)Bojian Hou, Xiaolong Liu, Xiaoyi Liu, Jiaqi Xu, Yasmine Badr, Mengyue Hang, Sudhanshu Chanpuriya, Junqing Zhou, Yuhang Yang, Han Xu, et al\.2026\.Kunlun: Establishing scaling laws for massive\-scale recommendation systems through unified architecture design\.*arXiv preprint arXiv:2602\.10016*\(2026\)\.
- Hua et al\.\(2024\)Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Youbang Sun, Yuchen Fan, Xuekai Zhu, Biqing Qi, Ning Ding, and Bowen Zhou\. 2024\.Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization\.*arXiv preprint arXiv:2412\.17739*\(2024\)\.
- Huang et al\.\(2019\)Cheng\-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M\. Dai, Matthew D\. Hoffman, Monica Dinculescu, and Douglas Eck\. 2019\.Music Transformer: Generating Music with Long\-Term Structure\. In*7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6\-9, 2019*\. OpenReview\.net\.[https://openreview\.net/forum?id=rJe4ShAcF7](https://openreview.net/forum?id=rJe4ShAcF7)
- Kang and McAuley \(2018\)Wang\-Cheng Kang and Julian McAuley\. 2018\.Self\-attentive sequential recommendation\. In*2018 IEEE international conference on data mining \(ICDM\)*\. IEEE, 197–206\.
- Kaplan et al\.\(2020\)Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei\. 2020\.Scaling laws for neural language models\.*arXiv preprint arXiv:2001\.08361*\(2020\)\.
- Kazemi et al\.\(2019\)Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker\. 2019\.Time2vec: Learning a vector representation of time\.*arXiv preprint arXiv:1907\.05321*\(2019\)\.
- Kim et al\.\(2026\)Byeongchan Kim, Arijit Sehanobish, Avinava Dubey, Min\-hwan Oh, and Krzysztof Choromanski\. 2026\.RelFlexformer: Efficient Attention 3D\-Transformers for Integrable Relative Positional Encodings\.*CoRR*abs/2605\.10706 \(2026\)\.arXiv:2605\.10706[doi:10\.48550/ARXIV\.2605\.10706](https://doi.org/10.48550/ARXIV.2605.10706)
- Lee\-Thorp et al\.\(2021\)James Lee\-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón\. 2021\.FNet: Mixing Tokens with Fourier Transforms\.*arXiv preprint arXiv:2105\.03824*\(2021\)\.
- Li et al\.\(2026\)Boyang Li, Yulin Wu, Sizhe Xu, Nuoxian Huang, Zhonghang Yuan, Shangyi Guo, Shu Yang, and Takahiro Yabe\. 2026\.nD\-RoPE: A Generalized RoPE for n\-Dimensional Position Embedding\.*arXiv preprint arXiv:2606\.12146*\(2026\)\.
- Li et al\.\(2020\)Jiacheng Li, Yujie Wang, and Julian McAuley\. 2020\.Time interval aware self\-attention for sequential recommendation\. In*Proceedings of the 13th international conference on web search and data mining*\. 322–330\.
- Li et al\.\(2023\)Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli\. 2023\.Functional Interpolation for Relative Positions Improves Long Context Transformers\.*CoRR*abs/2310\.04418 \(2023\)\.arXiv:2310\.04418[doi:10\.48550/ARXIV\.2310\.04418](https://doi.org/10.48550/ARXIV.2310.04418)
- Li et al\.\(2021\)Yang Li, Si Si, Gang Li, Cho\-Jui Hsieh, and Samy Bengio\. 2021\.Learnable fourier features for multi\-dimensional spatial positional encoding\.*Advances in Neural Information Processing Systems*34 \(2021\), 15816–15829\.
- Luo et al\.\(2021\)Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, and Tie\-Yan Liu\. 2021\.Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding\. In*Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6\-14, 2021, virtual*, Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N\. Dauphin, Percy Liang, and Jennifer Wortman Vaughan \(Eds\.\)\. 22795–22807\.[https://proceedings\.neurips\.cc/paper/2021/hash/c0f168ce8900fa56e57789e2a2f2c9d0\-Abstract\.html](https://proceedings.neurips.cc/paper/2021/hash/c0f168ce8900fa56e57789e2a2f2c9d0-Abstract.html)
- Peebles and Xie \(2023\)William Peebles and Saining Xie\. 2023\.Scalable diffusion models with transformers\. In*Proceedings of the IEEE/CVF international conference on computer vision*\. 4195–4205\.
- Raffel et al\.\(2020\)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J\. Liu\. 2020\.Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.*J\. Mach\. Learn\. Res\.*21 \(2020\), 140:1–140:67\.[https://jmlr\.org/papers/v21/20\-074\.html](https://jmlr.org/papers/v21/20-074.html)
- Rahimi and Recht \(2007\)Ali Rahimi and Benjamin Recht\. 2007\.Random features for large\-scale kernel machines\.*Advances in neural information processing systems*20 \(2007\)\.
- Reid et al\.\(2026\)Isaac Reid, Arijit Sehanobish, Cederik Höfs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Choromanski, Richard E\. Turner, and Petar Velickovic\. 2026\.Wavelet\-Induced Rotary Encodings: RoPE Meets Graphs\.*ICML*abs/2509\.22259 \(2026\)\.arXiv:2509\.22259[doi:10\.48550/ARXIV\.2509\.22259](https://doi.org/10.48550/ARXIV.2509.22259)
- Schenck et al\.\(2025\)Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Kumar Avinava Dubey, Ayzaan Wahid, Sumeet Singh, René Wagner, Tianli Ding, Chuyuan Fu, Arunkumar Byravan, Jake Varley, Alexey A\. Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson, Vikas Sindhwani, and Krzysztof Marcin Choromanski\. 2025\.Learning the RoPEs: Better 2D and 3D Position Encodings with STRING\. In*Forty\-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13\-19, 2025**\(Proceedings of Machine Learning Research, Vol\. 267\)*, Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste\-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu \(Eds\.\)\. PMLR / OpenReview\.net\.[https://proceedings\.mlr\.press/v267/schenck25a\.html](https://proceedings.mlr.press/v267/schenck25a.html)
- Shaw et al\.\(2018\)Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani\. 2018\.Self\-Attention with Relative Position Representations\. In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL\-HLT, New Orleans, Louisiana, USA, June 1\-6, 2018, Volume 2 \(Short Papers\)*, Marilyn A\. Walker, Heng Ji, and Amanda Stent \(Eds\.\)\. Association for Computational Linguistics, 464–468\.[doi:10\.18653/V1/N18\-2074](https://doi.org/10.18653/V1/N18-2074)
- Su et al\.\(2021\)Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu\. 2021\.RoFormer: Enhanced Transformer with Rotary Position Embedding\.*arXiv preprint arXiv:2104\.09864*\(2021\)\.
- Tancik et al\.\(2020\)Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich\-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng\. 2020\.Fourier features let networks learn high frequency functions in low dimensional domains\.*Advances in neural information processing systems*33 \(2020\), 7537–7547\.
- Vaswani et al\.\(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.Attention is all you need\. In*Advances in Neural Information Processing Systems \(NeurIPS\)*, Vol\. 30\.
- Wei et al\.\(2025\)Xiaokai Wei, Jiajun Wu, Daiyao Yi, Reza Shirkavand, and Michelle Gong\. 2025\.Rotate Both Ways: Time\-and\-Order RoPE for Generative Recommendation\.*arXiv preprint arXiv:2510\.20455*\(2025\)\.
- Xu et al\.\(2019\)Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan\. 2019\.Self\-attention with functional time representation learning\.*Advances in neural information processing systems*32 \(2019\)\.
- Zhai et al\.\(2024\)Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al\.2024\.Actions speak louder than words: Trillion\-parameter sequential transducers for generative recommendations\.*arXiv preprint arXiv:2402\.17152*\(2024\)\.

## Appendix AProofs

### A\.1\.Proof of Proposition 3\.2

###### Proof\.

Define the real\-valued function:

h​\(ξ0,…,ξd/2−1\)\\displaystyle h\(\\xi\_\{0\},\\ldots,\\xi\_\{d/2\-1\}\)=∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\\displaystyle=\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\(26\)=∑j=0d/2−1Re​\[𝒒m\(j\)​𝒌n\(j\)∗​ei​2​π​ξj​\(pm−pn\)\]\\displaystyle=\\sum\_\{j=0\}^\{d/2\-1\}\\mathrm\{Re\}\\left\[\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}e^\{i2\\pi\\xi\_\{j\}\(p\_\{m\}\-p\_\{n\}\)\}\\right\]
By Proposition 3\.1,𝔼​\[h\]=𝒒m⊤​𝒌n​f​\(pm−pn\)\\mathbb\{E\}\[h\]=\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)\.Changingξj\\xi\_\{j\}toξj′\\xi\_\{j\}^\{\\prime\}while keeping all other frequencies fixed, the change inhhis bounded by:

\|h​\(…,ξj,…\)−h​\(…,ξj′,…\)\|\\displaystyle\|h\(\\ldots,\\xi\_\{j\},\\ldots\)\-h\(\\ldots,\\xi\_\{j\}^\{\\prime\},\\ldots\)\|=\|Re​\[𝒒m\(j\)​𝒌n\(j\)∗​ei​2​π​ξj​\(pm−pn\)\]−Re​\[𝒒m\(j\)​𝒌n\(j\)∗​ei​2​π​ξj′​\(pm−pn\)\]\|\\displaystyle=\\left\|\\mathrm\{Re\}\\left\[\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}e^\{i2\\pi\\xi\_\{j\}\(p\_\{m\}\-p\_\{n\}\)\}\\right\]\-\\mathrm\{Re\}\\left\[\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}e^\{i2\\pi\\xi\_\{j\}^\{\\prime\}\(p\_\{m\}\-p\_\{n\}\)\}\\right\]\\right\|≤\|𝒒m\(j\)​𝒌n\(j\)∗\|⋅\|ei​2​π​ξj​\(pm−pn\)−ei​2​π​ξj′​\(pm−pn\)\|\\displaystyle\\leq\\left\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\\right\|\\cdot\\left\|e^\{i2\\pi\\xi\_\{j\}\(p\_\{m\}\-p\_\{n\}\)\}\-e^\{i2\\pi\\xi\_\{j\}^\{\\prime\}\(p\_\{m\}\-p\_\{n\}\)\}\\right\|≤2​\|𝒒m\(j\)​𝒌n\(j\)∗\|\\displaystyle\\leq 2\\left\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\\right\|=2​\|𝒒m\(j\)\|​\|𝒌n\(j\)∗\|\\displaystyle=2\\left\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\|\\left\|\{\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\}^\{\*\}\\right\|\(27\)=2​‖𝒒m\(j\)‖​‖𝒌n\(j\)‖\\displaystyle=2\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|
The last inequality follows from the maximum distance between any two points on the unit circle is 2\. The last equality comes from replacing complex norm with corresponding 2d vector norm\.

Denote boundcj=2​‖𝒒m\(j\)‖​‖𝒌n\(j\)‖c\_\{j\}=2\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|and apply McDiarmid’s inequality tohh:

P​\(\|∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)−𝒒m⊤​𝒌n​f​\(pm−pn\)\|≥ϵ\)\\displaystyle P\\left\(\\left\|\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)\-\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)\\right\|\\geq\\epsilon\\right\)≤2​exp⁡\(−2​ϵ2∑j=0d/2−1cj2\)\\displaystyle\\leq 2\\exp\\left\(\-\\frac\{2\\epsilon^\{2\}\}\{\\sum\_\{j=0\}^\{d/2\-1\}c\_\{j\}^\{2\}\}\\right\)\(28\)=2​exp⁡\(−ϵ22​∑j=0d/2−1\(‖𝒒m\(j\)‖​‖𝒌n\(j\)‖\)2\)\\displaystyle=2\\exp\\left\(\-\\frac\{\\epsilon^\{2\}\}\{2\\sum\_\{j=0\}^\{d/2\-1\}\(\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|\)^\{2\}\}\\right\)
replacingϵ\\epsilonwithd2​ϵ′\\frac\{d\}\{2\}\\epsilon^\{\\prime\}completes the proof\. ∎

### A\.2\.Proof of RFR Exponential Convergence with Feature Dimension at Initialization

LetXj=‖𝒒m\(j\)‖​‖𝒌n\(j\)‖X\_\{j\}=\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|\. When query and key projection matrices are initialized i\.i\.d\. with variance independent of d, we have\{Xj\}j=0d/2−1\\\{X\_\{j\}\\\}\_\{j=0\}^\{d/2\-1\}i\.i\.d\. acrossjjat initialization and𝔼​\[Xj2\]\\mathbb\{E\}\[X\_\{j\}^\{2\}\]remains constant as d scales\. Applying the law of large numbers toXj2X\_\{j\}^\{2\}, we obtain

1d/2​∑j=0d/2−1Xj2→𝔼​\[Xj2\],\\frac\{1\}\{d/2\}\\sum\_\{j=0\}^\{d/2\-1\}X\_\{j\}^\{2\}\\to\\mathbb\{E\}\[X\_\{j\}^\{2\}\],which implies∑j=0d/2−1Xj2=Θ​\(d\)\\sum\_\{j=0\}^\{d/2\-1\}X\_\{j\}^\{2\}=\\Theta\(d\)\. In Proposition 3\.2, substituting∑j=0d/2−1\(‖𝒒m\(j\)‖​‖𝒌n\(j\)‖\)2\\sum\_\{j=0\}^\{d/2\-1\}\(\\left\\\|\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\\right\\\|\\left\\\|\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\\right\\\|\)^\{2\}withΘ​\(d\)\\Theta\(d\)yields that

1d/2​∑j=0d/2−1\(𝑹m\(j\)​𝒒m\(j\)\)⊤​\(𝑹n\(j\)​𝒌n\(j\)\)\\frac\{1\}\{d/2\}\\sum\_\{j=0\}^\{d/2\-1\}\(\\boldsymbol\{R\}\_\{m\}^\{\(j\)\}\\boldsymbol\{q\}\_\{m\}^\{\(j\)\}\)^\{\\top\}\(\\boldsymbol\{R\}\_\{n\}^\{\(j\)\}\\boldsymbol\{k\}\_\{n\}^\{\(j\)\}\)converges to

1d/2​𝒒m⊤​𝒌n​f​\(pm−pn\)\\frac\{1\}\{d/2\}\\boldsymbol\{q\}\_\{m\}^\{\\top\}\\boldsymbol\{k\}\_\{n\}f\(p\_\{m\}\-p\_\{n\}\)exponentially fast indd\.

## Appendix BCan Rotation Frequencies Be Learnt?

We conducted experiments that jointly optimize rotation frequencies with network weights\. In Table[9](https://arxiv.org/html/2607.26369#A2.T9), we compare learnable frequencies with 0 initialization and ClockRoPE initializations\. We also list ClockRoPE variants with fixed frequencies for reference\. We find that \(1\) using ClockRoPE initialization improves performance over 0 initialization and \(2\) trainable frequencies initialized with ClockRoPE yields similar performance as fixed frequencies\.

Table 9\.Percentage MAP improvement over prod control\. Learnable frequncies initialized with ClockRoPE achieve similar performance as fixed frequencies\.MethodMAP@1MAP@50Learnable frequencies initialized from 0\.0\-1\.44%\-1\.40%ClockRoPE\-cosine\-fold\+1\.96%\+1\.37%\+ Learnable frequencies\+2\.01%\+1\.45%ClockRoPE\-gaussian\-fold\+3\.61%\+2\.25%\+ Learnable frequencies\+3\.23%\+2\.16%