Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

# How Positional Schemes Set the Default Spectral Algebra of Attention
Source: [https://arxiv.org/html/2607.06621](https://arxiv.org/html/2607.06621)
## Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

###### Abstract

The pre\-softmax score of an attention head is a bilinear formscore​\(i,j\)=xi⊤​M​xj\\mathrm\{score\}\(i,j\)=x\_\{i\}^\{\\top\}Mx\_\{j\}in a learned operatorM=Wq⊤​WkM=W\_\{q\}^\{\\top\}W\_\{k\}\. BecauseMMis generally non\-symmetric, hence non\-normal, it has a complex eigenspectrum and non\-orthogonal eigenvectors, the regime where non\-Hermitian and random\-matrix tools apply\. We ask what this spectrum encodes, at three levels for previous\-token and induction circuits\. Statically, across seven pretrained models spanning three positional schemes, the strongest previous\-token heads are spectrally rotational under RoPE and non\-rotational, or content\-like, where position enters outside QK \(learned\-absolute and ALiBi\); the model\-level separation is perfect at every top\-kkexamined \(exact permutationp=0\.029p\{=\}0\.029\), and zeroing the per\-frequency RoPE phaseIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)eliminates induction on a pre\-identified previous\-token head in all three RoPE models\. Dynamically, over public Pythia checkpoints every head originates at the random\-matrix \(Ginibre\) null; the rotational signature emerges with the behavior, not before it, and the population\-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor\. Causally, and at toy scale, no spectral channel is necessary: constrained two\-layer training reroutes around every ban with capability intact, albeit at a significant formation delay \(four pre\-registered contrasts,qBH≤0\.016q\_\{\\mathrm\{BH\}\}\{\\leq\}0\.016\)\. The cost structure exposes each scheme’s default: imposing symmetry slows learned\-absolute models by a factor of2\.92\.9, whereas a RoPE head with a fully symmetric staticMMstill routes directionally via the phase channel, impossible under absolute positions\. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head’s solution: a fingerprint sculpted after function, not a hard constraint upon it\.

## 1Introduction

For a single attention head with norm\-processed residual tokenx∈ℝdx\\in\\mathbb\{R\}^\{d\}and projectionsWq,WkW\_\{q\},W\_\{k\}, the pre\-softmax score is a bilinear form in one operator,

score​\(i,j\)=qi⊤​kj=xi⊤​\(Wq⊤​Wk\)​xj=xi⊤​M​xj,M=Wq⊤​Wk∈ℝd×d\.\\mathrm\{score\}\(i,j\)=q\_\{i\}^\{\\top\}k\_\{j\}=x\_\{i\}^\{\\top\}\(W\_\{q\}^\{\\top\}W\_\{k\}\)x\_\{j\}=x\_\{i\}^\{\\top\}Mx\_\{j\},\\qquad M=W\_\{q\}^\{\\top\}W\_\{k\}\\in\\mathbb\{R\}^\{d\\times d\}\.\(1\)MMis the natural, gauge\-invariant object of study: the reparametrizationWq→Wq​G,Wk→Wk​G−⊤W\_\{q\}\\\!\\to\\\!W\_\{q\}G,\\ W\_\{k\}\\\!\\to\\\!W\_\{k\}G^\{\-\\top\}leavesMMand every observable unchanged, so rawWq/WkW\_\{q\}/W\_\{k\}entries are gauge artifacts while functions ofMMare physical\([6](https://arxiv.org/html/2607.06621#bib.bib1)\)\.MMis low rank \(rank≤dk\\mathrm\{rank\}\\leq d\_\{k\}\) and, crucially,*generally non\-symmetric*—so it has a complex eigenspectrum\.

A growing literature reads non\-normal/complex spectral structure into transformers: Schur decompositions of the residual\-stream Jacobian\([7](https://arxiv.org/html/2607.06621#bib.bib8)\), “non\-Hermitian” operator\-theoretic framings\([3](https://arxiv.org/html/2607.06621#bib.bib9)\), and symmetric/antisymmetric decompositions of the QK matrix\([15](https://arxiv.org/html/2607.06621#bib.bib2)\); concurrent work catalogues per\-head complex eigenvalues of the attention interaction and ablates its skew/symmetric channels against corpus perplexity\([9](https://arxiv.org/html/2607.06621#bib.bib12)\)\. Yet the QK operator’s complex spectrum remains*behaviorally unanchored*— no random\-matrix nulls, no head\-function taxonomy, no head\-resolved causal tests, no positional\-scheme contrasts — and the field’s discipline demands we ask not whether the structure*exists*but whether it*does measurable work*\. Concretely: does the complex\-eigenvalue view ofMMpredict head behavior that the plain symmetric/antisymmetric split does not already capture?

We answer this at three levels:*statics*across seven pretrained models — GPT\-2 small, OPT\-1\.3B, and GPT\-Neo\-1\.3B \(learned\-absolute positions\), BLOOM\-1b1 \(ALiBi: position enters as a score bias, leaving QK positionally unconstrained\), Pythia\-410m/1\.4B \(25%25\\%RoPE\), and Llama\-3\-8B \(100%100\\%RoPE, RMSNorm, GQA\);*dynamics*across the public Pythia training checkpoints; and*interventions*in constrained\-from\-scratch training runs\. Our contributions:

1. 1\.A matched\-null spectral framework for the QK operator\(§[3](https://arxiv.org/html/2607.06621#S3)\): four decompositions of the norm\-foldedMM, a primary directionality metricDheadD\_\{\\text\{head\}\}and its plain\-split baselinedir\_frac, and—essential—a*random\-orientation \(Ginibre\) null*against which directionality must be read \(a random low\-rankMMalready hasdir\_frac≈1/2\\texttt\{dir\\\_frac\}\\approx 1/\\sqrt\{2\}\)\.
2. 2\.Descriptive: spectral directionality separates head function across all seven models \(§[4](https://arxiv.org/html/2607.06621#S4)\), extendingSaponatiet al\.\([15](https://arxiv.org/html/2607.06621#bib.bib2)\)from training objective to head function\.
3. 3\.Causal anatomyon GPT\-2 \(§[5](https://arxiv.org/html/2607.06621#S5)\): the symmetric partMSM\_\{S\}is6\.7×6\.7\\timesmore load\-bearing than the antisymmetricMAM\_\{A\};MAM\_\{A\}is causally critical only for the canonical induction/previous\-token circuit and inert elsewhere\. On GPT\-2 the complex refinementDheadD\_\{\\text\{head\}\}does*not*beatdir\_frac\.
4. 4\.The architecture\-conditional signature\(§[6](https://arxiv.org/html/2607.06621#S6)\): the strongest previous\-token heads are spectrally rotational under RoPE \(top\-quintileDheadD\_\{\\text\{head\}\}\) and non\-rotational under learned\-absolute*and*ALiBi positions \(bottom\-quartile, four models\); the model\-level separation is perfect at every top\-kktested \(exact permutationp=0\.029p\{=\}0\.029; head\-levelp=2\.5×10−5p\{=\}2\.5\{\\times\}10^\{\-5\}, descriptive\);dkd\_\{k\}held fixed across schemes\. Aggregate partials are shown to be bulk\-contaminated and unreliable for this contrast\.
5. 5\.Mechanism\(§[7](https://arxiv.org/html/2607.06621#S7)\): a per\-frequency complex decomposition tiesDheadD\_\{\\text\{head\}\}to the RoPE rotational phaseIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\); adding RoPE\-phase features as controls attenuatesDheadD\_\{\\text\{head\}\}’s advantage by∼47\{\\sim\}47–61%61\\%\(25%25\\%RoPE\) and∼70\{\\sim\}70–98%98\\%\(100%100\\%RoPE\), estimator\-dependent\.
6. 6\.Causal confirmation\(§[8](https://arxiv.org/html/2607.06621#S8)\): ablatingIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)symmetrizes the relative\-position kernel and destroys induction on exactly the flagged heads\.
7. 7\.A checkpoint natural history\(§[9](https://arxiv.org/html/2607.06621#S9)\): on Pythia\-410m/160m \(22 public checkpoints each\), all heads are born at the Ginibre null; the rotational signature locks in*with*circuit formation \(not before it\), and the population\-median suppression that produces the static profile follows formation — the profile is a consolidated fingerprint, not a precursor\.
8. 8\.Constrained\-training interventions\(§[10](https://arxiv.org/html/2607.06621#S10)\): a\{\\\{APE, RoPE\}×\{\\\}\\times\\\{free, sym\-MM,Im​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)\-suppressed\}\\\}grid \(n=5n\{=\}5seeds\) shows no spectral channel is*necessary*\(every arm reroutes\) while every constraint carries a significant search cost that reveals the scheme’s default solution, and dissociates the static\-antisymmetry and RoPE\-phase channels at the weight level\.

One sentence unifies the three levels:*the positional scheme sets the default spectral algebra of attention’s solutions — a fingerprint sculpted after function, an economy of search rather than a hard constraint\.*We report the negatives \(GPT\-2 decorativeness;MAM\_\{A\}’s inertness away from a few heads\) as plainly as the positives\.

## 2Related work

QK circuits and weight\-space interpretability\.Elhageet al\.\([6](https://arxiv.org/html/2607.06621#bib.bib1)\)frameM=Wq⊤​WkM\{=\}W\_\{q\}^\{\\top\}W\_\{k\}andWO​VW\_\{OV\}as the two low\-rank circuits and use*OV*eigenvalue positivity as a copying statistic; they perform no QK spectral analysis\.Millidge and Black \([12](https://arxiv.org/html/2607.06621#bib.bib7)\)find weight\-SVD interpretable but report it*fails*on QK—motivating an orientation\-recovering \(Schur/complex\) view\. Our object and the OV/QK distinction follow this line; our contribution is the QK\-side, imaginary\-axis, behavior\-anchored analog\.

Symmetric/antisymmetric QK\.Saponatiet al\.\([15](https://arxiv.org/html/2607.06621#bib.bib2)\)decomposeWq​k=Ms\+MnW\_\{qk\}\{=\}M\_\{s\}\{\+\}M\_\{n\}and define a Frobenius symmetry score, anchoring it to the*training objective*\. Ourdir\_fracis a monotone transform of their score \(s=1−2​dir\_frac2s=1\-2\\,\\texttt\{dir\\\_frac\}^\{2\}\); we anchor the split to*head function*and add the complex/Schur refinement and its causal test\.

Non\-normal / non\-Hermitian transformers\.Concurrently with this work,Jamil and Kapadia \([9](https://arxiv.org/html/2607.06621#bib.bib12)\)decompose the per\-head attention interaction into skew \(“routing”\) and symmetric \(“filtering”\) channels across1,7761\{,\}776heads of five pretrained models, reportmax⁡Re​λ\>0\\max\\mathrm\{Re\}\\,\\lambda\>0for every head, find by all\-heads truncation surgery on GPT\-2 Large that the symmetric channel is by far the more load\-bearing \(convergent, at the aggregate level, with our §[5](https://arxiv.org/html/2607.06621#S5)\), and train a stability\-constrained skew\-minus\-diagonal attention from scratch\. Their analysis stops at norms, ranks, and stability — no behavioral taxonomy, no random\-matrix null, no head\-resolved causal test, no positional\-scheme contrast, and their weight\-level routing–filtering ratio is \(like Saponati’s score\) a monotone transform ofdir\_frac— so it cannot ask whether the complex spectrum does work*beyond*the plain split; that anchoring is our contribution\.Fernando and Guitchounts \([7](https://arxiv.org/html/2607.06621#bib.bib8)\)Schur\-decompose the residual\-stream*Jacobian*and show learned non\-normality is functionally central—but on a different operator, without pseudospectra, and anchored to rank/propagation, not head function\.Chang \([3](https://arxiv.org/html/2607.06621#bib.bib9)\)labelsWq⊤​WkW\_\{q\}^\{\\top\}W\_\{k\}non\-Hermitian in a Dyson\-series theory with no computation\. We differ by analyzing the QK operator’s complex spectrum empirically and anchoring it to head behavior\.

RoPE and positional heads\.RoPE\([17](https://arxiv.org/html/2607.06621#bib.bib5)\)injectsei​m​θte^\{im\\theta\_\{t\}\}phase,θt=base−2​t/d\\theta\_\{t\}=\\mathrm\{base\}^\{\-2t/d\}\.Barberoet al\.\([1](https://arxiv.org/html/2607.06621#bib.bib6)\)andUrrutiaet al\.\([19](https://arxiv.org/html/2607.06621#bib.bib10)\)show positional heads use high RoPE frequencies\. We connect a*weight\-space*directionality scalar to per\-head RoPE\-frequency usage and to head function, which they do not\.

Circuit formation over training\.Olssonet al\.\([13](https://arxiv.org/html/2607.06621#bib.bib3)\)established the induction phase change;Tiggeset al\.\([18](https://arxiv.org/html/2607.06621#bib.bib13)\)track circuits across all Pythia checkpoints behaviorally \(induction at∼2×109\{\\sim\}2\{\\times\}10^\{9\}tokens\); toy\-model studies dissect formation\([2](https://arxiv.org/html/2607.06621#bib.bib14),[14](https://arxiv.org/html/2607.06621#bib.bib15),[16](https://arxiv.org/html/2607.06621#bib.bib16)\); andChenet al\.\([4](https://arxiv.org/html/2607.06621#bib.bib17)\)regularize an interpretable attention property during MLM pretraining — the template our intervention ports to autoregressive weight space\.Saponatiet al\.\([15](https://arxiv.org/html/2607.06621#bib.bib2)\)track their symmetry score during training \(per\-layer, own encoder/decoder runs\)\. None of these tracks a weight\-space spectral quantity per head across public checkpoints against the phase change \(§[9](https://arxiv.org/html/2607.06621#S9)\), and none constrains QK spectral structure during autoregressive training \(§[10](https://arxiv.org/html/2607.06621#S10)\)\.

Constrained\-attention training\.Hard symmetric \(shared\-QK\) causal LMs exist — Reformer\([10](https://arxiv.org/html/2607.06621#bib.bib18)\)and recent systematic sharing studies — and symmetric dot\-product BERT trains well\([5](https://arxiv.org/html/2607.06621#bib.bib19)\), but none analyzes circuits under the constraint, and the soft, decomposed versions \(penalize onlyMAM\_\{A\}, or onlyIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)\) are new here; concurrent observational work crosses positional schemes with induction formation in toy models\([8](https://arxiv.org/html/2607.06621#bib.bib20)\)\.

Head taxonomy\.We use standard detectors—previous\-token, duplicate, induction\([13](https://arxiv.org/html/2607.06621#bib.bib3)\)—and IOI\([20](https://arxiv.org/html/2607.06621#bib.bib4)\)head classes for validation\.

## 3Method

Object and folding\.We use TransformerLens withfold\_ln=Trueto absorb the norm gain intoWq,WkW\_\{q\},W\_\{k\}, and analyze the norm\-folded operatorMeff=C​M​CM\_\{\\text\{eff\}\}=CMCfor LayerNorm \(centeringC=I−1d​𝟏𝟏⊤C=I\-\\tfrac\{1\}\{d\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\) andMeff=MM\_\{\\text\{eff\}\}=Mfor RMSNorm\. All spectral work is float64\. We verify the convention by reconstructing the model’s own pre\-softmax scores fromMMand matching to<10−4<10^\{\-4\}\(learned\-absolute\) or, for RoPE models where position cancels only ati=ji\{=\}j, matching the diagonal \(bf16 models match to bf16 precision; the fast metric path is exact,dir\_frac​Δ=10−16\\texttt\{dir\\\_frac\}\\ \\Delta\{=\}10^\{\-16\}\)\. A randomG∈G​L​\(dk\)G\\in GL\(d\_\{k\}\)leavesMM, its eigenvalues, andDheadD\_\{\\text\{head\}\}invariant to∼10−14\{\\sim\}10^\{\-14\}while changing raw column norms—confirming gauge hygiene\.

Decompositions\.Per head we compute the SVD, the symmetric/antisymmetric splitM=MS\+MAM\{=\}M\_\{S\}\{\+\}M\_\{A\}, the real Schur form \(2×22\\times 2blocks→\\torotation anglesθ\\theta\), and the complex eigendecomposition \(with eigenvector condition number as a non\-normality flag\)\.

Metrics\.

dir\_frac=‖MA‖F/‖M‖F\\displaystyle=\\\|M\_\{A\}\\\|\_\{F\}/\\\|M\\\|\_\{F\}\(plain antisymmetric magnitude; Saponati’s score\)\(2\)Dhead\\displaystyle D\_\{\\text\{head\}\}=∑r\|Im​λr\|/∑r\|λr\|\\displaystyle=\\textstyle\\sum\_\{r\}\|\\mathrm\{Im\}\\,\\lambda\_\{r\}\|\\big/\\sum\_\{r\}\|\\lambda\_\{r\}\|\(primary complex/Schur directionality\)\(3\)content\_pos\_frac=∑λ\+/∑\|λ\|​of​MS\\displaystyle=\\textstyle\\sum\\lambda^\{\+\}\\big/\\sum\|\\lambda\|\\ \\text\{of\}\\ M\_\{S\}\(signed content axis; QK\-side echo of OV positivity\)\(4\)plushenrici=‖M‖F2−∑\|λ\|2/‖M‖F\\text\{henrici\}=\\sqrt\{\\\|M\\\|\_\{F\}^\{2\}\-\\sum\|\\lambda\|^\{2\}\}/\\\|M\\\|\_\{F\}\(departure from normality\) andself\_match\(diagonal dominance ofWE​M​WE⊤W\_\{E\}MW\_\{E\}^\{\\top\}\)\.

The random\-orientation null \(essential\)\.A random low\-rankMMis already highly “directional”: for independent GaussianWq,WkW\_\{q\},W\_\{k\}the symmetric and antisymmetric parts get equal expected Frobenius mass, sodir\_fracnull≈1/2≈0\.707\\texttt\{dir\\\_frac\}\_\{\\text\{null\}\}\\approx 1/\\sqrt\{2\}\\approx 0\.707andDheadD\_\{\\text\{head\}\}’s null is likewise high\. We therefore report every metric relative to a*matched\-Σ\\Sigmarandom\-orientation null*\(same singular values, random left/right frames\), computed exactly in the rank\-kkcore\. Empiricallydir\_fracnull=0\.707\\texttt\{dir\\\_frac\}\_\{\\text\{null\}\}\{=\}0\.707,Dhead,null=0\.608D\_\{\\text\{head,null\}\}\{=\}0\.608;141/144141/144GPT\-2 heads deviate \(\|z\|\>2\|z\|\{\>\}2ondir\_fracorDheadD\_\{\\text\{head\}\};dir\_fracalone:140/144140/144\)\. Nulls are computed exhaustively for GPT\-2; the cross\-model analyses of §[6](https://arxiv.org/html/2607.06621#S6)use raw metrics — on GPT\-2, null\-referenced metrics give an indistinguishable partial \(−0\.204\-0\.204vs−0\.206\-0\.206\), so this choice is not load\-bearing\.

Models & corpus\.GPT\-2 small \(144 heads\), OPT\-1\.3B \(768\), GPT\-Neo\-1\.3B \(384\), BLOOM\-1b1 \(384; ALiBi\), Pythia\-410m/1\.4B \(384\), Llama\-3\-8B \(1024; GQA expanded, RMSNorm, bf16\)\. Behavioral runs use a1​k×1281\\mathrm\{k\}\\times 128slice ofNeelNanda/pile\-10k; detectors use repeated\-random sequences\. All on one A100\.

Checkpoint and intervention pipelines\.For §[9](https://arxiv.org/html/2607.06621#S9)we load Pythia public checkpoints natively \(TransformerLenscheckpoint\_value\); the fused\-QKV unpacking and folding conventions were verified against the main\-revision pipeline to machine precision \(weight metrics at the final checkpoint match the independent pipeline exactly,Δ=0\\Delta\{=\}0\)\. For §[10](https://arxiv.org/html/2607.06621#S10)we train 2\-layer attention\-only models from scratch with soft spectral penalties; definitions and the crucial sym\-MM≠\\neqIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)distinction are given there\. Pre\-registration: the dynamics questions \(Q1–Q3\) and intervention predictions \(P1–P4\) were locked in the project’s intervention plan before any checkpoint was downloaded or any constrained run launched; we report the falsified predictions as findings\.

## 4Spectral directionality separates head function

On GPT\-2 \(n=144n\{=\}144, Spearman, BH\-FDRq<10−3q\{<\}10^\{\-3\}\), directionality maps onto head class \(Table[1](https://arxiv.org/html/2607.06621#S4.T1)\)\. Previous\-token heads are directional \(highdir\_frac, low content positivity\); duplicate/similarity heads are symmetric;induction heads read as content\-like\. We note plainly that our preregistered hypothesis predicted the opposite \(induction heads directional\); that prediction*failed*\. The two\-head\-circuit reading — the induction head’s QK consumes*composed content*written by a previous\-token head, so its single\-headMMis symmetric — was recorded in the analysis plan before these correlations were computed, and we verify it directly: by the K\-composition statistic \(the Frobenius overlap of an induction head’s QK operator with an earlier head’s OV write, ranked over all earlier heads\),the canonical previous\-token head ranks \#1 for every induction head tested— GPT\-2’s 4\.11 for 5\.1/5\.5/6\.9 \(ranks 1/60, 1/60, 1/72\) and Pythia\-410m’s 5\.2 for 8\.6/10\.9/11\.14 \(1/128, 1/160, 1/176\) — confirming that induction QK is wired to consume the prev head’s composed output\.Copying \(OV\) is orthogonal to every QK metric, a clean control\. The content signature is legible:self\_matchequalscontent\_pos\_fracatr=0\.96r\{=\}0\.96; empirical pre\-softmax score asymmetry on real inputs tracksdir\_fracat0\.660\.66\. This replicates on Pythia and, attenuated, on Llama \(content\_pos\_frac→\\toprev−0\.30\-0\.30vs GPT\-2’s−0\.63\-0\.63; §[6](https://arxiv.org/html/2607.06621#S6)\), establishing spectral directionality as an architecture\-general head\-function signature and extendingSaponatiet al\.\([15](https://arxiv.org/html/2607.06621#bib.bib2)\)from objective to function \(Fig\.[1](https://arxiv.org/html/2607.06621#S4.F1)\)\.

Table 1:Spearman correlations of spectral metrics with head\-type scores \(GPT\-2,n=144n\{=\}144; all shown are FDR\-significant\)\. Copying is an orthogonal OV\-side control\.![Refer to caption](https://arxiv.org/html/2607.06621v1/x1.png)Figure 1:The content–direction plane \(GPT\-2\)\. Each point is a head; dashed lines mark the random\-orientation null\(dir\_fracnull≈0\.707,Dhead,null≈0\.608\)\(\\texttt\{dir\\\_frac\}\_\{\\text\{null\}\}\{\\approx\}0\.707,\\ D\_\{\\text\{head,null\}\}\{\\approx\}0\.608\)\. A content cluster \(low on both, mostly layer 0\) sits below the null; directionality is read as deviation from it\.
## 5Causal anatomy on GPT\-2:MS≫MAM\_\{S\}\\gg M\_\{A\}

Symmetrization ablation\.Replacing a head’sMMwithMSM\_\{S\}\(killingMAM\_\{A\}\) orMAM\_\{A\}\(killingMSM\_\{S\}\) and measuring held\-out CE, thesymmetric part is6\.7×6\.7\\timesmore load\-bearing:∑Δ​CEanti=2\.60\\sum\\Delta\\mathrm\{CE\}\_\{\\text\{anti\}\}\{=\}2\.60vs∑Δ​CEsym=0\.39\\sum\\Delta\\mathrm\{CE\}\_\{\\text\{sym\}\}\{=\}0\.39; killingMAM\_\{A\}costs≤0\.02\\leq 0\.02general CE anywhere\. But a*targeted induction task*exposes thatMAM\_\{A\}is causally*essential*for specific heads: killing it on the canonical induction head 5\.1 raises induction loss8×8\\times\(0\.21→1\.770\.21\\to 1\.77;15×15\\timesthe next\-largest effect\) and on the previous\-token head 4\.11 by∼50%\{\\sim\}50\\%\. Canonical induction heads 5\.5/6\.9 show smaller positive effects \(\+0\.03\+0\.03, ranks 7–8 of 144\); the top\-5 by effect are 5\.1, 4\.11, 6\.7, 3\.0, 8\.5, so top\-5 overlap with the canonical circuit is 2/5 \(hypergeometricp=0\.014p\{=\}0\.014\), with the argmax landing on a canonical induction head — circuit\-level enrichment, not an exact circuit match\. Crucially this importance is adiscrete circuit fact: no spectral scalar \(dir\_frac,DheadD\_\{\\text\{head\}\},imag\_mass,‖MA‖\\\|M\_\{A\}\\\|\) predicts*which*heads needMAM\_\{A\}\(all\|ρ\|≲0\.1\|\\rho\|\\lesssim 0\.1\), so the 144\-head aggregate is null\.

Incremental validity\.For predicting the previous\-token score, the complex refinement adds no reliable positive increment on GPT\-2:partial​ρ​\(Dhead,prev∣dir\_frac\)=−0\.21\\mathrm\{partial\}\\,\\rho\(D\_\{\\text\{head\}\},\\text\{prev\}\\mid\\texttt\{dir\\\_frac\}\)=\-0\.21\(layer\-clustered 95% CI\[−0\.51,\+0\.08\]\[\-0\.51,\+0\.08\]— the point estimate is negative but not reliably so; among prev\-candidate heads,prev\>0\.05\\text\{prev\}\{\>\}0\.05, the residual association is strongly negative,−0\.63\-0\.63\), and in a nested regression on the ablation effectdir\_fracexplains0\.0010\.001of variance rising only to0\.0280\.028withDheadD\_\{\\text\{head\}\}added, the added term wrong\-signed\. On learned\-absolute positions, the plain antisymmetric magnitude captures everything the complex spectrum reliably offers about its strong prev heads; §[6](https://arxiv.org/html/2607.06621#S6)shows this head\-profile pattern generalizes to OPT\-1\.3B and GPT\-Neo\-1\.3B, though their aggregate statistics do not\.

## 6The architecture\-conditional signature \(main result\)

We run the incremental\-validity test across positional schemes on seven models \(Table[2](https://arxiv.org/html/2607.06621#S6.T2)\)\. On every RoPE model,DheadD\_\{\\text\{head\}\}’s incremental contribution beyonddir\_fracis positive and robust \(clustered CIs\[\+0\.21,\+0\.59\]\[\+0\.21,\+0\.59\]\)\. Under learned\-absolute positions, however, the*aggregate*statistic is heterogeneous \(GPT\-2−0\.21\-0\.21; OPT\-1\.3B\+0\.35\+0\.35; GPT\-Neo\-1\.3B\+0\.20\+0\.20\) — and decomposing it shows why it is the wrong statistic: a positive gradation among near\-zero\-prev*bulk*heads is present in most models \(including GPT\-2:\+0\.25\+0\.25withinprev≤0\.05\\text\{prev\}\{\\leq\}0\.05\), while the signal that differs by architecture lives in the*strong*previous\-token heads themselves\. The last column of Table[2](https://arxiv.org/html/2607.06621#S6.T2)gives each model’s top\-5 prev heads’ within\-modelDheadD\_\{\\text\{head\}\}percentile:bottom\-quartile under learned absolute\(medians0\.21/0\.23/0\.140\.21/0\.23/0\.14; e\.g\. GPT\-2’s 4\.11 at0\.220\.22; all with content\-like symmetric profiles,content\_pos\_frac0\.460\.46–0\.570\.57\) and ALiBi \(0\.260\.26\) versustop\-quintile under RoPE\(medians0\.89/0\.85/0\.790\.89/0\.85/0\.79; Fig\.[2](https://arxiv.org/html/2607.06621#S6.F2)a\)\. Holdingdk=64d\_\{k\}\{=\}64fixed \(GPT\-2, OPT\-1\.3B, Pythia\-410m:0\.21/0\.230\.21/0\.23vs0\.890\.89\) ties the contrast to the positional scheme, not head dimension\.

Model\-level inference and sensitivity\.Heads within a model share training, layers, and detectors, so the model is the honest experimental unit\. At model level the separation is*perfect*— every non\-RoPE model’s top\-5 median sits below every RoPE model’s — giving an exact model\-level permutationp=1/\(73\)=0\.029p\{=\}1/\\binom\{7\}\{3\}\{=\}0\.029, the minimum attainable with seven models; the head\-level joint Mann–Whitneyp=2\.5×10−5p\{=\}2\.5\{\\times\}10^\{\-5\}is reported as descriptive only\. The contrast is not an artifact of thek=5k\{=\}5choice: model\-level separation is perfect at everyk∈\{1,3,5,10\}k\\in\\\{1,3,5,10\\\}, and leave\-one\-model\-out preserves it for all seven models\. The binding statistical limit is the model count itself — one ALiBi and one full\-RoPE family — which more model families, not more heads, would address \(Limitations\)\. The defensible claim is*profile\-level*:the heads that implement previous\-token attention are spectrally rotational under RoPE and spectrally non\-rotational \(content\-like\) under learned absolute positions\.Mechanistically, this is what the positional algebra dictates: with learned absolute embeddings, prev\-token attention can be built by*matching*adjacent position vectors — a symmetric, content\-like operation — whereas RoPE builds it from*rotations*, imprinting complex eigenstructure\.ALiBi provides the cleanest test: position enters as a score*bias*, so QK needs no positional structure at all — and BLOOM\-1b1’s top prev heads are indeed non\-rotational \(percentile0\.260\.26\), patterning with the learned\-absolute group exactly as the mechanism predicts\. Under RoPE these heads sit at thedir\_fracnull \(∼0\.70\{\\sim\}0\.70\) withDheadD\_\{\\text\{head\}\}at the Ginibre value \(≈0\.61\{\\approx\}0\.61\) while the population suppresses below it \(medians0\.400\.40–0\.450\.45; per\-model matched nulls: 410m prev heads’DheadD\_\{\\text\{head\}\}z∈\[−0\.1,\+5\.0\]z\\in\[\-0\.1,\+5\.0\]vs population medianz=−11z\{=\}\{\-\}11; 1\.4B prev\-head medianz=−3\.7z\{=\}\{\-\}3\.7vs populationz=−13\.9z\{=\}\{\-\}13\.9\): they*retain*near\-random rotational structure that other heads learn to suppress, and the Schur/complex view detects this where the Frobenius magnitude cannot\.

Robustness and scope\.Three qualifications sharpen the claim\. \(i\)*Metric specificity*: among complex\-spectrum summaries,imag\_massis incrementally positive on all models and Henrici non\-normality negative on all;DheadD\_\{\\text\{head\}\}composites these two channels\. No single aggregate scalar separates the schemes — the head\-profile statistic does\. \(ii\)*Class vs bulk*: the RoPE positives are class\-level separation \(rank\-biserial\+0\.44/\+0\.60\+0\.44/\+0\.60atprev\>0\.2\\text\{prev\}\{\>\}0\.2in Pythia;\+0\.29\+0\.29atprev\>0\.1\\text\{prev\}\{\>\}0\.1in Llama\)*plus*bulk gradation; within confirmed prev heads there is no further gradation \(subset partials n\.s\.\)\. Under learned\-absolute positions the class\-level tests are null\-to\-negative at the strong end \(GPT\-2−0\.39\-0\.39atprev\>0\.2\\text\{prev\}\{\>\}0\.2, significant; OPT−0\.11\-0\.11n\.s\.; GPT\-Neo\+0\.13\+0\.13n\.s\.\), though GPT\-Neo shows a positive signal at the looserprev\>0\.1\\text\{prev\}\{\>\}0\.1band \(\+0\.34\+0\.34\) driven by its moderate\-prev heads — its*top*prev heads still sit at percentile0\.140\.14\. The learned\-absolute models are thus heterogeneous in their moderate bands \(candidate sources: training corpus, GPT\-Neo’s alternating local\-attention layers\) while uniform at the top of the class\. \(iii\) GPT\-2’s negative aggregate concentrates exactly among prev\-candidate heads \(−0\.63\-0\.63atprev\>0\.05\\text\{prev\}\{\>\}0\.05\); OPT’s positive aggregate is entirely bulk \(−0\.18\-0\.18among its prev candidates\)\. Aggregate partials should not be read as the architecture signal in either direction\.

Table 2:Incremental validity ofDheadD\_\{\\text\{head\}\}overdir\_fracfor the previous\-token score, seven models / three positional schemes\.ρdir=ρ​\(dir\_frac,prev\)\\rho\_\{\\texttt\{dir\}\}=\\rho\(\\texttt\{dir\\\_frac\},\\text\{prev\}\),ρD=ρ​\(Dhead,prev\)\\rho\_\{D\}=\\rho\(D\_\{\\text\{head\}\},\\text\{prev\}\); partial=partial​ρ​\(Dhead,prev∣dir\_frac\)=\\mathrm\{partial\}\\,\\rho\(D\_\{\\text\{head\}\},\\text\{prev\}\\mid\\texttt\{dir\\\_frac\}\)\(Spearman of OLS residuals\); CIs: layer\-clustered bootstrap \(heads share layers/training, design effects3\.63\.6–6\.96\.9; Llama heads additionally share K\-projections within GQA groups of 4\)\.The aggregate partial is heterogeneous under learned\-absolute positions\(−0\.21/\+0\.35/\+0\.20\-0\.21/\+0\.35/\+0\.20\) because it mixes a near\-ubiquitous positive bulk gradient \(among near\-zero\-prev heads\) with the class\-level signal;the architecture\-conditional statistic is the last column— the median within\-modelDheadD\_\{\\text\{head\}\}percentile of the top\-5 previous\-token heads: bottom\-quartile under learned\-absolute and ALiBi vs top\-quintile under RoPE \(perfect model\-level separation, exact permutationp=0\.029p\{=\}0\.029; head\-levelp=2\.5×10−5p\{=\}2\.5\{\\times\}10^\{\-5\}, descriptive\)\.dk=64d\_\{k\}\{=\}64is shared by GPT\-2, OPT\-1\.3B, and Pythia\-410m, so the contrast tracks the positional scheme, not head dimension\. All RoPE partials survive BH\-FDR\.![Refer to caption](https://arxiv.org/html/2607.06621v1/x2.png)Figure 2:\(a\)Within\-modelDheadD\_\{\\text\{head\}\}percentile of each model’s top\-5 previous\-token heads: bottom\-quartile under learned\-absolute \(red\) and ALiBi \(orange\) positions, top\-quintile under RoPE \(blue\); joint Mann–Whitneyp=2\.5×10−5p\{=\}2\.5\{\\times\}10^\{\-5\}; bars mark per\-model medians\.\(b\)Attenuation ofDheadD\_\{\\text\{head\}\}’s advantage by RoPE\-phase controls grows with RoPE fraction \(hybrid estimator:61%/38%61\\%/38\\%at25%25\\%,∼98%\{\\sim\}98\\%at100%100\\%; see Table[3](https://arxiv.org/html/2607.06621#S7.T3)for estimator sensitivity — with two dose levels this is suggestive, not an established dose–response\)\.
## 7Mechanism:DheadD\_\{\\text\{head\}\}tracks the RoPE phase

RoPE pairs each head’sdkd\_\{k\}dims \(rotate\-half convention, verified\) into frequenciesθt\\theta\_\{t\}; the QK interaction per frequency is a rank\-1 complex sub\-operatorMt=wqt​wkt¯⊤M\_\{t\}=w\_\{q\}^\{t\}\\,\\overline\{w\_\{k\}^\{t\}\}^\{\\top\}, with the staticM=∑tRe​\(Mt\)M=\\sum\_\{t\}\\mathrm\{Re\}\(M\_\{t\}\)\. The*directional*\(relative\-position\-asymmetric\) content is carried by the imaginary part:

score​\(Δ\)−score​\(−Δ\)=∑t2​sin⁡\(Δ​θt\)​n⊤​Im​\(Mt\)​n\.\\mathrm\{score\}\(\\Delta\)\-\\mathrm\{score\}\(\-\\Delta\)=\\sum\_\{t\}2\\sin\(\\Delta\\theta\_\{t\}\)\\,n^\{\\top\}\\mathrm\{Im\}\(M\_\{t\}\)\\,n\.\(5\)\(Full\-score reconstruction from\{Mt,θt\}\\\{M\_\{t\},\\theta\_\{t\}\\\}matches the model to10−6/10−210^\{\-6\}/10^\{\-2\}in fp32/bf16\.\) Empirically, previous\-token heads have high rotary directional fractionrope\_imag\_frac\(correlation with prev:\+0\.53/\+0\.42/\+0\.50\+0\.53/\+0\.42/\+0\.50for 410m/1\.4B/Llama\) concentrated at*high frequencies*\(\+0\.68\+0\.68on full\-RoPE Llama—aBarberoet al\.\([1](https://arxiv.org/html/2607.06621#bib.bib6)\), Urrutiaet al\.\([19](https://arxiv.org/html/2607.06621#bib.bib10)\)replication\), andDheadD\_\{\\text\{head\}\}tracks this deployment\. Adding per\-frequency RoPE\-phase summaries as controls*attenuates*DheadD\_\{\\text\{head\}\}’s residual association substantially \(Table[3](https://arxiv.org/html/2607.06621#S7.T3)\) — most completely on full\-RoPE Llama, where the default\-estimator residual is indistinguishable from zero \(\+0\.006\+0\.006,p=0\.84p\{=\}0\.84; fully rank\-based:\+0\.075\+0\.075,p=0\.016p\{=\}0\.016\)\. Two qualifications\. First, all quantities are co\-derived from the same weights, so this is shared\-variance accounting, not causal mediation\. Second, the attenuating covariate*changes identity*with RoPE fraction: on partial\-RoPE Pythia the attenuation runs throughrope\_imag\_frac\(how*much*rotary content is directional;freq\_centroidalone attenuates nothing,\+0\.35\+0\.35\), whereas on full\-RoPE Llama it runs almost entirely throughfreq\_centroid\(*where in frequency*the directional mass sits: alone\+0\.33→−0\.00\+0\.33\\to\-0\.00, whilerope\_imag\_fracalone leaves\+0\.32\+0\.32\) — consistent with all directional content being rotary by construction at100%100\\%, shifting the informative axis from amount to location\. The pattern is consistent with fuller attenuation at higher RoPE fraction, but with two dose levels and within\-dose spread \(61%61\\%vs38%38\\%\) comparable to the dose gap, we do not press a dose–response reading\. The relative\-position kernel confirms these heads peak atΔ=−1\\Delta\{=\}\{\-\}1with RoPE\-periodic oscillation \(Fig\.[3](https://arxiv.org/html/2607.06621#S8.F3), left\)\.

Placebo controls\.Two controls calibrate what the attenuation means\.*\(i\) RoPE\-model specificity:*on all three learned\-absolute models \(no RoPE\), computing a pseudo\-rope\_imag\_fracwith a fake rotate\-half pairing over all head dims yields no prev signal \(GPT\-2−0\.14\-0\.14, OPT\-1\.3B−0\.06\-0\.06, GPT\-Neo\-1\.3B−0\.09\-0\.09; all n\.s\., vs\+0\.42\+0\.42–\+0\.53\+0\.53on RoPE models\), and adding the pseudo\-phase features barely attenuates \(44–22%22\\%\): the recipe finds nothing where RoPE is absent\.*\(ii\) Pairing specificity:*randomly permuting the rotary columns ofWQ,WKW\_\{Q\},W\_\{K\}— which leavesMM\(henceDheadD\_\{\\text\{head\}\},dir\_frac\) unchanged and preserves rotary\-*block*membership, but scrambles the pairing and frequency assignment — still attenuates3131–38%38\\%on average, so much of the attenuation is carried by rotary\-block membership alone\. The increment specific to the*true*pairing/frequency assignment is marginal on Pythia\-410m \(true60\.7%60\.7\\%vs the permutation null, empiricalp=0\.039p\{=\}0\.039,n=50n\{=\}50permutations\) and not significant on Pythia\-1\.4B \(37\.7%37\.7\\%,p=0\.12p\{=\}0\.12\)\. The mechanism is thus RoPE\-specific and largely block\-level; the finer pairing\-level reading should be held lightly\.

Table 3:Attenuation ofDheadD\_\{\\text\{head\}\}’s residual prev\-token association \(partial=partial​ρ​\(Dhead,prev∣dir\_frac\)=\\mathrm\{partial\}\\,\\rho\(D\_\{\\text\{head\}\},\\text\{prev\}\\mid\\texttt\{dir\\\_frac\}\)\) when per\-frequency RoPE\-phase summaries \(rope\_imag\_frac,freq\_centroid\) are added as controls\. “Hybrid” is the paper’s default estimator \(Spearman of OLS residuals\); “rank” is fully rank\-based, under which the Llama residual is small but nonzero \(\+0\.075\+0\.075,p=0\.016p\{=\}0\.016\)\. All quantities are deterministic functions of the same weights: this is shared\-variance accounting among co\-derived features,*not*causal mediation\.
## 8Causal confirmation: ablatingIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)

We ablate a head’s directional RoPE content by zeroingIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)\(dropping thesin⁡\(Δ​θ\)\\sin\(\\Delta\\theta\)component\), whichsymmetrizesits relative\-position kernel; the decompositionfull=symmetric\+directional\\text\{full\}=\\text\{symmetric\}\+\\text\{directional\}is verified to6×10−66\\times 10^\{\-6\}\(fp32;2×10−52\\times 10^\{\-5\}on bf16 Llama\)\. Sweeping the ablation over*all*heads of all three RoPE models, the largest induction\-loss effect landson a top\-2 prev\-scoring head every time— an identity predicted a priori: Pythia\-410m head 5\.2 \(prev rank 1/384; prev0\.950\.95,Dhead​0\.61D\_\{\\text\{head\}\}\\,0\.61\): induction CE0\.52→2\.240\.52\\to 2\.24\(∼4×\{\\sim\}4\\times; exact permutationp=1/384p\{=\}1/384\); Pythia\-1\.4B head 1\.11 \(rank 1/384\):0\.56→1\.110\.56\\to 1\.11\(∼2×\{\\sim\}2\\times;p=1/384p\{=\}1/384\); Llama\-3\-8B head L14\.H26 \(rank 2/1024; prev0\.650\.65,Dhead​0\.61D\_\{\\text\{head\}\}\\,0\.61\):0\.30→0\.910\.30\\to 0\.91\(∼3×\{\\sim\}3\\times;p=2/1024p\{=\}2/1024\) — jointly≈10−8\{\\approx\}10^\{\-8\}under random assignment\. \(Llama’s \#1 prev head sits in layer 0, too early to feed induction; the causally\-critical head is the mid\-depth prev head\.\) Effects are highly concentrated:9292–99%99\\%of heads move by\|Δ​Ind\|<0\.01\|\\Delta\\text\{Ind\}\|<0\.01; the*other*prev\>0\.5\\text\{prev\}\{\>\}0\.5heads show≈0\{\\approx\}0effects \(consistent with redundancy across a multi\-head circuit\), and a few low\-prev heads show small effects \(≤0\.17\\leq 0\.17\), suggesting downstream circuit participation the prev detector does not capture; head\-level aggregate correlations are correspondingly null\. The before/after kernel shows theΔ=−1\\Delta\{=\}\{\-\}1peak flattening \(Fig\.[3](https://arxiv.org/html/2607.06621#S8.F3), right\)\. As in §[5](https://arxiv.org/html/2607.06621#S5), causal importance is concentrated, not a smooth spectral gradient; what the ablation establishes is an*existence\-with\-predicted\-identity*claim: the rotational phaseDheadD\_\{\\text\{head\}\}reads is causally load\-bearing for positional routing in the specific heads that carry it\.

![Refer to caption](https://arxiv.org/html/2607.06621v1/x3.png)

![Refer to caption](https://arxiv.org/html/2607.06621v1/x4.png)

Figure 3:Left:corpus\-averaged relative\-position score kernels of top previous\-token heads \(Pythia\-410m\) peak atΔ=−1\\Delta\{=\}\{\-\}1with RoPE\-periodic oscillation\.Right:ablatingIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)on the induction\-feeding head 5\.2 flattens theΔ=−1\\Delta\{=\}\{\-\}1peak \(and raises induction loss4×4\\times\)\.
## 9When the imprint arises: a checkpoint natural history

The static profile \(§[6](https://arxiv.org/html/2607.06621#S6)\) is an end\-state\. To see how it arises we track 22 public checkpoints each of Pythia\-410m and Pythia\-160m \(log\-spaced steps0​…​143,0000\\ldots 143\{,\}000; the dense early grid straddles the induction onset at∼2×109\{\\sim\}2\{\\times\}10^\{9\}tokens\([18](https://arxiv.org/html/2607.06621#bib.bib13)\)\), measuring per head: the audited weight metrics \(dir\_frac,DheadD\_\{\\text\{head\}\},rope\_imag\_frac\), behavioral previous\-token and prefix\-matching scores, an in\-context\-learning proxy \(second\-copy minus first\-copy NLL on repeated\-random sequences\), and the K\-composition wiring between each model’s eventual prev head and its induction heads\. \(The “eventual prev head” is identified retrospectively at the final checkpoint; Q3 below tests — and rejects — its prospective identifiability\.\)

A three\-act history at half\-doubling checkpoint resolution, replicated across both models\(Fig\.[4](https://arxiv.org/html/2607.06621#S9.F4)\)\.*Act I \(0–0\.5B tokens\): silence at the null\.*Every head sits at the Ginibre null \(behavior0\.010\.01;rope\_imag\_frac0\.5000\.500; population medianDheadD\_\{\\text\{head\}\}0\.610\.61; K\-composition at baseline\) — step 0 doubles as an in\-vivo verification of the paper’s random\-orientation null\.*Act II \(1–4B tokens\): sharp formation\.*Previous\-token behavior jumps first \(0\.370\.37at step 512→\\to0\.950\.95at step 1000 — the same steps, and even the same0\.370\.37waypoint, in both models\), with induction behavior, the ICL proxy, and the K\-composition wiring following within the same window\. The prev head’s*within\-model rotary\-phase \(rope\_imag\_frac\) percentile*snaps from at\-or\-below chance \(0\.030\.03–0\.040\.04at step 512, mid\-formation, in both models\) to the top decile \(0\.900\.90–0\.990\.99\) by step 1000: the signature locks in*with*the behavior — at our half\-doubling resolution, neither leads\.*Act III \(4–300B tokens\): slow differential sculpting\.*The population medianDheadD\_\{\\text\{head\}\}is suppressed well below the null \(0\.61→0\.3950\.61\\to 0\.395/0\.4530\.453\) over the next100×100\\timesof training while the previous\-token heads retain it, and the absolute rotary\-phase content of the prev head consolidates slowly \(0\.500→0\.5780\.500\\to 0\.578, same endpoint in both models\)\. The static “retain\-vs\-suppress” profile of §[6](https://arxiv.org/html/2607.06621#S6)is therefore the end state of a post\-formation differentiation process\.

Pre\-registered answers\.Q1 \(lead/lag\): the percentile signature is*simultaneous*with formation; the absolute consolidation*lags*\. Q2: population suppression is a post\-formation process\. Q3 \(predictive signature\):no— before formation the eventual prev head is not reliably identifiable from its spectrum \(its percentile even sits low mid\-formation\); we report this pre\-registered negative plainly\. The natural history thus bounds what statics can claim: the rotational imprint is the fingerprint the algorithm leaves in weight space, consolidated after function — and because observation cannot order structure and function within the formation window, necessity can only be settled by intervention\.

![Refer to caption](https://arxiv.org/html/2607.06621v1/x5.png)Figure 4:Checkpoint natural history, Pythia\-410m \(blue\) and 160m \(red\); grey band = the formation window \(11–4×1094\\times 10^\{9\}tokens\)\.\(a\)previous\-token behavior forms sharply and identically in both models\.\(b\)the prev head’srope\_imag\_fracpercentile locks in*with*behavior, not before\.\(c\)the population medianDheadD\_\{\\text\{head\}\}is suppressed below the Ginibre null only*after*formation\.
## 10Is the rotational channel necessary? Constrained\-training interventions

We train 2\-layer attention\-only models \(d=128\{=\}128, 4 heads,dk=32d\_\{k\}\{=\}32\) from scratch on a per\-sequence random\-map task —xt\+1=fseq​\(xt\)x\_\{t\+1\}=f\_\{\\mathrm\{seq\}\}\(x\_\{t\}\)w\.p\.0\.90\.9, uniform noise w\.p\.0\.10\.1, withfseqf\_\{\\mathrm\{seq\}\}resampled every sequence — which forces induction \(no global memorization; no fixed\-offset shortcut\) and yields a crisp formation time\. Grid:\{\\\{APE, RoPE\}×\{\\\}\\times\\\{free; sym\-MM: penalize‖MA‖F2/‖M‖F2\\\|M\_\{A\}\\\|\_\{F\}^\{2\}/\\\|M\\\|\_\{F\}^\{2\};Im​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)\-suppressed \(RoPE only\): penalize∑t∥Im\(Mt\)∥F2/∑t\(∥Re∥2\+∥Im∥2\)\}\\sum\_\{t\}\\\|\\mathrm\{Im\}\(M\_\{t\}\)\\\|\_\{F\}^\{2\}/\\sum\_\{t\}\(\\\|\\mathrm\{Re\}\\\|^\{2\}\{\+\}\\\|\\mathrm\{Im\}\\\|^\{2\}\)\\\},n=5n\{=\}5seeds\. A load\-bearing algebraic point shapes the grid:symmetrizing the staticMMdoes not zeroIm​\(Mt\)\\mathrm\{Im\}\(M\_\{t\}\)\(the static operator contains only theRe\\mathrm\{Re\}parts,M=∑tRe​\(Mt\)\+Mnon\-rotM=\\sum\_\{t\}\\mathrm\{Re\}\(M\_\{t\}\)\+M\_\{\\text\{non\-rot\}\}\), so the two constraints dissociate the static\-antisymmetry and phase channels\. Both penalties enforce hard \(finalIm\\mathrm\{Im\}share0\.0000\.000;dir\_frac0\.0040\.004–0\.0060\.006\) and no arm loses final capability \(all reach the task floor with strong prev and induction heads\)\.

Results against the pre\-registered predictions\(Fig\.[5](https://arxiv.org/html/2607.06621#S10.F5); Table[4](https://arxiv.org/html/2607.06621#S10.T4)\)\. P1 \(free base rates\) holds: APE forms fastest \(600±0600\\pm 0steps\), RoPE free at940±55940\\pm 55\. P2’s strong form isfalsified: suppressing the phase to zero delays formation only∼1\.4×\{\\sim\}1\.4\\timesand the circuit*reroutes*— the prev head re\-forms withrope\_imag\_frac=0\.005\\texttt\{rope\\\_imag\\\_frac\}\{=\}0\.005and a qualitatively different \(cos\-only\) relative\-position kernel whose peak still lands atΔ=−1\\Delta\{=\}\{\-\}1\. P3 isfalsified in the opposite direction, our largest surprise: forcing symmetry on APE is the*costliest*constraint in the grid \(600→1740±241600\\to 1740\\pm 241,2\.9×2\.9\\times\) — the fast APE solution is*antisymmetric*embedding\-matching, and the symmetric variant \(nearby\-kernel\+\+causal mask\), while reachable, is much harder to find; trained learned\-absolute LLMs end at symmetric profiles \(§[6](https://arxiv.org/html/2607.06621#S6)\) with1000×1000\\timesmore data, and the toy exposes the search cost of that end state\. P4’s formation\-time version fails \(RoPE\+sym delays1\.66×1\.66\\times\) but itsmechanistic dissociation lands exactly: the sym\-arm’s prev head carries*full*phase content \(rope\_imag\_frac​0\.52\\texttt\{rope\\\_imag\\\_frac\}\\ 0\.52\) through a*fully symmetric*static operator \(dir\_frac​0\.004\\texttt\{dir\\\_frac\}\\ 0\.004–0\.0050\.005\), and its positional kernel is indistinguishable from the free arm’s — directional attention through a symmetricMM, which is impossible under absolute positions\.

Table 4:Constrained\-training formation delays \(n=5n\{=\}5seeds per cell; one\-sided exact Mann–Whitney, BH\-FDR over the four pre\-registered contrasts\)\. Every constraint delays formation significantly; none blocks it\.![Refer to caption](https://arxiv.org/html/2607.06621v1/x6.png)Figure 5:Constrained\-training interventions\.\(a\)induction\-capability curves: every arm reaches the floor; constraints delay the drop\.\(b\)formation times \(n=5n\{=\}5\)\.\(c\)the dissociation: under sym\-MMthe prev head keeps full phase content \(rope\_imag\_frac\) with a fully symmetric staticMM\(dir\_frac≈0\{\\approx\}0\); under Im\-suppression the reverse\.\(d\)positional kernels: sym\-MMis indistinguishable from free \(the kernel is carried by the untouched phase\); Im\-suppression reshapes the kernel but its peak stays atΔ=−1\\Delta\{=\}\{\-\}1\(reroute\)\.Verdict\.No spectral channel is*necessary*— the solution space is degenerate and training reroutes around every constraint we imposed — but each constraint carries a significant, quantifiable search cost \(Table[4](https://arxiv.org/html/2607.06621#S10.T4)\), and the cost structure identifies each scheme’s*default*solution\. With §[5](https://arxiv.org/html/2607.06621#S5)and §[8](https://arxiv.org/html/2607.06621#S8)this completes a three\-way distinction the field often conflates:*trained\-circuit dependence*\(post\-hoc ablations are catastrophic\),*developmental preference*\(defaults are found much faster\), and*necessity*\(nothing here is strictly necessary\)\. A companion note\([11](https://arxiv.org/html/2607.06621#bib.bib22)\)tests the constructive converse — ban\-free*assistance*— and finds it selects the same implementations far more cheaply than banning them \(1\.3×1\.3\\timesvs\.2\.9×2\.9\\timesfor the antisymmetric\-to\-symmetric flip\) while solution\-specific initialization accelerates formation outright, evidence that this spectral economy is not only priced but steerable\.

## 11Discussion

What earns its keep, and where\.The plain symmetric/antisymmetric split ofMMis an architecture\-general descriptor of head function \(§[4](https://arxiv.org/html/2607.06621#S4)\) and, causally, the symmetric part is the workhorse \(§[5](https://arxiv.org/html/2607.06621#S5)\)\. The*complex\-eigenvalue refinement*is architecture\-conditional at head level: the previous\-token solution is spectrally non\-rotational \(content\-like\) under learned\-absolute positions and rotational under RoPE, whereDheadD\_\{\\text\{head\}\}tracks the rotational phase whose causal load §[8](https://arxiv.org/html/2607.06621#S8)establishes head\-locally \(§[6](https://arxiv.org/html/2607.06621#S6)–[8](https://arxiv.org/html/2607.06621#S8)\)\. This gives the recent “non\-Hermitian transformer” program a concrete, controlled home—the complex structure of the*QK operator*does measurable, causal work precisely when position is encoded as complex phase—while honestly bounding it: on GPT\-2 the non\-Hermitian label is decorative\.

Practical reading\.For weight\-only head triage: under RoPE,rope\_imag\_frac\(or theDheadD\_\{\\text\{head\}\}percentile\) flags positional\-routing heads without a forward pass; under absolute/ALiBi schemes the same statistics are*misleading in aggregate*\(bulk gradients\) and only the strong\-head profile is informative\. For interpretability methodology, the dynamics result is a caution: a clean static weight–function correspondence need not be predictive during training, and post\-hoc ablation severity must not be read as developmental necessity\.

Companion work \(context only\)\.Companion work in preparation analyzes the*iterated*attention propagator with pseudospectral tools under mask\-structure nulls; none of this paper’s claims relies on it\. We flag two of its directions as context: a division of labor consistent with our tool choice \(eigenvalue\-level summaries suffice for the static scoring form; resolvent\-level tools are needed for the depth\-iterated propagator\), and a state\-dependent character of induction heads that static weight fingerprints cannot capture — consistent with their content\-like static profile in §[4](https://arxiv.org/html/2607.06621#S4)\.

Limitations\.\(i\) The RoPE evidence in §[6](https://arxiv.org/html/2607.06621#S6)–[7](https://arxiv.org/html/2607.06621#S7)is correlational \(shared\-variance\) over heads; §[8](https://arxiv.org/html/2607.06621#S8)supplies causation but as a concentrated per\-head effect \(now on all three RoPE models\)\. \(ii\) Three learned\-absolute models are tested with a consistent strong\-head profile but heterogeneous aggregate and moderate\-band statistics, whose sources \(training corpus; GPT\-Neo’s alternating local attention\) are unresolved; the ALiBi scheme is covered by a single model \(BLOOM\-1b1\)\. \(iii\) Inference treats heads as units: heads share layers and training \(layer\-clustered design effects3\.63\.6–6\.96\.9; we report clustered CIs\), and Llama heads share K\-projections within GQA groups of 4 \(between\-group variance0\.550\.55–0\.790\.79\) — group\-level dependence is only partially resolved\. \(iv\) Llama runs in bf16 \(float64 metrics from bf16 weights are exact w\.r\.t\. the model; the diagonal convention check matches to bf16 precision only\)\. \(v\) Single\-head ablations miss composition; the prev→\\toinduction wiring is verified by K\-composition \(§[4](https://arxiv.org/html/2607.06621#S4)\) but full path patching is future work\. \(vi\) QK biases lie outsideMMand are held fixed in ablations; the prev\-detector’s bulk\-ordering reliability is unquantified\. \(vii\) A non\-Llama full\-RoPE family \(Qwen/Mistral\) would broaden the third architecture point\.

## 12Conclusion

For the attention QK operator, the plain antisymmetric split describes head function architecture\-generally; the complex\-eigenvalue refinement earns its keep only under RoPE, where it tracks the rotational phase whose causal load head\-level ablations establish\. Tracking training shows this spectral signature is born at the random\-matrix null, locks in with circuit formation, and is sculpted into today’s profiles after function arrives; constrained training shows it is a*default*, not a necessity — every spectral channel we blocked was rerouted around, at a significant and interpretable search cost\. The non\-Hermitian view of attention is neither uniformly profound nor uniformly decorative:*the positional scheme sets the default spectral algebra of attention’s solutions*, and we measure that claim statically, dynamically, and causally in the settings analyzed\. Scaled constrained pretraining is its direct falsification path\.

### Reproducibility & pre\-registration\.

Code insrc/:extract,decompose,metrics,observables,semantics,p4\_rope,rope,p5\_rope\_ablation,p9\_checkpoint\_dynamics,p10\_training\_intervention\. Per\-head tables:results/cache/\*\_head\_full\.parquet; checkpoint trajectories and training histories underresults/cache/\. The dynamics questions \(Q1–Q3\) and intervention predictions \(P1–P4\) were pre\-registered in the project plan before data collection; falsified predictions \(Q3, P2\-strong, P3, P4\-timing\) are reported as such\. The full artifact bundle — code, per\-head tables, checkpoint trajectories, training histories, figure scripts, and the pre\-registration documents — is released at[https://github\.com/HengyuLi\-Ozaki\-lab/qk\-spectral\-fingerprint](https://github.com/HengyuLi-Ozaki-lab/qk-spectral-fingerprint); the repository’s version history stamps which analyses were specified before their data existed\.

## References

- \[1\]F\. Barbero, A\. Vitvitskyi, C\. Perivolaropoulos, R\. Pascanu, and P\. Veličković\(2025\)Round and round we go\! what makes rotary positional encodings useful?\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.06205Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p4.2),[§7](https://arxiv.org/html/2607.06621#S7.p1.21)\.
- \[2\]A\. Bietti, V\. Cabannes, D\. Bouchacourt, H\. Jégou, and L\. Bottou\(2023\)Birth of a transformer: a memory viewpoint\.InNeurIPS,Note:arXiv:2306\.00802Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1)\.
- \[3\]P\. Chang\(2026\)From embeddings to dyson series: transformer mechanics as non\-hermitian operator theory\.Note:arXiv:2603\.11322Cited by:[§1](https://arxiv.org/html/2607.06621#S1.p2.1),[§2](https://arxiv.org/html/2607.06621#S2.p3.3)\.
- \[4\]A\. Chen, R\. Shwartz\-Ziv, K\. Cho, M\. L\. Leavitt, and N\. Saphra\(2024\)Sudden drops in the loss: syntax acquisition, phase transitions, and simplicity bias in mlms\.InICLR,Note:arXiv:2309\.07311Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1)\.
- \[5\]M\. Courtois, M\. Ostendorff, L\. Hennig, and G\. Rehm\(2024\)Symmetric dot\-product attention for efficient training of BERT language models\.InFindings of ACL,Note:arXiv:2406\.06366Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p6.2)\.
- \[6\]N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread\.Note:[https://transformer\-circuits\.pub/2021/framework/index\.html](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§1](https://arxiv.org/html/2607.06621#S1.p1.9),[§2](https://arxiv.org/html/2607.06621#S2.p1.2)\.
- \[7\]J\. Fernando and G\. Guitchounts\(2026\)Dynamics of the transformer residual stream: coupling spectral geometry to network topology\.Note:arXiv:2605\.14258Cited by:[§1](https://arxiv.org/html/2607.06621#S1.p2.1),[§2](https://arxiv.org/html/2607.06621#S2.p3.3)\.
- \[8\]Huanget al\.\(2026\)Dissecting multimodal in\-context learning: modality asymmetries and circuit dynamics in modern transformers\.Note:arXiv:2601\.20796Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p6.2)\.
- \[9\]S\. Jamil and R\. Kapadia\(2026\)The routing and filtering structure of attention\.Note:arXiv:2605\.18826; concurrent workCited by:[§1](https://arxiv.org/html/2607.06621#S1.p2.1),[§2](https://arxiv.org/html/2607.06621#S2.p3.3)\.
- \[10\]N\. Kitaev, L\. Kaiser, and A\. Levskaya\(2020\)Reformer: the efficient transformer\.InICLR,Note:arXiv:2001\.04451Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p6.2)\.
- \[11\]H\. Li\(2026\)Steering is cheaper than banning: assistance\-based selection of attention’s spectral solutions\.Note:Companion note; preprintCited by:[§10](https://arxiv.org/html/2607.06621#S10.p3.2)\.
- \[12\]B\. Millidge and S\. Black\(2022\)The singular value decompositions of transformer weight matrices are highly interpretable\.Note:AI Alignment Forum / Conjecture[https://www\.alignmentforum\.org/posts/mkbGjzxD8d8XqKHzA](https://www.alignmentforum.org/posts/mkbGjzxD8d8XqKHzA)Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p1.2)\.
- \[13\]C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph,et al\.\(2022\)In\-context learning and induction heads\.Transformer Circuits Thread\.Note:arXiv:2209\.11895Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1),[§2](https://arxiv.org/html/2607.06621#S2.p7.1)\.
- \[14\]G\. Reddy\(2024\)The mechanistic basis of data dependence and abrupt learning in an in\-context classification task\.InICLR,Note:arXiv:2312\.03002Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1)\.
- \[15\]M\. Saponati, P\. Sager, P\. V\. Aceituno, T\. Stadelmann, and B\. Grewe\(2025\)The underlying structures of self\-attention: symmetry, directionality, and emergent dynamics in transformer training\.InInternational Conference on Machine Learning \(ICML\),Note:arXiv:2502\.10927Cited by:[item 2](https://arxiv.org/html/2607.06621#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2607.06621#S1.p2.1),[§2](https://arxiv.org/html/2607.06621#S2.p2.2),[§2](https://arxiv.org/html/2607.06621#S2.p5.1),[§4](https://arxiv.org/html/2607.06621#S4.p1.8)\.
- \[16\]A\. K\. Singh, T\. Moskovitz, F\. Hill, S\. C\.Y\. Chan, and A\. M\. Saxe\(2024\)What needs to go right for an induction head? a mechanistic study of in\-context learning circuits and their formation\.InICML,Note:arXiv:2404\.07129Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1)\.
- \[17\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing\.Note:arXiv:2104\.09864 \(2021\)Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p4.2)\.
- \[18\]C\. Tigges, M\. Hanna, Q\. Yu, and S\. Biderman\(2024\)LLM circuit analyses are consistent across training and scale\.InNeurIPS,Note:arXiv:2407\.10827Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p5.1),[§9](https://arxiv.org/html/2607.06621#S9.p1.3)\.
- \[19\]F\. Urrutia, J\. Salas, A\. Kozachinskiy, C\. B\. Calderon, H\. Pasten, and C\. Rojas\(2025\)Decoupling positional and symbolic attention behavior in transformers\.Note:arXiv:2511\.11579Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p4.2),[§7](https://arxiv.org/html/2607.06621#S7.p1.21)\.
- \[20\]K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt\(2023\)Interpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2211\.00593Cited by:[§2](https://arxiv.org/html/2607.06621#S2.p7.1)\.