单头注意力中长度泛化的位置可实现性理论

# A Positional Realizability Theory of Length Generalizationin Single-Head Attention
Source: [https://arxiv.org/html/2610.41218](https://arxiv.org/html/2610.41218)
Jiasong Hanhanjiasongdhf@outlook\.comYunkai Chaichaiyunkai1@gmail\.comAffiliation:School of Information and Communication Engineering, Shanghai Jiao Tong UniversityJintao Wangwangjintao@tsinghua\.edu\.cnAffiliation:Department of Electronic Engineering, Tsinghua University

###### Abstract

Why does length generalization \(LG\) depend so sharply on the positional encoding \(PE\)? Relative schemes extrapolate fixed offsets, absolute schemes fix positions, and neither handles growing offsets\. We give a*realizability theory*for a deliberately*controlled*abstraction, a single positional\-only attention head, that*derives*this well\-documented empirical pattern and says precisely where it holds and breaks\. The device is a*simulation transfer operator*TLT\_\{L\}that re\-evaluates a selected length\-L0L\_\{0\}\-realizing positional kernel on the longer grid, with a scalarαL∈\[0,1\]\\alpha\_\{L\}\\in\[0,1\]read off*before any training*from\(Φ,tn\)\(\\Phi,t\_\{n\}\)alone\. HereαL\\alpha\_\{L\}serves as the computable*witness*of that taxonomy’s derivation, not as a stronger empirical predictor\. Our results are arelative\-head deep\-row characterizationfor kernels with a strict unique global maximizer \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\), anabsolute/relative dualitythat derives the pattern \(Proposition[4](https://arxiv.org/html/2610.41218#Thmproposition4)\), alearned\-absolute underdeterminationresult, and astatic expressivity bound; a sufficiency theorem \(Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\) bridgesαL\\alpha\_\{L\}to LG, and the general deterministic case is a precise open conjecture \(correcting a tempting but invalid “outside\-colspan” argument\)\. Experiments confirm the structure: in the controlled single\-head regimeαL\\alpha\_\{L\}*exactly separates*the taxonomy’s generalizing and non\-generalizing \(PE, task\) cells \(26252625trained\-model measurements; stable at vocab6464/d=128d\{=\}128\)\. Together, these results turn the received PE taxonomy into a*positional\-kernel transfer theorem*, and identify the content and depth mechanisms that escape it\. The theory concerns the positional channel alone: tasks whose targets depend on token content, including the standard length\-generalization suite such as addition and sorting, fall outside Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\.

## 1Introduction

Length generalization \(LG\), training on short sequences and testing on longer ones, is a sharp probe of whether a transformer has learned an*algorithm*rather than a length\-specific shortcut, and a practical bottleneck for tasks from arithmetic to long\-context reasoning\([Anil et al\., 2022](https://arxiv.org/html/2610.41218#bib.bib1);[Jelassi et al\., 2023](https://arxiv.org/html/2610.41218#bib.bib11);[Zhou et al\., 2024b](https://arxiv.org/html/2610.41218#bib.bib25)\)\. A large body of empirical work has distilled a clear and consistent picture: LG depends acutely on the positional encoding \(PE\)\. Learned\-absolute PEs typically fail, relative schemes such as RoPE and ALiBi extrapolate*fixed*relative offsets, absolute schemes lock*fixed*absolute positions, and*growing*\-offset tasks such as copy and reverse defeat essentially everything\([Kazemnejad et al\., 2023](https://arxiv.org/html/2610.41218#bib.bib12);[Press et al\., 2022](https://arxiv.org/html/2610.41218#bib.bib15);[Su et al\., 2024](https://arxiv.org/html/2610.41218#bib.bib19);[Ruoss et al\., 2023](https://arxiv.org/html/2610.41218#bib.bib16);[Wang et al\., 2024](https://arxiv.org/html/2610.41218#bib.bib22)\)\. Existing theory captures pieces of this picture either qualitatively \(the RASP\-L conjecture,[Zhou et al\., 2024a](https://arxiv.org/html/2610.41218#bib.bib24); the C\-RASP framework,[Huang et al\., 2025](https://arxiv.org/html/2610.41218#bib.bib9)\) or through bounds evaluated on the trained model\([Izzo et al\., 2025](https://arxiv.org/html/2610.41218#bib.bib10)\)\.What is missing is an account that*derives*this picture from the attention mechanism itself: a quantity, computable from the PE and the task*before any training*, that says which \(PE, task\) pairs can length\-generalize and why\.

##### The quantity, in one example \(Figure[1](https://arxiv.org/html/2610.41218#S1.F1)\)\.

Fit a single positional attention head to the prev\-token target at training lengthL0L\_\{0\}: it attends from each position to the one before it\. Now re\-evaluate the same kernel on a longer grid and count the fraction of new positions whose attention still peaks on the previous token\. For RoPE this fraction is11\(the relative kernel is length\-independent\); for a copy task, whose target offset grows with the sequence, it is00\. That fraction is the*simulation alignment*αL∈\[0,1\]\\alpha\_\{L\}\\in\[0,1\], the object we study\. It is read off from the PE’s feature map and the task’s target function alone, with*no end\-to\-end Transformer training, no data samples, and no learned Transformer parameters*\(only a deterministic kernel\-selection on the PE basis\)\. It is the computable witness of a*simulation transfer operator*TLT\_\{L\}that re\-evaluates the selected length\-L0L\_\{0\}\-realizing kernel on the length\-LLgrid \(§[4](https://arxiv.org/html/2610.41218#S4)\)\. It is model\-free and data\-free, but*selection\-rule dependent*: the value depends on which realizing kernel theL0L\_\{0\}\-fit selects \(§[4](https://arxiv.org/html/2610.41218#S4)\)\.

Such a pre\-training, data\-free quantity is useful in two ways\. Practically, it lets one triage \(PE, task\) choices before spending any compute\. Scientifically, which is our focus and the more durable contribution, it converts the empirical PE pattern from a list of observations into a*derived consequence*of what a single attention head can realize and transfer across lengths, with theorems that say exactly where the pattern holds and proofs of where it must break\.

Training\-free:αL\\alpha\_\{L\}is computed from the positional feature mapΦ\\Phiand the task targettnt\_\{n\}*alone*, with no Transformer training, no data samples, and no learned Transformer parameters\.Object certified:the*fitted positional kernel*\(this pipeline, Thm[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\)\.Object measured:the*trained Transformer*\(§[6](https://arxiv.org/html/2610.41218#S6)\)\.Gap:selection / content / depth \(§[7](https://arxiv.org/html/2610.41218#S7)\)\.1\. Fit kernel atL0L\_\{0\}required patternRL0R\_\{L\_\{0\}\}\(here prev\-token,δ=1\\delta\{=\}1\);fit any kernelΦL0​W​ΦL0⊤\\Phi\_\{L\_\{0\}\}W\\Phi\_\{L\_\{0\}\}^\{\\\!\\top\}realizing it2\. TransferTLT\_\{L\}sameWW, longer grid:PL=ΦL​W​ΦL⊤P\_\{L\}=\\Phi\_\{L\}W\\Phi\_\{L\}^\{\\\!\\top\}no retraining3\. Simulate at lengthLLold: argmax kept \(Thm[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\)new rows\[L0,L\)\[L\_\{0\},L\)✓✓✓4\. Read offαL\\alpha\_\{L\}0011αL=\#​\{correct\}\#​\{scored new rows\}\\alpha\_\{L\}=\\dfrac\{\\\#\\\{\\text\{correct\}\\\}\}\{\\\#\\\{\\text\{scored new rows\}\\\}\}αL→1\\alpha\_\{L\}\\\!\\to\\\!1: generalizesαL→0\\alpha\_\{L\}\\\!\\to\\\!0: fails

Figure 1:Overview of the method\.Select \(fit\) a single positional head’s kernel at lengthL0L\_\{0\}realizing the task’s target patternRL0R\_\{L\_\{0\}\};*transfer*the same kernel to a longer grid viaTLT\_\{L\}\(re\-evaluateΦL​W​ΦL⊤\\Phi\_\{L\}W\\Phi\_\{L\}^\{\\top\}, no retraining\); old rows keep their argmax by the causal\-mask identity \(Thm[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\); score the new rows\[L0,L\)\[L\_\{0\},L\)to read offαL∈\[0,1\]\\alpha\_\{L\}\\in\[0,1\]\. The whole pipeline uses only\(Φ,tn\)\(\\Phi,t\_\{n\}\), with no Transformer training, no data samples, and no learned Transformer parameters \(only a deterministic kernel\-selection on the PE basis\)\.We prove thatαL=1\\alpha\_\{L\}=1implies exact length generalization \(Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\), and we characterize precisely when relative encodings reach it \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\): the required offset must settle on a single value on all but a vanishing fraction of the “deep” rows of the attention grid, and that value must be attainable as a strict maximizer inside the encoding’s own kernel family\. So prev/skip\-kkgeneralize while copy and reverse provably cannot \(for kernels with a strict unique global maximizer\), recovering the received taxonomy as a theorem\. Empirically, across26252625trained\-model measurements,αL\\alpha\_\{L\}cleanly separates the \(PE, task\) pairs that length\-generalize from those that do not\. We are deliberate about scope: we study a*controlled abstraction*, the positional channel*in isolation*, precisely so the theory can be exact\. Content and depth escapes are explicitly mapped \(§[6](https://arxiv.org/html/2610.41218#S6)–[7](https://arxiv.org/html/2610.41218#S7)\); e\.g\. an induction task length\-generalizes through the content channel exactly where positionalαL=0\\alpha\_\{L\}\{=\}0\. We aim to characterize the positional channel’s realizability and to locate where it ceases to be the operative mechanism; explaining all of Transformer LG is outside our scope\.

##### Contributions\.

1. 1\.A realizability framework\.We cast LG as kernel transfer: a*simulation transfer operator*TLT\_\{L\}re\-evaluates the selected length\-L0L\_\{0\}\-realizing kernel at lengthLL, and a scalarαL\\alpha\_\{L\}\(§[4](https://arxiv.org/html/2610.41218#S4)\) reads off realizability from\(Φ,tn\)\(\\Phi,t\_\{n\}\)alone, using no end\-to\-end Transformer training, no data samples, and no learned Transformer parameters \(only a deterministic kernel\-selection\)\.
2. 2\.Theory: a characterization of single\-head positional LG\.The core results are a*deep\-row characterization*for relative heads \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2): among kernels with a strict unique global maximizer, a relative family transfers a task iff the required offset is constant on a density\-one set of deep rows and that offset is attainable within the family, so prev/skip\-kkgeneralize and copy/reverse do not\), which subsumes growing\-offset impossibility and an*absolute/relative duality*\(Proposition[4](https://arxiv.org/html/2610.41218#Thmproposition4)\) that*derives*the empirical PE pattern; a*learned\-absolute underdetermination*result \(Proposition[2](https://arxiv.org/html/2610.41218#Thmproposition2)\); and a*static expressivity bound*\(Proposition[1](https://arxiv.org/html/2610.41218#Thmproposition1)\)\. A*sufficiency theorem*\(Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\) is an elementary but essential*bridge*that makesαL\\alpha\_\{L\}meaningful: what it establishes is that the earlier, already\-seen rows keep the same masked attention pattern at longer lengths\. Only the general deterministic\-PE case remains a precise*conjecture*\(Conjecture[1](https://arxiv.org/html/2610.41218#Thmconjecture1); we correct a tempting but invalid argument\)\. Several of the formal claims are accompanied by executable checks in the supplement; App\.[J](https://arxiv.org/html/2610.41218#A10)lists which\.
3. 3\.Empirical confirmation and scope\.Across26252625trained\-model measurements the theory’s structure holds:αL\\alpha\_\{L\}separates generalizing from non\-generalizing \(PE, task\) cells with AUC=1\.0=1\.0\(robust to vocab/width\)\. We delineate the boundary precisely \(§[6](https://arxiv.org/html/2610.41218#S6), §[7](https://arxiv.org/html/2610.41218#S7)\): where the*content*channel carries the target \(ALiBi prev, induction\) LG occurs without positional realizability, andαL\\alpha\_\{L\}under\-predicts low\-frequency relative PEs\. Tasks whose targets depend on token content, including the standard length\-generalization suite such as addition and sorting\([Kazemnejad et al\., 2023](https://arxiv.org/html/2610.41218#bib.bib12)\), therefore fall outside Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\. As a practical predictorαL\\alpha\_\{L\}*matches but does not beat*the coarse taxonomy it explains \(we searched for a winning cell and report the negative\); its contribution lies in the theory, and we do not frame it as a benchmark comparison\.

Organization\.§[2](https://arxiv.org/html/2610.41218#S2)contrasts prior LG theory; §[3](https://arxiv.org/html/2610.41218#S3)–[4](https://arxiv.org/html/2610.41218#S4)set up the abstraction, the operatorTLT\_\{L\}, andαL\\alpha\_\{L\}; §[5](https://arxiv.org/html/2610.41218#S5)proves the results; §[6](https://arxiv.org/html/2610.41218#S6)confirms them empirically and maps the boundary where token content identifies the target; §[7](https://arxiv.org/html/2610.41218#S7)states scope and limitations\.

## 2Related Work

Positional encodings for LG\.[Kazemnejad et al\. \(2023\)](https://arxiv.org/html/2610.41218#bib.bib12)benchmark PEs and find NoPE competitive;[Wang et al\. \(2024\)](https://arxiv.org/html/2610.41218#bib.bib22)analyze why causal Transformers extrapolate without explicit PEs; ALiBi\([Press et al\., 2022](https://arxiv.org/html/2610.41218#bib.bib15)\)and RoPE\([Su et al\., 2024](https://arxiv.org/html/2610.41218#bib.bib19)\)bias attention by relative position;[Ruoss et al\. \(2023\)](https://arxiv.org/html/2610.41218#bib.bib16)boost LG with randomized PEs, and[Cho et al\. \(2024\)](https://arxiv.org/html/2610.41218#bib.bib2)via task\-structured position coupling; arithmetic and reasoning benchmarks drive much of this work\([Anil et al\., 2022](https://arxiv.org/html/2610.41218#bib.bib1);[Jelassi et al\., 2023](https://arxiv.org/html/2610.41218#bib.bib11);[Zhou et al\., 2024b](https://arxiv.org/html/2610.41218#bib.bib25)\)\. These are empirical or architectural; we ask the*a priori*question of which \(PE, task\) pairs can extrapolate at all\.Theory of LG\.RASP\-L\([Zhou et al\., 2024a](https://arxiv.org/html/2610.41218#bib.bib24)\)conjectures LG holds iff a length\-independent RASP program exists, a discrete and qualitative criterion\.[Izzo et al\. \(2025\)](https://arxiv.org/html/2610.41218#bib.bib10)prove a simulation sufficiency bound on the required*training length*, evaluated on trained models, and explicitly leave open the*a priori*PE\-ranking question\. That is exactly our target, which we hit with a continuous PE\-computable quantity\.[Huang et al\. \(2025\)](https://arxiv.org/html/2610.41218#bib.bib9)give the C\-RASP expressibility framework \(task\-level, PE\-class\-bound\);[Golowich et al\. \(2025\)](https://arxiv.org/html/2610.41218#bib.bib5)tie LG to attention sparsity\. Against this line, our Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\(relative kernels with a strict unique global maximizer cannot length\-generalize growing offsets\) overlaps the RASP\-L/C\-RASP intuition that copy/reverse are hard; its*novel*content is thetie\-rule\-independentdeep\-row argument and thecapacity corollary\(acctrain<1\\mathrm\{acc\_\{train\}\}<1under any training\-length distribution with unbounded support\), neither of which those frameworks state\.Spectral views\.[Gu et al\. \(2025\)](https://arxiv.org/html/2610.41218#bib.bib6)analyze PE spectra of*trained*models and prove no impossibility; ourTLT\_\{L\}is a*pre\-training*operator yielding a sufficiency theorem, a relative\-encoding necessity theorem, and a \(conjectural\) general necessary condition\.Expressivity, RASP, and circuits\.Our impossibility results sit in the lineage of attention expressivity bounds: RASP\([Weiss et al\., 2021](https://arxiv.org/html/2610.41218#bib.bib23)\)\(which RASP\-L and C\-RASP build on\), the length\-dependent self\-attention limits of[Hahn \(2020\)](https://arxiv.org/html/2610.41218#bib.bib8), the constant\-depth threshold\-circuit view that grounds our static bound\([Merrill et al\., 2022](https://arxiv.org/html/2610.41218#bib.bib13)\), and the single\-head capacity limits of[Sanford et al\. \(2023\)](https://arxiv.org/html/2610.41218#bib.bib17); the content/depth escape routes we note are the induction\-head and data\-routing mechanisms of[Olsson et al\. \(2022\)](https://arxiv.org/html/2610.41218#bib.bib14);[Elhage et al\. \(2021\)](https://arxiv.org/html/2610.41218#bib.bib4);[Csordás et al\. \(2022\)](https://arxiv.org/html/2610.41218#bib.bib3)\.Implicit bias\.αL\\alpha\_\{L\}is computed at the direction a regularized cross\-entropy fit realizes\([Soudry et al\., 2018](https://arxiv.org/html/2610.41218#bib.bib18);[Gunasekar et al\., 2018](https://arxiv.org/html/2610.41218#bib.bib7)\), which for attention is the token\-selection / SVM geometry of[Tarzanagh et al\. \(2023b\)](https://arxiv.org/html/2610.41218#bib.bib21);[Tarzanagh et al\. \(2023a\)](https://arxiv.org/html/2610.41218#bib.bib20); Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)is stated for any realizingWW, so it covers this object directly \(no min\-norm idealization\)\.

##### Positioning\.

Table[1](https://arxiv.org/html/2610.41218#S2.T1)situatesαL\\alpha\_\{L\}against the closest prior frameworks\. The nearest is C\-RASP\[Pos\]\([Huang et al\., 2025](https://arxiv.org/html/2610.41218#bib.bib9)\), which is also predictive*before*training; the differences are that it is*task\-level*and binary, tied to specific PE classes, whereasαL\\alpha\_\{L\}is*keyed to the PE’s feature map*and continuous, computed at the single\-attention\-head level from\(Φ,tn\)\(\\Phi,t\_\{n\}\)\. We do not claim to answer the*a priori*question first; C\-RASP\[Pos\] does at the task level\. What we add is finer resolution at the level of the PE feature map, a continuous, not a binary verdict witness, and a kernel\-transfer*operator*that yields both the sufficiency theorem and the relative growing\-offset*necessity*theorem \(with its tie\-rule\-independent capacity corollary\) in one formalism\. Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)’s treatment of the already\-seen rows is a single\-head specialization of the simulation/transfer idea of[Izzo et al\. \(2025\)](https://arxiv.org/html/2610.41218#bib.bib10)\(fit atL0L\_\{0\}, re\-evaluate atLL\); the new pieces are the closed\-form PE\-computable witnessαL\\alpha\_\{L\}and the necessity results\. Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)in particular is a relative\-head necessity theorem with a deep\-row characterization and a tie\-rule\-independent capacity corollary, which goes beyond the qualitative observation that copy and reverse are hard\. We do not claim to be first to observe the empirical pattern; we derive it at the single\-head positional level\.

Table 1:Positioning against the closest LG frameworks\.αL\\alpha\_\{L\}’s niche: pre\-training,*keyed to the PE’s feature map*, continuous, single\-head, with matched sufficiency and relative\-head necessity \(the general necessary condition is conjectural\)\.

## 3Setup

We follow the single causal attention head abstraction\. A tokenxix\_\{i\}at positioniihas content embeddinge⁡\(xi\)∈ℝde\(x\_\{i\}\)\\in\\mathbb\{R\}^\{d\}; the PE supplies a positional feature mapφ:ℕ→ℝm\\varphi:\\mathbb\{N\}\\to\\mathbb\{R\}^\{m\}\. We model the query–key logit as separable,

ai​j=e​\(xi\)⊤​Wc​e​\(xj\)⏟content\+φ​\(i\)⊤​Wp​φ​\(j\)⏟position,a\_\{ij\}\\;=\\;\\underbrace\{e\(x\_\{i\}\)^\{\\top\}W\_\{c\}\\,e\(x\_\{j\}\)\}\_\{\\text\{content\}\}\\;\+\\;\\underbrace\{\\varphi\(i\)^\{\\top\}W\_\{p\}\\,\\varphi\(j\)\}\_\{\\text\{position\}\},\(1\)withWc∈ℝd×dW\_\{c\}\\in\\mathbb\{R\}^\{d\\times d\},Wp∈ℝm×mW\_\{p\}\\in\\mathbb\{R\}^\{m\\times m\}learnable\. We study*position\-driven*tasks: for each queryiiat lengthnn, the head must attend to a task\-determined target positiontn​\(i\)t\_\{n\}\(i\)\(e\.g\. prev\-tokent=i−1t=i\-1; copyt=i−⌈n/2⌉t=i\-\\lceil n/2\\rceil; reverse; attend\-firstt=0t=0\)\. The required pattern isRn\[i,j\]=𝟏\[j=tn\(i\)\]R\_\{n\}\[i,j\]=\\mathbf\{1\}\[j=t\_\{n\}\(i\)\], causal \(j≤ij\\leq i\)\. Scope hypotheses \(S1\) single head, one layer, positional\-only; \(S2\) “solve” means correct masked row\-argmax with margin\>0\>0; \(S3\) content\-neutrality on evaluated rows\. These are discussed and stress\-tested in §[7](https://arxiv.org/html/2610.41218#S7)\.

##### Realizable positional\-kernel space\.

Stacking featuresΦn∈ℝn×m\\Phi\_\{n\}\\in\\mathbb\{R\}^\{n\\times m\}\(rowi=φ​\(i\)⊤i=\\varphi\(i\)^\{\\top\}\), the head’s realizable positional logits form the linear space𝒦n:=\{Φn​W​Φn⊤:W∈ℝm×m\}\\mathcal\{K\}\_\{n\}:=\\\{\\Phi\_\{n\}W\\Phi\_\{n\}^\{\\top\}:W\\in\\mathbb\{R\}^\{m\\times m\}\\\}, of dimensionrn2r\_\{n\}^\{2\}withrn=rank⁡\(Φn\)r\_\{n\}=\\mathrm\{rank\}\(\\Phi\_\{n\}\)\. NoPE hasrn=1r\_\{n\}=1\(constant kernel\); learned\-absolute hasφ⁡\(i\)\\varphi\(i\)undefined fori≥L0i\\geq L\_\{0\}; sinusoidal/RoPE/ALiBi are deterministic at all lengths, with RoPE/ALiBi depending only oni−ji\-j\.

## 4The Simulation Transfer Operator andαL\\alpha\_\{L\}

A length\-L0L\_\{0\}\-trained head is a parameterW∈ℝm×mW\\in\\mathbb\{R\}^\{m\\times m\}whose positional kernel realizes the target patternRL0R\_\{L\_\{0\}\}\. Length generalization asks whether the sameWWsolves lengthLL\. We therefore define the simulation transfer operator on the head’s*parameter*WWrather than as a map on kernel*blocks*\. A block map would be ill\-defined, since a blockΦL0​W​ΦL0⊤\\Phi\_\{L\_\{0\}\}W\\Phi\_\{L\_\{0\}\}^\{\\top\}does not determineWWwhenΦL0\\Phi\_\{L\_\{0\}\}is rank\-deficient \(then a nullspace directionNNhasΦL0​N​ΦL0⊤=0\\Phi\_\{L\_\{0\}\}N\\Phi\_\{L\_\{0\}\}^\{\\top\}\{=\}0butΦL​N​ΦL⊤≠0\\Phi\_\{L\}N\\Phi\_\{L\}^\{\\top\}\\neq 0, so two heads with the sameL0L\_\{0\}kernel transfer differently\):

TL​\(W\):=ΦL​W​ΦL⊤\(the head’s kernel, re\-evaluated on the length\-​L​grid\)\.T\_\{L\}\(W\)\\;:=\\;\\Phi\_\{L\}W\\Phi\_\{L\}^\{\\top\}\\qquad\(\\text\{the head's kernel, re\-evaluated on the length\-\}L\\text\{ grid\}\)\.\(2\)A*selection rule*𝒮\\mathcal\{S\}picks a realizingW𝒮W^\{\\mathcal\{S\}\}from theL0L\_\{0\}fit\. We use a regularized cross\-entropy fit on the kernel coefficients \(400 Adam steps, weight decay10−410^\{\-4\}\), which mirrors how an attention head is trained and respects the PE\-specific constraints \(RoPE relative kernel, canonical ALiBi recency slope\); as the regularization vanishes this direction approaches the max\-margin one\([Soudry et al\., 2018](https://arxiv.org/html/2610.41218#bib.bib18)\)\. The min\-norm interpolantW∗=ΦL0\+​RL0​\(ΦL0\+\)⊤W^\{\*\}=\\Phi\_\{L\_\{0\}\}^\{\+\}R\_\{L\_\{0\}\}\(\\Phi\_\{L\_\{0\}\}^\{\+\}\)^\{\\top\}is a different special case\. The*simulated pattern*isPL:=TL​\(W𝒮\)P\_\{L\}:=T\_\{L\}\(W^\{\\mathcal\{S\}\}\), and thesimulation alignmentis

αL:=1\|𝒮L\|∑i∈𝒮L\[argmaxj≤iPL\[i,j\]=tL\(i\)\]∈\[0,1\],𝒮L:=\{query positions of the task\}∩\[L0,L\)\.\\alpha\_\{L\}\\;:=\\;\\frac\{1\}\{\|\\mathcal\{S\}\_\{L\}\|\}\\sum\_\{i\\in\\mathcal\{S\}\_\{L\}\}\\mathbf\{1\}\\\!\\left\[\\arg\\max\_\{j\\leq i\}P\_\{L\}\[i,j\]=t\_\{L\}\(i\)\\right\]\\;\\in\\;\[0,1\],\\qquad\\mathcal\{S\}\_\{L\}:=\\\{\\text\{query positions of the task\}\\\}\\cap\[L\_\{0\},L\)\.\(3\)The scored set𝒮L\\mathcal\{S\}\_\{L\}is the task’s own query region restricted to the new rows: all of\[L0,L\)\[L\_\{0\},L\)for prev/skip\-kkand the fixed\-position tasks, and the part of the second half that lies beyondL0L\_\{0\}for copy and reverse, which is where their targets are defined\. We use this one normalization everywhere, including in Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)and in the implementation; averaging instead over all of\[L0,L\)\[L\_\{0\},L\)would score rows on whichtL​\(i\)t\_\{L\}\(i\)does not exist and would change the copy/reverse values by a factor that does not vanish withLL\. The argmax in \([3](https://arxiv.org/html/2610.41218#S4.E3)\) is required to be*strict*: a row counts as correct only if the target beats every competitor by a positive margin, so an exact tie counts as a failure\. This matters for the clamped learned\-absolute encoding, whose new\-row logits tie exactly \(Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\)\.TLT\_\{L\}, and henceαL\\alpha\_\{L\}, is thus dependent on which realization is selected: different realizingWWtransfer differently, the existence\-vs\-selection gap we quantify in §[6](https://arxiv.org/html/2610.41218#S6)\. The reportedαL\\alpha\_\{L\}uses the regularized cross\-entropy section \(400 Adam steps, weight decay10−410^\{\-4\}\); an ablation \(App\.[D](https://arxiv.org/html/2610.41218#A4)\) shows it tracks LG withρ=0\.91\\rho\{=\}0\.91versusρ=0\.56\\rho\{=\}0\.56for the min\-norm interpolant, which is why we adopt it\. To be precise about what the computation does and does not involve: the kernel\-selection fit \(the regularized cross\-entropy section above\) is a deterministic convex optimization over the PE’s kernel coefficients on theL0L\_\{0\}constraint set\. It involves*no end\-to\-end Transformer training, no data samples, and no learned Transformer parameters*, onlyΦ\\Phi,tnt\_\{n\}, and the selection rule\. We therefore callαL\\alpha\_\{L\}model\-free and data\-free, but*selection\-rule dependent*, rather than “gradient\-free”\.

## 5Theory

##### One question behind the results\.

The statements below are PE\-specific, but they are all instances of a single question\. A positional encodingφ\\varphiinduces a family of realizable kernels𝒦⁡\(φ\)=\{φ​\(i\)⊤​W​φ​\(j\):W\}\\mathcal\{K\}\(\\varphi\)=\\\{\\varphi\(i\)^\{\\top\}W\\varphi\(j\):W\\\}, and a task demands, at rowiiof lengthnn, a target keytn​\(i\)t\_\{n\}\(i\)\. Writeδn​\(i\)=i−tn​\(i\)\\delta\_\{n\}\(i\)=i\-t\_\{n\}\(i\)for the required offset\. Then

*does𝒦⁡\(φ\)\\mathcal\{K\}\(\\varphi\)contain a kernel whose masked argmax matchesδn​\(i\)\\delta\_\{n\}\(i\)on all but a vanishing fraction of the new rows, at every length?*

Everything that follows answers this for one encoding class at a time\. Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)says a kernel that answers yes at lengthLLtransfers, which is what makesαL\\alpha\_\{L\}meaningful\. Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)answers it for relative families, and the answer has exactly two parts: the demanded offset must settle on a single valueδ∗\\delta^\{\*\}on a density\-one set of deep rows, andδ∗\\delta^\{\*\}must be attainable as a strict maximizer inside𝒦⁡\(φ\)\\mathcal\{K\}\(\\varphi\)\. Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)shows the affine ALiBi family fails the second part for every interior offset; Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)shows the learned\-absolute family fails it on new rows because those coordinates are untrained; Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)shows the sinusoidal family satisfies it yet theL0L\_\{0\}\-fit does not select the kernel that does; and Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)reads off the resulting relative/absolute split\. Conjecture[1](https://arxiv.org/html/2610.41218#Thmconjecture1)is the general form of the same question, for an arbitrary deterministicφ\\varphi, and it is the piece we do not have\.

###### Theorem 1\(Simulation sufficiency, for any realizingWW\)\.

Assume \(S1\)–\(S3\)\. LetW∈ℝm×mW\\in\\mathbb\{R\}^\{m\\times m\}be any positional\-kernel parameter such that \(i\)WWsolves lengthL0L\_\{0\}by argmax with marginγ0\>0\\gamma\_\{0\}\>0, and \(ii\) the alignment computed atWW,αL​\(W\)\\alpha\_\{L\}\(W\), equals11with marginγL\>0\\gamma\_\{L\}\>0on the new rows\[L0,L\)\[L\_\{0\},L\)\. ThenWWsolves the task at lengthLLwith marginmin⁡\(γ0,γL\)\\min\(\\gamma\_\{0\},\\gamma\_\{L\}\)\.

###### Proof\.

New rowsi∈\[L0,L\)i\\in\[L\_\{0\},L\)are correct with marginγL\\gamma\_\{L\}by \(ii\), sinceΦL​W​ΦL⊤\\Phi\_\{L\}W\\Phi\_\{L\}^\{\\top\}are exactly the logitsWWproduces at lengthLL\. For old rowsi<L0i<L\_\{0\}, the causal mask restricts competitors toj≤i<L0j\\leq i<L\_\{0\}, so the length\-LLlogits there are*identical*to the length\-L0L\_\{0\}logitsφ​\(i\)⊤​W​φ​\(j\)\\varphi\(i\)^\{\\top\}W\\varphi\(j\)\(keysj∈\[L0,i\]j\\in\[L\_\{0\},i\]do not exist wheni<L0i<L\_\{0\}\); by \(i\) those rows are correct with marginγ0\\gamma\_\{0\}\. Hence every row is correct\. ∎

The proof uses no property ofWWbeyond \(i\)–\(ii\); in particular it does not require the min\-norm interpolant\. This matters for the experiments: the reportedαL\\alpha\_\{L\}is computed at the regularized cross\-entropy fitWceW\_\{\\mathrm\{ce\}\}, and Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)applies verbatim toWceW\_\{\\mathrm\{ce\}\}*whenever itsαL\\alpha\_\{L\}is11*– so the theorem certifies the*fitted kernel*we actually compute \. It does not certify the trained transformer, a different object; how oftenαL​\(Wmm\)=1\\alpha\_\{L\}\(W\_\{\\mathrm\{mm\}\}\)\{=\}1co\-occurs with trained\-model LG is the empirical question of §[6](https://arxiv.org/html/2610.41218#S6)\. Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)is intentionally elementary: it isolates why the transfer scoreαL\\alpha\_\{L\}is meaningful, namely that the earlier, already\-seen rows keep the same masked attention pattern at longer lengths, and the substantive open question is which\(φ,t\)\(\\varphi,t\)make theL0L\_\{0\}\-fit reachαL=1\\alpha\_\{L\}\{=\}1\(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2), Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)answer the negative side\)\.

All three propositions below were obtained under independent adversarial verification \(numeric counterexample search\+\+proof audit\); we state their*corrected*cores and record every defect in §[7](https://arxiv.org/html/2610.41218#S7)\.

###### Proposition 1\(Affine features force endpoint argmax\)\.

Assume \(S1\)–\(S3\) withm=2m=2,φ⁡\(i\)=\[1,i\]⊤\\varphi\(i\)=\[1,i\]^\{\\top\}\. For everyWWand rowii, the logitai​j=c0​\(i\)\+c1​\(i\)​ja\_\{ij\}=c\_\{0\}\(i\)\+c\_\{1\}\(i\)\\,jis affine in the key indexjj; hence the unique masked maximizer is the right endpointj=ij=iifc1​\(i\)\>0c\_\{1\}\(i\)\>0, the left endpointj=0j=0ifc1​\(i\)<0c\_\{1\}\(i\)<0, and a flat tie \(margin00\) ifc1​\(i\)=0c\_\{1\}\(i\)=0\. Thus*no strict\-interior key0<j<i0<j<iis ever the unique positive\-margin argmax*, and any interior\-target task \(e\.g\. prev\-token,i≥2i\\geq 2\) is unrealizable at every lengthn≥3n\\geq 3for anyWW\.

This is not a strawman: canonical ALiBi’s positional basis is exactly affine in the offset,\{1,\(i−j\)\}\\\{1,\(i\-j\)\\\}with a sign\-constrained slope, so Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)is why ALiBi\-prev hasαL=0\\alpha\_\{L\}\{=\}0in our experiments \(§[6](https://arxiv.org/html/2610.41218#S6)\) – a single ALiBi head cannot place an interior argmax\.

###### Proof\.

ai​ja\_\{ij\}affine inj∈\{0,…,i\}j\\in\\\{0,\\dots,i\\\}is monotone, so its maximizer is an endpoint, unique unlessc1​\(i\)=0c\_\{1\}\(i\)=0\. An interior target0<t<i0<t<iwould needai​t−ai​0=t​c1​\(i\)\>0a\_\{it\}\-a\_\{i0\}=t\\,c\_\{1\}\(i\)\>0andai​t−ai​i=\(t−i\)​c1​\(i\)\>0a\_\{it\}\-a\_\{ii\}=\(t\-i\)\\,c\_\{1\}\(i\)\>0, forcingc1​\(i\)\>0c\_\{1\}\(i\)\>0andc1​\(i\)<0c\_\{1\}\(i\)<0simultaneously – a contradiction\. The competitor set is\{0,…,i\}\\\{0,\\dots,i\\\}at every length, so the conclusion is length\-independent\. ∎

###### Proposition 2\(Learned\-absolute underdetermination\)\.

For learned\-absolute PE withφ⁡\(i\)=vi\\varphi\(i\)=v\_\{i\}fixed fori<L0i<L\_\{0\}and i\.i\.d\. absolutely\-continuous fori≥L0i\\geq L\_\{0\}, and any non\-degenerate selection ruleW=S⁡\(v0,…,vL0−1\)W=S\(v\_\{0\},\\dots,v\_\{L\_\{0\}\-1\}\)depending on trained rows only, the new\-row argmaxMi=arg⁡maxj≤i​vi⊤​W​φ​\(j\)M\_\{i\}=\\arg\\max\_\{j\\leq i\}v\_\{i\}^\{\\top\}W\\varphi\(j\)has a law functionally*independent*of the new\-key labels\. WriteNi=\{L0≤j<i\}N\_\{i\}=\\\{L\_\{0\}\\leq j<i\\\}for the off\-diagonal new keys andk=\|\{L0≤j≤i\}\|k=\|\\\{L\_\{0\}\\leq j\\leq i\\\}\|, and assume the row’s target is one of them,tL​\(i\)∈Nit\_\{L\}\(i\)\\in N\_\{i\}, with\|Ni\|=k−1≥1\|N\_\{i\}\|=k\-1\\geq 1\. Then

Pr\[Mi=tL\(i\)∣Mi∈Ni\]=1k−1,and hencePr\[Mi=tL\(i\)\]=Pr\[Mi∈Ni\]k−1≤1k−1\.\\Pr\[M\_\{i\}=t\_\{L\}\(i\)\\mid M\_\{i\}\\in N\_\{i\}\]=\\frac\{1\}\{k\-1\},\\qquad\\text\{and hence\}\\qquad\\Pr\[M\_\{i\}=t\_\{L\}\(i\)\]=\\frac\{\\Pr\[M\_\{i\}\\in N\_\{i\}\]\}\{k\-1\}\\ \\leq\\ \\frac\{1\}\{k\-1\}\.\(4\)Extrapolation is therefore at or below chance on such rows: the selection rule carries no information about which untrained key is the target\. The hypothesistL​\(i\)∈Nit\_\{L\}\(i\)\\in N\_\{i\}is what makes the statement about the untrained coordinates; it holds for the moving targets of prev and skip\-kkoncei≥L0\+ki\\geq L\_\{0\}\+k, which is the case the proposition is used for\.

###### Proof sketch\.

For new keysg⁡\(j\)=W​vjg\(j\)=Wv\_\{j\}depends only on the independentvjv\_\{j\}; the mapvi↦arg⁡maxj​vi⊤​g​\(j\)v\_\{i\}\\mapsto\\arg\\max\_\{j\}v\_\{i\}^\{\\top\}g\(j\)is a fixed function in which the labeltL​\(i\)t\_\{L\}\(i\)never appears, andSS\(trained\-rows only\) cannot encode it\. Conditioned onWW, the off\-diagonalg⁡\(j\)g\(j\),j∈Nij\\in N\_\{i\}, are i\.i\.d\. and independent ofviv\_\{i\}, so their indices are exchangeable and the argmax is uniform onNiN\_\{i\}given that the maximum is attained there, which is \([4](https://arxiv.org/html/2610.41218#S5.E4)\)\. We assume as a hypothesis that the maximizerMiM\_\{i\}is almost surely unique\. This does not follow fromW≠0W\\neq 0, nor fromWWmerely being injective on the compared keys: for an antisymmetricWWwithW​v0=0Wv\_\{0\}=0one hasvi⊤​W​vi=vi⊤​W​v0=0v\_\{i\}^\{\\top\}Wv\_\{i\}=v\_\{i\}^\{\\top\}Wv\_\{0\}=0identically, so the self\-key and the trained key00tie for everyviv\_\{i\}\(App\.[A](https://arxiv.org/html/2610.41218#A1)\)\. ∎

*Why both hypotheses are needed\.*The probability in \([4](https://arxiv.org/html/2610.41218#S5.E4)\) is conditional because the trained \(old\) keys and the self\-keyj=ij=iare not exchangeable with the new keys: the self\-key contributes the quadraticvi⊤​W​viv\_\{i\}^\{\\top\}Wv\_\{i\}, so the global maximum need not lie inNiN\_\{i\}at all\. Monte\-Carlo confirms both parts: atL=30L=30the conditional hit rate is0\.1150\.115against1/\(k−1\)=0\.1111/\(k\-1\)=0\.111, while the unconditional rate is0\.0340\.034because the winner lands inNiN\_\{i\}only30%30\\%of the time; atL=60L=60the figures are0\.0250\.025/0\.0260\.026/0\.0160\.016with65%65\\%\(experiments/verify\_prop2\_conditional\.py\)\. The unconditional hit rate is therefore*at or below*chance\. The hypothesistL​\(i\)∈Nit\_\{L\}\(i\)\\in N\_\{i\}is equally necessary: if the target is an*old*key \(as in attend\-first, wheretL​\(i\)=0t\_\{L\}\(i\)=0\), the trained keys are precisely the ones theL0L\_\{0\}\-fit has shaped, and the hit probability can exceed1/\(k−1\)1/\(k\-1\)\. With a min\-normWWfitted on the trained rows we measurePr\[Mi=0\]\\Pr\[M\_\{i\}=0\]at up to5\.2×5\.2\\timesthe bound \(feature dimensionm=64m=64,L=128L=128; the ratio grows withmmandLLand is below11for smallmm;experiments/verify\_prop2\_hypothesis\.py\)\. That regime is governed by Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)instead\. Finally, a degenerate selection such asW=0W=0makes every logit equal, which is why almost\-sure uniqueness needs non\-degeneracy\.*This is underdetermination rather than an empty constraint cone:*a rescuingWWrealizing the target on new rows exists with positive probability \(≈20\\approx 20–35%35\\%in Monte\-Carlo\); the inductive\-bias rule is simply blind to the untrained coordinates and cannot select it\.

###### Proposition 3\(Sinusoidal: a realizing kernel exists but is not selected\)\.

For sinusoidal absolute PE with incommensurate frequencies: \(A\) for any*fixed*offsetδ\\deltaaWWrealizing it exists at all lengths \(the multi\-frequency kernel has a unique global max atd=δd=\\delta, disambiguated arbitrarily far beyond the lowest\-frequency period\); \(B\) for*growing*offsets \(copy/reverse\), no translation\-invariant kernel*with a strict unique global maximizer*tracks a length\-dependent offset \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\), soαL→0\\alpha\_\{L\}\\to 0; empirically the fitted kernel givesαL≈0\\alpha\_\{L\}\\approx 0at every length we test\.

*The qualifier in \(B\) is necessary, and the sinusoidal family itself shows why\.*A translation\-invariant kernel that is*monotone increasing*over the accessible offset range has no interior maximizer, and the causal window then hands it the largest available offset for free\. Within the sinusoidal span with the standard1616frequencies and base10410^\{4\}, the single componentκ⁡\(d\)=sin⁡\(d​θmin\)\\kappa\(d\)=\\sin\(d\\theta\_\{\\min\}\)withθmin≈1\.78⋅10−4\\theta\_\{\\min\}\\approx 1\.78\\cdot 10^\{\-4\}is increasing ford≲8833d\\lesssim 8833; atL=17665L=17665the demanded copy offset is exactly88338833and this kernel attainsαL=1\\alpha\_\{L\}=1\. This is far outside the lengths we study \(our sinusoidal experiments reachL=128L=128, where the kernel is nowhere near its turning point\), but it shows the impossibility must be stated for kernels with a global peak, not for translation invariance alone\.*In plain terms:*sinusoidal PEs are not impossible in principle for a fixed offset since a realizing kernel exists at every length\. What fails is*selection*: the training\-length fit does not pick that kernel\. We are therefore not claiming “sinusoidal fixed\-offset cannot generalize”; we are isolating where it does and does not\.*Existence≠\\neqselection\.*The transfer operator’sL0L\_\{0\}\-fitW∗W^\{\*\}does not recover the fixed\-offset rescue: sinusoidal prev decays \(αL=0\.45\\alpha\_\{L\}=0\.45atL=60L\{=\}60,0\.340\.34atL=128L\{=\}128\)\. The common intuition that a sinusoidal encoding must fail once the required offset exceeds its lowest\-frequency period is thus*false*; the correct statement is conditional, and the sub\-unit transferred accuracy is exactlyαL\\alpha\_\{L\}measuring the existence\-vs\-selection gap\.

We now give a genuine*necessity*theorem for the most important failure case – growing offsets under relative encodings – which explains theαL=0\\alpha\_\{L\}\{=\}0copy/reverse cells of RoPE/ALiBi\.

###### Theorem 2\(Relative\-head deep\-row characterization\)\.

Assume \(S1\)–\(S3\)\. Let𝒮L\\mathcal\{S\}\_\{L\}denote the scored rows \(the task’s query positions in\[L0,L\)\[L\_\{0\},L\)\), so thatαL\\alpha\_\{L\}is the fraction of𝒮L\\mathcal\{S\}\_\{L\}on which the masked argmax hits the target; we use this one normalization throughout, matching the implementation\. Fix a relative \(translation\-invariant\) family𝒦\\mathcal\{K\}of kernelsai​j=κ⁡\(i−j\)a\_\{ij\}=\\kappa\(i\-j\), and consider thoseκ∈𝒦\\kappa\\in\\mathcal\{K\}that admit a*strict unique global maximizer*c⁡\(κ\)=arg⁡maxd≥0⁡κ⁡\(d\)c\(\\kappa\)=\\arg\\max\_\{d\\geq 0\}\\kappa\(d\), under a fixed tie rule\. \(A constant kernel has no strict maximizer, so NoPE, whose family is spanned by the constant, is excluded\.\) Call a scored row*deep*ifi≥c⁡\(κ\)i\\geq c\(\\kappa\); the remaining scored rows form a setℋ⁡\(κ\)⊆\[L0,c⁡\(κ\)\)\\mathcal\{H\}\(\\kappa\)\\subseteq\[L\_\{0\},c\(\\kappa\)\)of size at mostc⁡\(κ\)−L0c\(\\kappa\)\-L\_\{0\}, a bound independent ofLL\. Assume the scored set grows,\|𝒮L\|→∞\|\\mathcal\{S\}\_\{L\}\|\\to\\infty, and write𝒦0\\mathcal\{K\}\_\{0\}for the kernels in𝒦\\mathcal\{K\}that*realize the task atL0L\_\{0\}*with positive margin, which is the hypothesis Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)needs in order to turnαL\\alpha\_\{L\}into length generalization\. Thensomeκ∈𝒦0\\kappa\\in\\mathcal\{K\}\_\{0\}with a strict unique global maximizer achievesαL→1\\alpha\_\{L\}\\to 1if and only if there is an offsetδ∗\\delta^\{\*\}with

*\(i\) density:*\#⁡\{i∈𝒮L:δL​\(i\)≠δ∗\}\#​𝒮L→L→∞0,*\(ii\) attainability:*δ∗=c\(κ\)for someκ∈𝒦0\.\\text\{\\emph\{\(i\) density:\}\}\\quad\\frac\{\\\#\\\{i\\in\\mathcal\{S\}\_\{L\}:\\ \\delta\_\{L\}\(i\)\\neq\\delta^\{\*\}\\\}\}\{\\\#\\mathcal\{S\}\_\{L\}\}\\xrightarrow\[L\\to\\infty\]\{\}0,\\qquad\\text\{\\emph\{\(ii\) attainability:\}\}\\quad\\delta^\{\*\}=c\(\\kappa\)\\ \\text\{for some \}\\kappa\\in\\mathcal\{K\}\_\{0\}\.\(5\)For a givenκ\\kappa,αL→1\\alpha\_\{L\}\\to 1iff \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(i\) holds withδ∗=c⁡\(κ\)\\delta^\{\*\}=c\(\\kappa\)\. Both hypotheses are needed\. Attainability must be inside𝒦0\\mathcal\{K\}\_\{0\}rather than𝒦\\mathcal\{K\}: a kernel peaked at the new rows’ offset but failing atL0L\_\{0\}hasαL→1\\alpha\_\{L\}\\to 1without solving the task at either length, so the existential quantifier alone would not deliver length generalization\. And the proof compares\|ℋ⁡\(κ\)\|\|\\mathcal\{H\}\(\\kappa\)\|against\|𝒮L\|\|\\mathcal\{S\}\_\{L\}\|, which is informative only when the scored set grows\. The exact conclusionαL=1\\alpha\_\{L\}=1requires more than \([5](https://arxiv.org/html/2610.41218#S5.E5)\): the realized offset must hit the target on every scored row, including the finitely many rows inℋ⁡\(κ\)\\mathcal\{H\}\(\\kappa\), where the causal window clips the argmax belowc⁡\(κ\)c\(\\kappa\)\. Consequently fixed\-relative\-offset tasks \(prev, skip\-kk\) are realizable by any family containing a kernel whose global maximizer is atkk, while growing\-offset \(copy\) and within\-length\-varying \(reverse\) tasks are not:αL→0\\alpha\_\{L\}\\to 0for everyκ∈𝒦\\kappa\\in\\mathcal\{K\}with a strict unique global maximizer\.

###### Proof sketch \(full proof, both tie rules, App\.[A](https://arxiv.org/html/2610.41218#A1)\)\.

*Deep\-row constancy of the realized offset:*for a deep rowi≥c⁡\(κ\)i\\geq c\(\\kappa\)the causal window\[0,i\]\[0,i\]containsc⁡\(κ\)c\(\\kappa\), andc⁡\(κ\)c\(\\kappa\)maximizesκ\\kappaover*all*offsetsd≥0d\\geq 0, hence over the window; so the masked argmax offset equals the constantc⁡\(κ\)c\(\\kappa\), independent ofnn\. \(This is why the maximizer must be global: a merely local peak could be beaten by a larger offset that the window admits at long lengths\.\) Aκ\\kappatherefore hits a deep row iffc⁡\(κ\)c\(\\kappa\)equals that row’s required offset, so

αL=\#\{i∈𝒮L,i≥c\(κ\):δL\(i\)=c\(κ\)\}\+r\(κ\)\#​𝒮L,0≤r\(κ\)≤\|ℋ\(κ\)\|,\\alpha\_\{L\}=\\frac\{\\\#\\\{i\\in\\mathcal\{S\}\_\{L\},\\ i\\geq c\(\\kappa\):\\ \\delta\_\{L\}\(i\)=c\(\\kappa\)\\\}\+r\(\\kappa\)\}\{\\\#\\mathcal\{S\}\_\{L\}\},\\qquad 0\\leq r\(\\kappa\)\\leq\|\\mathcal\{H\}\(\\kappa\)\|,\(6\)wherer⁡\(κ\)r\(\\kappa\)counts the finitely many clipped rows that happen to be correct\. \(⇐\\Leftarrow\) Given \(ii\), takeκ\\kappawithc⁡\(κ\)=δ∗c\(\\kappa\)=\\delta^\{\*\}\. By \([6](https://arxiv.org/html/2610.41218#S5.E6)\) the deep rows it misses are exactly those withδL​\(i\)≠δ∗\\delta\_\{L\}\(i\)\\neq\\delta^\{\*\}, a vanishing fraction by \(i\), and\|ℋ⁡\(κ\)\|/\#​𝒮L→0\|\\mathcal\{H\}\(\\kappa\)\|/\\\#\\mathcal\{S\}\_\{L\}\\to 0sinceℋ⁡\(κ\)\\mathcal\{H\}\(\\kappa\)is fixed while\#​𝒮L→∞\\\#\\mathcal\{S\}\_\{L\}\\to\\infty\. HenceαL→1\\alpha\_\{L\}\\to 1\. \(⇒\\Rightarrow\) SupposeαL→1\\alpha\_\{L\}\\to 1for such aκ\\kappa\. Putδ∗:=c⁡\(κ\)\\delta^\{\*\}:=c\(\\kappa\), attainable by construction, giving \(ii\); by \([6](https://arxiv.org/html/2610.41218#S5.E6)\) the fraction of scored rows withδL​\(i\)≠δ∗\\delta\_\{L\}\(i\)\\neq\\delta^\{\*\}is1−αL\+o⁡\(1\)→01\-\\alpha\_\{L\}\+o\(1\)\\to 0, giving \(i\)\. For the corollaries, noδ∗\\delta^\{\*\}satisfies \(i\) when the offsetgrows\(copy,δL=⌈L/2⌉\\delta\_\{L\}=\\lceil L/2\\rceil: forL\>2​c​\(κ\)L\>2c\(\\kappa\)every copy row is deep with realized offsetc⁡\(κ\)c\(\\kappa\), so the missed fraction tends to11\) orvaries within a lengthwith positive density \(reverse: a nested\-window monotonicity argument forces failure already atL0L\_\{0\}\)\. ∎

The hypotheses are each necessary, and the scope of the characterization is discussed with counterexamples in App\.[A\.6](https://arxiv.org/html/2610.41218#A1.SS6)\.

###### Proposition 4\(Absolute/relative duality\)\.

Assume \(S1\)–\(S3\)\.*\(a\) Relative side\.*A translation\-invariant family containing a kernel whose strict unique global maximizer is atkklength\-generalizes the fixed relative offsetkk\(prev/skip\-kk:αL=1\\alpha\_\{L\}\{=\}1at all lengths\)\. On a fixed absolute position the demanded offseti−ti\-tgrows with the row, so by Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)no kernel*with a strict unique global maximizer*attainsαL→1\\alpha\_\{L\}\\to 1; and empirically the kernel theL0L\_\{0\}\-fit selects decays,αL=0\.80,0\.40,0\.20,0\.12,0\.055,0\.039\\alpha\_\{L\}=0\.80,0\.40,0\.20,0\.12,0\.055,0\.039atL=30,40,60,128,256,512L=30,40,60,128,256,512for RoPE on attend\-first\.*\(b\) Absolute side, for the clamped learned\-absolute encoding used in our experiments\.*Letφ⁡\(i\)=vmin⁡\(i,L0−1\)\\varphi\(i\)=v\_\{\\min\(i,L\_\{0\}\-1\)\}, so all positionsi≥L0i\\geq L\_\{0\}share one feature\. Such a head length\-generalizes a fixed*absolute*position in the trained range \(αL=1\\alpha\_\{L\}\{=\}1at all lengths, with anLL\-independent margin\), and fails on a fixed relative offset: every new key carries the same feature, so the logits overj≥L0j\\geq L\_\{0\}tie exactly and the target cannot be singled out \(αL=0\\alpha\_\{L\}\{=\}0with margin00\)\. For the i\.i\.d\.\-extended variant the corresponding statement is probabilistic and is Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\.*\(c\)*Neither side generalizes a growing offset \(copy/reverse\), again for kernels with a strict unique global maximizer\. Empirically \(App\.[A](https://arxiv.org/html/2610.41218#A1)\): attend\-first clamped\-absolute1\.001\.00/relative0\.000\.00; prev clamped\-absolute0\.000\.00\(an exact tie\)/relative1\.001\.00\. The i\.i\.d\.\-extended variant instead scores at the chance level of Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\(≈0\.05\\approx 0\.05in our Monte\-Carlo\), as it should: the two objects fail for different reasons\.

*Which “absolute” object\.*Stating part \(b\) for “an absolute kernel” without qualification would conflate three distinct objects\.*\(1\) An unrestricted absolute kernel*K⁡\[i,j\]K\[i,j\], with a free entry for every pair, is not limited to absolute targets at all: settingK⁡\[i,i−1\]K\[i,i\{\-\}1\]large and the rest small encodes the sub\-diagonal directly, so it realizes prev\-token at every length\.*\(2\) A learned\-absolute table extended by i\.i\.d\. unseen rows*is at chance on new rows \(Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\)\.*\(3\) The clamped learned\-absolute implementation*used in our experiments collapses all positionsi≥L0i\\geq L\_\{0\}onto one feature, so distinct new rows become indistinguishable and a moving target cannot be resolved\. The duality claim is about \(2\) and \(3\), the objects a trained model actually has, and part \(b\) above now says so\. The same care applies on the relative side: sinusoidal*absolute*features can represent translation\-invariant fixed\-offset kernels, as Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)itself shows, so “absolute features” and “absolute\-only behaviour” are not the same thing\. What separates the regimes is the realizable kernel family a given encoding induces, which is exactly the quantity condition \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(ii\) makes explicit\.

*Existence versus selection on the relative side\.*It is tempting to say that a relative kernel simply*cannot*reach a fixed absolute position because that would need a peak at the growing offsetii\. That reasoning is wrong, because the causal window truncates atd=id=i: a monotone increasingκ\\kappa, e\.g\.κ⁡\(d\)=d\\kappa\(d\)=d, puts the masked argmax atj=0j=0on every row at every length, so it realizes attend\-first exactly\. Such a kernel has no strict unique global maximizer, which is why Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)does not cover it, and its realizable analogue is present in genuine PE families: within the RoPE span,sin⁡\(d​θmin\)\\sin\(d\\theta\_\{\\min\}\)is increasing over the whole range of lengths we study\. What rules this out in practice is not translation invariance but two other things: the canonical ALiBi slope is sign\-constrained to be non\-positive \(dropping that constraint does let an ALiBi\-form kernel solve attend\-first, while leaving prev\-token unrealizable as Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)requires\), and theL0L\_\{0\}\-fit does not select the monotone solution, so the transferred accuracy decays withLLas reported in part \(a\)\. This is the same existence\-versus\-selection distinction we draw for sinusoidal encodings in Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3), and it applies here too\.

Fixed relative offsetδ=k\\delta=k\(prev, skip\-kk\)Fixed absolute positiont=0t=0\(attend\-first\)Growing offsetδn=⌈n/2⌉\\delta\_\{n\}=\\lceil n/2\\rceil\(copy\)Relativeκ⁡\(i−j\)\\kappa\(i\\\!\-\\\!j\)Clampedlearned\-abs\.✓peak==target, allLL\(αL→1\\alpha\_\{L\}\\\!\\to\\\!1\)✗needs offsetii\(grows\): no fit✗can’t track⌈n/2⌉\\lceil n/2\\rceil\(Thm[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\)✗moving target: new keys share one feature✓column\-00kernel \(length\-indep\.\)✗target drifts withnn: no fit

Figure 2:The offset taxonomy, derived \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2), Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\)\.A relative kernel length\-generalizes a*fixed relative offset*; a clamped learned\-absolute encoding does the reverse; and*neither*handles a*growing*offset \(the shared blind spot, right column\)\. Throughout, the relative statements are for kernels with a strict unique global maximizer, the hypothesis of Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2); the accompanying text explains why it is needed and what theL0L\_\{0\}\-fit contributes\. This2×32\\times 3verdict is exactly the empirical PE taxonomy, here a consequence of what a single positional head can realize\.###### Conjecture 1\(General length\-generalization impossibility\)\.

For a deterministic PE defined at all lengths, ifRLnew∉Π𝒦LR\_\{L\}^\{\\mathrm\{new\}\}\\notin\\Pi\_\{\\mathcal\{K\}\_\{L\}\}and a quantitative separation holds – for everyWWrealizingRL0R\_\{L\_\{0\}\}with old\-row margin≥γ\\geq\\gamma, the residual\(I−Π𝒦L\)​RLnew\(I\-\\Pi\_\{\\mathcal\{K\}\_\{L\}\}\)R\_\{L\}^\{\\mathrm\{new\}\}forcesmaxj≤i,j≠t⁡φ​\(i\)⊤​W​φ​\(j\)≥φ​\(i\)⊤​W​φ​\(tL​\(i\)\)\\max\_\{j\\leq i,j\\neq t\}\\varphi\(i\)^\{\\top\}W\\varphi\(j\)\\geq\\varphi\(i\)^\{\\top\}W\\varphi\(t\_\{L\}\(i\)\)on some new rowii– then no single head solves bothL0L\_\{0\}andLL\.

*Remark \(why the colspan argument is insufficient\)\.*One might conclude impossibility from “φ⁡\(i\)∉colspan⁡\(ΦL0\)\\varphi\(i\)\\notin\\mathrm\{colspan\}\(\\Phi\_\{L\_\{0\}\}\)”\. That does not suffice: the components ofWWacting on the orthogonal complement are*unconstrained*by theL0L\_\{0\}fit and permit a rescuingWWas easily as a failing one\. The argmax flip is the substance of the statement and requires proof; we leave it open\.

##### Partial progress \(App\.[A](https://arxiv.org/html/2610.41218#A1); the conjecture stays open\)\.

We can say more\.*\(i\) A saturation lemma\.*For a finite\-dimensional deterministic PE withΦL0\\Phi\_\{L\_\{0\}\}of full column rankmm, the mapW↦ΦL0​W​ΦL0⊤W\\mapsto\\Phi\_\{L\_\{0\}\}W\\Phi\_\{L\_\{0\}\}^\{\\top\}is injective \(σmin=σmin​\(ΦL0\)2\\sigma\_\{\\min\}=\\sigma\_\{\\min\}\(\\Phi\_\{L\_\{0\}\}\)^\{2\}\), so a given kernel block determinesWWand hencePLP\_\{L\}\.*\(ii\) But this does not decide LG*, because “solving atL0L\_\{0\}” \(S2\) is an argmax\+margin*cone*ofWW, not a single block: distinct blocks with identicalL0L\_\{0\}argmax transfer to oppositeLL\-verdicts\. LG is decidable only*relative to a declared selection rule*\(e\.g\. min\-norm\), not from “theL0L\_\{0\}fit” alone – the existence\-vs\-selection gap again\.*\(iii\) The obstruction is not cone\-emptiness*: an LP witness shows a singleWWsolving bothL0L\_\{0\}andLLfor prev\-token sinusoidal in the under\-determined regime \(rank=12≪m=48\\mathrm\{rank\}=12\\ll m=48\), soCL0∩CL≠∅C\_\{L\_\{0\}\}\\cap C\_\{L\}\\neq\\emptyset; the failure mechanism is selection \(Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\) rather than an empty cone\.*\(iv\) Sinusoidal escapes the saturation route entirely*: at realistic frequency countsΦL0\\Phi\_\{L\_\{0\}\}is rank\-deficient or wildly ill\-conditioned \(cond∼104\\mathrm\{cond\}\\sim 10^\{4\}–101110^\{11\}\), soWWis never robustly pinned\. The sharpest remaining open sub\-case: a deterministic PE with a non\-empty joint cone on a non\-realizable*growing*\-offset target and selection failure\.

## 6Empirical confirmation of the theory

We test whether trained transformers obey the realizability structure the theory predicts\. These experiments are meant to confirm the theorems rather than to benchmark a predictor:αL\\alpha\_\{L\}is the theory’s computable witness, and the relative/absolute taxonomy it should track is precisely the empirical pattern the theory derives \(Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\)\.

What is certified vs\. what is measured\.The theory certifies a*fitted positional kernel*:αL\\alpha\_\{L\}is a property of\(Φ,tn\)\(\\Phi,t\_\{n\}\), and Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)guarantees that a kernel that realizes the task atL0L\_\{0\}with positive margin and reachesαL=1\\alpha\_\{L\}\{=\}1with positive margin on the new rows still solves the task at lengthLL; both premises are required\. The*trained Transformer*is a*separate*object, a non\-convex optimizer over both positional and content channels\. These experiments test the empirical bridge: whether certified positional realizability \(αL→1\\alpha\_\{L\}\\\!\\to\\\!1\)*co\-occurs*with the trained model’s length generalization\. A gap between them is therefore expected and informative\. It is exactly the selection/content/depth boundary of §[7](https://arxiv.org/html/2610.41218#S7), and does not contradict the theory\.

##### Protocol\.

Tiny decoder\-only transformers \(vocab1212,d=64d=64\) trained at variable length up toL0=20L\_\{0\}=20and evaluated on extrapolation rows\[L0,L\)\[L\_\{0\},L\)\. Three depth tiers: the*1\-layer 1\-head*setting the theory describes \(1\-layer, 1\-head\) and two*scale/escape*tiers \(2\-layer 4\-head, 4\-layer 4\-head\)\. Tasks span bounded\-offset \(prev, skip\-kk\), growing\-offset \(copy, reverse\), and absolute \(first\) targets, across five PEs \(learned\-abs, NoPE, ALiBi, sinusoidal, RoPE\);26252625measurements \(3×5×7×5×53\\times 5\\times 7\\times 5\\times 5\) over55seeds and55test lengths\. Where we report correlations on a learnable subset \(train​acc\>0\.8\\mathrm\{train\\ acc\}\>0\.8\), it is because LG of a task a model never learned atL0L\_\{0\}is undefined; we always give the unconditioned numbers too\.The tasks are position\-dependent \(small vocabulary, repeated tokens\) so that content cannot localize the target position\.This statistically discourages content shortcuts but does not make the trained model’s content logit literally argmax\-neutral as \(S3\) assumes \(the value/content path remains\)\. So these experiments confirm the positional theory*within its assumed scope*; they do not confirm it independently of \(S3\)\. Where content genuinely carries the target the impossibility results are vacuous, which is exactly the boundary we map in §[7](https://arxiv.org/html/2610.41218#S7)and App\.[H](https://arxiv.org/html/2610.41218#A8)\.

##### The evaluation grid\.

The taxonomy these experiments test is*derived*\(Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\), so the axes follow the theory’s own categories\. Table[2](https://arxiv.org/html/2610.41218#S6.T2)lists every condition we evaluate together with the reason for each level\. The task set covers all three offset regimes \(fixed\-relative, fixed\-absolute, growing\), and the PE set is the five canonical classes rather than a subset\.

Table 2:The evaluation grid\. Each axis follows the theory’s categories or standard practice; the reason given for each level is the reason it was included\.
##### Primary confirmation:αL\\alpha\_\{L\}separates generalizing from non\-generalizing cells\.

Overall3535\(PE, task\) cells\(no learnability filter\), cell\-levelαL\\alpha\_\{L\}separates the66that length\-generalize \(LG≥0\.5\\geq 0\.5\) from the2929that do not withAUC=1\.00=1\.00\(base rate0\.170\.17; permutationp=5×10−5p\{=\}5\\times 10^\{\-5\},2020k shuffles\)\. This is not a fragile1\.01\.0: the margin is wide \(minαL\\alpha\_\{L\}among positives=1\.00=1\.00vs\. max among negatives=0\.36=0\.36\), and restricting to the1111non\-trivial cells \(dropping negatives withαL≈0\\alpha\_\{L\}\{\\approx\}0\) keeps AUC=1\.0=1\.0\. The screen is thus validated*on the hard cells too*\(all2020NoPE/ALiBi/copy/reverse cells sit atαL≈0\\alpha\_\{L\}\{\\approx\}0, std0\.020\.02, and none generalize in this single\-head regime\)\. As a*graded*\(rank\) statistic over the same3535cells the unconditioned Spearman is modest \(ρ=0\.47\\rho\{=\}0\.47\), because the signal is dominantly the clean binary separation; we report it for transparency and turn to the learnable subset for the graded view\.

##### αL\\alpha\_\{L\}acts primarily as a binary realizability indicator\.

On the learnable subset \(train​acc\>0\.8\\mathrm\{train\\ acc\}\>0\.8, which retains1515of3535cells;all NoPE, ALiBi, and copy/reverse are excluded, so this subset can only probe the bounded/absolute regime\), a naive cell\-levelρ=0\.90\\rho=0\.90\(bootstrap95%95\\%CI\[0\.73,0\.95\]\[0\.73,0\.95\],1515cells,55seeds\)\. But this is almost entirely the binary00\-vs\-11separation re\-expressed \(all numbers cell\-level\): binarizingαL\\alpha\_\{L\}at0\.50\.5already givesρ=0\.85\\rho=0\.85, and restricting to non\-trivial cells \(αL\>0\.05\\alpha\_\{L\}\>0\.05\) givesρ=0\.77\\rho=0\.77\. On the only family with genuinely interiorαL\\alpha\_\{L\}values \(sinusoidal bounded\-offset\) there isno positive graded signal:ρ=−0\.20\\rho=\-0\.20at the cell level \(n=4n\{=\}4cells\) and≈0\\approx 0at finer aggregations \(0\.040\.04row\-level,0\.090\.09at\(PE,task,L\)\(\\text\{PE,task\},L\)\)\. We therefore do not claimαL\\alpha\_\{L\}is a graded predictor; it is a*binary realizability indicator*, which is what the theory predicts\. \(The unconditioned cell\-level Spearman over all3535cells is0\.470\.47\.\)

##### Sufficiency is approximate, with characterized violations\.

The “≥92%\\geq 92\\%\-precision sufficient screen” claim must be qualified\. The precision ofαL≥τ⇒\\alpha\_\{L\}\\\!\\geq\\\!\\tau\\RightarrowLG≥0\.7\\geq 0\.7rises withτ\\tau:0\.9250\.925atτ=0\.5\\tau\{=\}0\.5,0\.950\.95atτ=0\.7\\tau\{=\}0\.7,0\.9870\.987atτ=0\.9\\tau\{=\}0\.9\. Atτ=0\.5\\tau\{=\}0\.5there are1212violating rows out of160160, and they come from three families:rope\-firstatL=30L\{=\}30\(αL=0\.80\\alpha\_\{L\}\{=\}0\.80, LG=0\.13=0\.13, all55seeds\),sinusoidal\-prevatL=40L\{=\}40\(αL=0\.50\\alpha\_\{L\}\{=\}0\.50, LG=0\.14=0\.14, all55seeds\), andsinusoidal\-firstatL=60,80L\{=\}60,80\(αL=1\.0\\alpha\_\{L\}\{=\}1\.0, one seed each, the worst being LG=0\.49=0\.49against a55\-seed cell mean of0\.770\.77\)\. These are the*existence\-vs\-selection*gap of Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)surfacing in the*validated*predictor: the CE fit over\-sharpensαL\\alpha\_\{L\}at smallLLwhere the true selected kernel decays\. We therefore do not claim a clean row\-level sufficient indicator; we claim a strong*binary screen*\(AUC1\.01\.0\) with a positive cell\-level association on the learnable subset, with violations confined to the relative/sinusoidal selection gap\.

##### Relation to the received taxonomy\.

A PE\-*blind*rule \(“is the offset bounded?”\) attains*negative*correlation \(ρ=−0\.24\\rho=\-0\.24\), so PE\-awareness is essential\. The hand\-built taxonomy baseline \(relative\-PE×\\timesbounded∨\\lorabsolute\-PE×\\timesabsolute\-task\) reachesρ=0\.85\\rho=0\.85, within our bootstrap CI:αL\\alpha\_\{L\}*matches but does not beat*it\.In the theory framing this is the expected outcome rather than a shortfall:that taxonomy is exactly the coarse rule Proposition[4](https://arxiv.org/html/2610.41218#Thmproposition4)*derives*, soαL\\alpha\_\{L\}tracking it confirms the theory\. The role ofαL\\alpha\_\{L\}is to be the directly computed witness of why the taxonomy holds and*where*it breaks \(the boundary where token content identifies the target below\); it is not offered as a way to beat a baseline\. We attempted to find a PE the taxonomy cannot classify butαL\\alpha\_\{L\}can \(a low\-frequency RoPE, still “relative”\): the attempt instead exposed anαL\\alpha\_\{L\}*false negative*: on the content\-neutral skip\-kktask the trained model generalizes a target the linear\-kernel fit deems unrealizable\. Since the task is content\-neutral this is not a content rescue in the sense of App\.[H](https://arxiv.org/html/2610.41218#A8); we have not isolated the mechanism further and report it as an unexplained disagreement between the trained model and the fitted kernel, and as a negative result for the comparison with the taxonomy\.

##### What the theorem certifies vs\. what we measure\.

Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\(any realizingWW\) certifies the*fitted kernel*WWat which we computeαL\\alpha\_\{L\}\. It does not certify the*trained transformer*, which is a different parameter\. SoαL​\(Wfit\)=1\\alpha\_\{L\}\(W\_\{\\mathrm\{fit\}\}\)\{=\}1guarantees the fitted kernel solves lengthLL; whether the trained model does is the empirical question, and the two can diverge:rope\-first\(αL=0\.80\\alpha\_\{L\}\{=\}0\.80, LG0\.130\.13atL=30L\{=\}30\) andsinusoidal\-first\(αL=1\.0\\alpha\_\{L\}\{=\}1\.0atL=80L\{=\}80, LG0\.770\.77on average and as low as0\.490\.49on the worst seed\) are cases whereWtrained≠WfitW\_\{\\mathrm\{trained\}\}\\neq W\_\{\\mathrm\{fit\}\}\. We therefore do not claim “the certified object is the validated one\.” The right test of the theorem’s relevance is co\-occurrence ofαL​\(Wfit\)=1\\alpha\_\{L\}\(W\_\{\\mathrm\{fit\}\}\)\{=\}1with trained\-model LG, which holds for≥92%\\geq 92\\%of high\-αL\\alpha\_\{L\}rows \(rising to98%98\\%atαL≥0\.9\\alpha\_\{L\}\{\\geq\}0\.9\); this is strong but not exact\. Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)covers any realizing kernel, so the cross\-entropy fit we use is as certified as the min\-norm interpolant; the two simply differ in how well they predict the trained model, atρ=0\.91\\rho\{=\}0\.91againstρ=0\.56\\rho\{=\}0\.56\(App\.[D](https://arxiv.org/html/2610.41218#A4)\)\. We therefore frame the empirical link as a co\-occurrence rate rather than a theorem\.

Table 3:Training\-freeαL\\alpha\_\{L\}\(atL=60L\{=\}60\) vs\. empirical LG accuracy \(mean over55seeds\), 1\-layer 1\-head and scale \(2L4H\) tiers\. Top:αL\\alpha\_\{L\}tracks 1L1H LG\. Bottom: at scale, the*content/recency channel*\(outside the positional\-only abstraction, S3\) lets ALiBi/NoPE generalize whereαL=0\\alpha\_\{L\}\{=\}0, making the predictor’s one\-sidedness visible\.PEtaskαL\\alpha\_\{L\}LG \(1L1H\)LG \(2L4H\)RoPEskip51\.001\.00±0\.001\.00\{\\pm\}0\.000\.97±0\.020\.97\{\\pm\}0\.02RoPEprev1\.000\.87±0\.050\.87\{\\pm\}0\.050\.85±0\.130\.85\{\\pm\}0\.13RoPEcopy0\.000\.10±0\.010\.10\{\\pm\}0\.010\.14±0\.010\.14\{\\pm\}0\.01sinusoidalfirst1\.000\.87±0\.130\.87\{\\pm\}0\.130\.92±0\.040\.92\{\\pm\}0\.04sinusoidalskip30\.250\.22±0\.030\.22\{\\pm\}0\.030\.11±0\.010\.11\{\\pm\}0\.01learned\-absfirst1\.001\.00±0\.001\.00\{\\pm\}0\.001\.00±0\.001\.00\{\\pm\}0\.00learned\-absprev0\.000\.08±0\.000\.08\{\\pm\}0\.000\.08±0\.000\.08\{\\pm\}0\.00*Content\-channel cases \(S3 violations\):αL\\alpha\_\{L\}under\-predicts at scale*ALiBiprev0\.000\.18±0\.000\.18\{\\pm\}0\.001\.00±0\.00\\mathbf\{1\.00\{\\pm\}0\.00\}ALiBifirst0\.000\.15±0\.010\.15\{\\pm\}0\.011\.00±0\.00\\mathbf\{1\.00\{\\pm\}0\.00\}NoPEfirst0\.000\.16±0\.010\.16\{\\pm\}0\.011\.00±0\.00\\mathbf\{1\.00\{\\pm\}0\.00\}Figure 3:Training\-freeαL\\alpha\_\{L\}vs\. empirical LG across the three tiers \(55seeds\)\.*Left*\(1L1H, the setting the theory describes\): learnable cells \(blue\) give Spearmanρ=0\.90\\rho\{=\}0\.90\.*Center, right*\(scale, 2L4H, 4L4H\): the*graded*rank correlation weakens \(its bootstrap CI includes00\) as the content channel rescues low\-αL\\alpha\_\{L\}cells, yet highαL\\alpha\_\{L\}remains a*sufficient screen*at every depth \(precision≥0\.92\\geq 0\.92\)\.Figure 4:Empirical LG \(solid\) and training\-freeαL\\alpha\_\{L\}\(dashed\) vs\. test lengthLL\(trainL0=20L\_\{0\}\{=\}20\), 1L1H\.αL\\alpha\_\{L\}tracks both who generalizes and how LG decays withLL\.
##### Scale and depth \(H3\): one\-sidedness\.

At22\- and44\-layer/44\-head, the graded rank correlation on learnable cells falls from0\.900\.90toρ=0\.31\\rho=0\.31and0\.280\.28, with bootstrap CIs that*include00*\(Table[4](https://arxiv.org/html/2610.41218#A4.T4)\), so beyond depth11αL\\alpha\_\{L\}is no longer a significant graded predictor\. What*does*survive at depth is the*sufficient*\-screen property: precision rises to0\.940\.94and0\.970\.97\(CIs\[0\.89,0\.97\]\[0\.89,0\.97\],\[0\.94,0\.99\]\[0\.94,0\.99\]\), i\.e\. highαL\\alpha\_\{L\}still implies LG \(Table[4](https://arxiv.org/html/2610.41218#A4.T4), Fig\.[6](https://arxiv.org/html/2610.41218#A4.F6)\)\. The lost graded signal is entirely attributable to low\-αL\\alpha\_\{L\}cells that generalize via mechanisms outside the positional\-only abstraction: ALiBi and NoPE realize prev\-token andfirstthrough the content/recency channel \(αL=0\\alpha\_\{L\}\{=\}0, LG=1\.0=1\.0; Table[3](https://arxiv.org/html/2610.41218#S6.T3), bottom\), and the set of such cells*grows*with depth\. This is the predicted \(S3\) boundary:αL\\alpha\_\{L\}is a*sufficient*indicator rather than a necessary one, mirroring exactly that sufficiency \(Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\) and the relative growing\-offset necessity \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\) are proven while the*fully general*necessity \(Conjecture[1](https://arxiv.org/html/2610.41218#Thmconjecture1)\) is open\. The heatmaps in Fig\.[5](https://arxiv.org/html/2610.41218#S6.F5)localize where content rescues position\.

![Refer to caption](https://arxiv.org/html/2610.41218v1/heatmaps.png)Figure 5:Training\-freeα60\\alpha\_\{60\}\(left\) and empirical LG@60 \(right\) over \(PE, task\), 1L1H\. Agreement is strong in this single\-head regime; the content\-driven disagreements \(which emerge at depth, not shown here\) are characterized in §[7](https://arxiv.org/html/2610.41218#S7)\.

## 7Limitations

App\.[B](https://arxiv.org/html/2610.41218#A2)sets out, with evidence, the five waysαL\\alpha\_\{L\}departs from a trained model’s behaviour: a selection gap, content rescue, a depth escape, low\-frequency under\-prediction, and overprediction when the trained model selects a different kernel\. The first four are failures of*necessity*and leave the sufficiency direction intact, which is why we callαL\\alpha\_\{L\}a one\-sided screen; the fifth is the one case where highαL\\alpha\_\{L\}does not co\-occur with length generalization\.

The remaining scope limits: \(1\)Single head / single layer\.Multi\-head and depth provably escape the impossibility \(an anchor head plus a content\-match head solves copy at depth 1\); all impossibility claims are scoped to the positional\-only subclass\. \(2\)Content channel\.Where content carries the target,αL\\alpha\_\{L\}under\-predicts; this is the binding limitation and the source of the indicator’s one\-sidedness\. We show it is closable \(App\.[H](https://arxiv.org/html/2610.41218#A8)\): on a content\-driven induction task, positionalαL=0\\alpha\_\{L\}\{=\}0for every PE yet a22\-layer ALiBi model reaches LG0\.840\.84, and a content\-awareαL\+\\alpha\_\{L\}^\{\+\}flags the regimeαL\\alpha\_\{L\}misses \(with residual PE\-modulation left to future work\)\. \(3\)The forward question is open\.Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\(stated for any realizingWW\) certifies thatαL​\(W\)=1⇒\\alpha\_\{L\}\(W\)\{=\}1\\RightarrowLG, but it does not say which\(φ,t\)\(\\varphi,t\)admit a fittedWWreachingαL=1\\alpha\_\{L\}\{=\}1; we characterize the negative side \(Theorems[2](https://arxiv.org/html/2610.41218#Thmtheorem2), Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)\) but a positive forward criterion is open\.

A list of statements we explicitly do not claim, with the reason for each, is given in App\.[B\.1](https://arxiv.org/html/2610.41218#A2.SS1)\.

## 8Conclusion

We gave a realizability theory of length generalization for single\-head positional attention: a simulation transfer operator and its scalar witnessαL\\alpha\_\{L\}, a sufficiency theorem, and adeep\-row characterization for relative heads\(for kernels with a strict unique global maximizer,αL→1\\alpha\_\{L\}\\to 1exactly when the offset is constant on a density\-one set of deep rows and that offset is attainable within the encoding’s kernel family\) that subsumes growing\-offset impossibility and the absolute/relative duality*deriving*the empirical pattern, alongside a learned\-absolute characterization; only the general deterministic\-PE case remains conjectural\. Selected theoretical claims are accompanied by executable checks that probe their assumptions\. Experiments confirm the structure: in the controlled single\-head regimeαL\\alpha\_\{L\}*exactly separates*the taxonomy’s generalizing and non\-generalizing cells \(AUC=1\.0=1\.0\)\. They also locate the theory’s boundary: the content channel, outside the abstraction, can carry the target where positional realizability cannot \(ALiBi prev, induction\), andαL\\alpha\_\{L\}under\-predicts low\-frequency relative PEs\. Rather than replacing the received taxonomy,αL\\alpha\_\{L\}*derives*it from positional\-kernel transfer and exposes its escape routes\. Extending the theory across the content channel and to multi\-head/depth, which provably escape the single\-head impossibilities, is the natural next step\.

#### Acknowledgments

We thank the Action Editor, Zhanxing Zhu, and the three anonymous reviewers\. Their careful reading identified errors in earlier versions of Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)and Propositions[2](https://arxiv.org/html/2610.41218#Thmproposition2)and[4](https://arxiv.org/html/2610.41218#Thmproposition4), and their comments substantially improved both the results and the presentation\.

#### Use of language models

Large language models \(Claude and Codex\) were used extensively in preparing this work: to draft and edit the text, to write the experiment, analysis and verification code, and to develop and check the mathematical arguments, including searches for counterexamples to the stated results\. The authors take full responsibility for the content\. Every reported number can be reproduced from the released code and data\.

## References

- Anil et al\. \(2022\)Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur\-Ari, Ethan Dyer, and Behnam Neyshabur\.Exploring length generalization in large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Cho et al\. \(2024\)Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, and Chulhee Yun\.Position coupling: Improving length generalization of arithmetic transformers using task structure\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- Csordás et al\. \(2022\)Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber\.The neural data router: Adaptive control flow in transformers improves systematic generalization\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Elhage et al\. \(2021\)Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, et al\.A mathematical framework for transformer circuits\.*Transformer Circuits Thread*, 2021\.[https://transformer\-circuits\.pub/2021/framework/index\.html](https://transformer-circuits.pub/2021/framework/index.html)\.
- Golowich et al\. \(2025\)Noah Golowich, Samy Jelassi, David Brandfonbrener, Sham M\. Kakade, and Eran Malach\.The role of sparsity for length generalization in transformers\.*arXiv preprint arXiv:2502\.16792*, 2025\.
- Gu et al\. \(2025\)Zihan Gu, Ruoyu Chen, Han Zhang, Hua Zhang, and Yue Hu\.Deconstructing positional information: From attention logits to training biases\.*arXiv preprint arXiv:2505\.13027*, 2025\.
- Gunasekar et al\. \(2018\)Suriya Gunasekar, Jason D\. Lee, Daniel Soudry, and Nathan Srebro\.Implicit bias of gradient descent on linear convolutional networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2018\.
- Hahn \(2020\)Michael Hahn\.Theoretical limitations of self\-attention in neural sequence models\.*Transactions of the Association for Computational Linguistics \(TACL\)*, 8, 2020\.
- Huang et al\. \(2025\)Xinting Huang, Andy Yang, Satwik Bhattamishra, Yash Sarrof, Andreas Krebs, Hattie Zhou, Preetum Nakkiran, and Michael Hahn\.A formal framework for understanding length generalization in transformers\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Izzo et al\. \(2025\)Zachary Izzo, Eshaan Nichani, and Jason D\. Lee\.Quantitative bounds for length generalization in transformers\.*arXiv preprint arXiv:2510\.27015*, 2025\.
- Jelassi et al\. \(2023\)Samy Jelassi, Stéphane d’Ascoli, Carles Domingo\-Enrich, Yuhuai Wu, Yuanzhi Li, and François Charton\.Length generalization in arithmetic transformers\.*arXiv preprint arXiv:2306\.15400*, 2023\.
- Kazemnejad et al\. \(2023\)Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy\.The impact of positional encoding on length generalization in transformers\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Merrill et al\. \(2022\)William Merrill, Ashish Sabharwal, and Noah A\. Smith\.Saturated transformers are constant\-depth threshold circuits\.*Transactions of the Association for Computational Linguistics \(TACL\)*, 10, 2022\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, et al\.In\-context learning and induction heads\.*Transformer Circuits Thread*, 2022\.
- Press et al\. \(2022\)Ofir Press, Noah A\. Smith, and Mike Lewis\.Train short, test long: Attention with linear biases enables input length extrapolation\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- Ruoss et al\. \(2023\)Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau\-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness\.Randomized positional encodings boost length generalization of transformers\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2023\.
- Sanford et al\. \(2023\)Clayton Sanford, Daniel Hsu, and Matus Telgarsky\.Representational strengths and limitations of transformers\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Soudry et al\. \(2018\)Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro\.The implicit bias of gradient descent on separable data\.*Journal of Machine Learning Research*, 19, 2018\.
- Su et al\. \(2024\)Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu\.Roformer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568, 2024\.
- Tarzanagh et al\. \(2023a\)Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak\.Transformers as support vector machines\.*arXiv preprint arXiv:2308\.16898*, 2023a\.
- Tarzanagh et al\. \(2023b\)Davoud Ataee Tarzanagh, Yingcong Li, Xuechen Zhang, and Samet Oymak\.Max\-margin token selection in attention mechanism\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023b\.
- Wang et al\. \(2024\)Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang\.Length generalization of causal transformers without position encoding\.*arXiv preprint arXiv:2404\.12224*, 2024\.
- Weiss et al\. \(2021\)Gail Weiss, Yoav Goldberg, and Eran Yahav\.Thinking like transformers\.In*International Conference on Machine Learning \(ICML\)*, 2021\.
- Zhou et al\. \(2024a\)Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran\.What algorithms can transformers learn? a study in length generalization\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.
- Zhou et al\. \(2024b\)Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou\.Transformers can achieve length generalization but not robustly\.*arXiv preprint arXiv:2402\.09371*, 2024b\.

## Appendix AFull proofs

### A\.1Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)\(Simulation sufficiency, any realizingWW\)

Fix anyWWsatisfying \(i\)–\(ii\)\. Under \(S1\)–\(S3\) the row\-iilogits at any lengthnnareai​j\(n\)=φ​\(i\)⊤​W​φ​\(j\)a^\{\(n\)\}\_\{ij\}=\\varphi\(i\)^\{\\top\}W\\varphi\(j\)forj≤ij\\leq i, with the content term argmax\-neutral\.*New rows*i∈\[L0,L\)i\\in\[L\_\{0\},L\): by \(ii\) the simulatedPL\[i,⋅\]=ai⋅\(L\)P\_\{L\}\[i,\\cdot\]=a^\{\(L\)\}\_\{i\\cdot\}has its unique masked maximum attL​\(i\)t\_\{L\}\(i\)with gap≥γL\\geq\\gamma\_\{L\}\.*Old rows*i<L0i<L\_\{0\}: the causal mask restricts the key set to\{0,…,i\}⊆\{0,…,L0−1\}\\\{0,\\dots,i\\\}\\subseteq\\\{0,\\dots,L\_\{0\}\-1\\\}, and for thosejjthe featuresφ⁡\(j\)\\varphi\(j\)are exactly those present at lengthL0L\_\{0\}; henceai​j\(L\)=ai​j\(L0\)a^\{\(L\)\}\_\{ij\}=a^\{\(L\_\{0\}\)\}\_\{ij\}entrywise, so the masked argmax and gap≥γ0\\geq\\gamma\_\{0\}are inherited verbatim from \(i\)\. Every row’s unique masked maximizer istL​\(i\)t\_\{L\}\(i\)with gap≥min⁡\(γ0,γL\)\>0\\geq\\min\(\\gamma\_\{0\},\\gamma\_\{L\}\)\>0\.□\\square

*Remark \(why not the min\-norm interpolant\)\.*Stating this for the min\-normW∗W^\{\*\}would invokeΦL0​W∗​ΦL0⊤=RL0\\Phi\_\{L\_\{0\}\}W^\{\*\}\\Phi\_\{L\_\{0\}\}^\{\\top\}=R\_\{L\_\{0\}\}, which is false unlessRL0∈𝒦L0R\_\{L\_\{0\}\}\\in\\mathcal\{K\}\_\{L\_\{0\}\}\. The argument needs no such exactness and no choice ofWW: only that the chosenWWis argmax\-correct atL0L\_\{0\}and hasαL​\(W\)=1\\alpha\_\{L\}\(W\)\{=\}1\. Hence it covers the fittedWWused in the experiments, though not the trained transformer\. The theorem’s force comes entirely from the invariance of the already\-seen rows under the causal mask; the new\-row clause restates \(ii\)\.

### A\.2Proposition[1](https://arxiv.org/html/2610.41218#Thmproposition1)\(Affine features force endpoint argmax\)

Withφ⁡\(i\)=\[1,i\]⊤\\varphi\(i\)=\[1,i\]^\{\\top\}writeW=\(W00W01W10W11\)W=\\begin\{pmatrix\}W\_\{00\}&W\_\{01\}\\\\ W\_\{10\}&W\_\{11\}\\end\{pmatrix\}\. Thenai​j=φ​\(i\)⊤​W​φ​\(j\)=c0​\(i\)\+c1​\(i\)​ja\_\{ij\}=\\varphi\(i\)^\{\\top\}W\\varphi\(j\)=c\_\{0\}\(i\)\+c\_\{1\}\(i\)\\,jwithc0​\(i\)=W00\+i​W10c\_\{0\}\(i\)=W\_\{00\}\+iW\_\{10\},c1​\(i\)=W01\+i​W11c\_\{1\}\(i\)=W\_\{01\}\+iW\_\{11\}, which is affine in the key indexj∈\{0,…,i\}j\\in\\\{0,\\dots,i\\\}\. An affine function on a finite integer chain is monotone, so its maximizers are endpoints, and the maximizer is unique unlessc1​\(i\)=0c\_\{1\}\(i\)=0\(a flat row, margin00, excluded by \(S2\)\)\. For a strict interior target0<t<i0<t<i, optimality would requireai​t≥ai​0a\_\{it\}\\geq a\_\{i0\}andai​t≥ai​ia\_\{it\}\\geq a\_\{ii\}, i\.e\.t​c1​\(i\)≥0t\\,c\_\{1\}\(i\)\\geq 0and\(t−i\)​c1​\(i\)≥0\(t\-i\)c\_\{1\}\(i\)\\geq 0; witht\>0,t−i<0t\>0,\\ t\-i<0this forcesc1​\(i\)≥0c\_\{1\}\(i\)\\geq 0andc1​\(i\)≤0c\_\{1\}\(i\)\\leq 0, hencec1​\(i\)=0c\_\{1\}\(i\)=0and a zero margin\. Therefore no interior key is ever the*unique, positive\-margin*maximizer\. Since the competitor set\{0,…,i\}\\\{0,\\dots,i\\\}does not depend onnn, the conclusion holds at every length; prev\-token \(t=i−1t=i\-1, interior fori≥2i\\geq 2\) is thus unrealizable for alln≥3n\\geq 3\.□\\square

*Caveat\.*The statement is about the unique positive\-margin argmax, not the set inclusionarg⁡max⊆\{0,i\}\\arg\\max\\subseteq\\\{0,i\\\}, which fails on the measure\-zero locusc1​\(i\)=0c\_\{1\}\(i\)=0\(there the argmax set is the whole chain\)\.

### A\.3Proposition[2](https://arxiv.org/html/2610.41218#Thmproposition2)\(Learned\-absolute underdetermination\)

Letφ⁡\(i\)=vi\\varphi\(i\)=v\_\{i\}withv0,…,vL0−1v\_\{0\},\\dots,v\_\{L\_\{0\}\-1\}fixed andvL0,…,vL−1∼μv\_\{L\_\{0\}\},\\dots,v\_\{L\-1\}\\sim\\mui\.i\.d\. from an absolutely continuous isotropic law, andW=S⁡\(v0,…,vL0−1\)W=S\(v\_\{0\},\\dots,v\_\{L\_\{0\}\-1\}\)any rule of the trained rows only \(soW⟂\{vj\}j≥L0W\\perp\\\{v\_\{j\}\\\}\_\{j\\geq L\_\{0\}\}\)\. Putg⁡\(j\)=W​φ​\(j\)g\(j\)=W\\varphi\(j\)\.*Uniqueness:*for two keysj≠j′j\\neq j^\{\\prime\}withj,j′≠ij,j^\{\\prime\}\\neq ithe differenceai​j−ai​j′=vi⊤​W​\(vj−vj′\)a\_\{ij\}\-a\_\{ij^\{\\prime\}\}=v\_\{i\}^\{\\top\}W\(v\_\{j\}\-v\_\{j^\{\\prime\}\}\)is affine in the absolutely continuousviv\_\{i\}, and it is a nonzero functional precisely whenW⁡\(vj−vj′\)≠0W\(v\_\{j\}\-v\_\{j^\{\\prime\}\}\)\\neq 0; a comparison against the self\-key contributesvi⊤​W​vj−vi⊤​W​viv\_\{i\}^\{\\top\}Wv\_\{j\}\-v\_\{i\}^\{\\top\}Wv\_\{i\}, which is a polynomial of degree two inviv\_\{i\}rather than an affine functional\. For pairs of distinct non\-self keys the coefficientW⁡\(vj−vj′\)W\(v\_\{j\}\-v\_\{j^\{\\prime\}\}\)is nonzero wheneverWWdoes not collapse them, and the zero set of a nonzero affine functional is null\. Comparisons involving the self\-key are not covered by this argument, and no condition of that kind suffices: takeWWantisymmetric withW​v0=0Wv\_\{0\}=0\. Thenvi⊤​W​vi=0v\_\{i\}^\{\\top\}Wv\_\{i\}=0becauseWWis antisymmetric, andvi⊤​W​v0=0v\_\{i\}^\{\\top\}Wv\_\{0\}=0becauseW​v0=0Wv\_\{0\}=0, so the self\-key and the trained key00carry*identical*logits for everyviv\_\{i\}, whileW​vj≠W​viWv\_\{j\}\\neq Wv\_\{i\}still holds generically\. With positive probability the remaining logits are all smaller and this identical pair is the maximum\. We therefore take almost\-sure uniqueness ofMiM\_\{i\}as an explicit hypothesis of the proposition rather than deriving it; it holds, for instance, whenever the realized logit vector has distinct entries almost surely, which we verify numerically for the fitted kernels we use\.*Target\-independence:*the mapvi↦arg⁡maxj​vi⊤​g​\(j\)v\_\{i\}\\mapsto\\arg\\max\_\{j\}v\_\{i\}^\{\\top\}g\(j\)is a fixed measurable function of\(vi,\{g⁡\(j\)\}\)\(v\_\{i\},\\\{g\(j\)\\\}\)in which the labeltL​\(i\)t\_\{L\}\(i\)never appears, andSScannot encode it into the new\-key geometry; thusLaw⁡\(Mi\)\\mathrm\{Law\}\(M\_\{i\}\)does not referencetL​\(i\)t\_\{L\}\(i\)\.*Chance rate:*conditioned onWW, the off\-diagonal new keys\{g⁡\(j\):L0≤j<i\}=\{W​vj\}\\\{g\(j\):L\_\{0\}\\leq j<i\\\}=\\\{Wv\_\{j\}\\\}are i\.i\.d\. and independent ofviv\_\{i\}, so by exchangeability their indices are equally likely to carry the maximum given that the maximum is attained inNiN\_\{i\}\. WithtL​\(i\)∈Nit\_\{L\}\(i\)\\in N\_\{i\}andk=\|\{L0≤j≤i\}\|k=\|\\\{L\_\{0\}\\leq j\\leq i\\\}\|this givesPr⁡\[Mi=tL​\(i\)∣Mi∈Ni\]=1/\(k−1\)\\Pr\[M\_\{i\}=t\_\{L\}\(i\)\\mid M\_\{i\}\\in N\_\{i\}\]=1/\(k\-1\), and since\{Mi=tL\(i\)\}⊆\{Mi∈Ni\}\\\{M\_\{i\}=t\_\{L\}\(i\)\\\}\\subseteq\\\{M\_\{i\}\\in N\_\{i\}\\\}, the chain rule givesPr\[Mi=tL\(i\)\]=Pr\[Mi∈Ni\]/\(k−1\)≤1/\(k−1\)\\Pr\[M\_\{i\}=t\_\{L\}\(i\)\]=\\Pr\[M\_\{i\}\\in N\_\{i\}\]/\(k\-1\)\\leq 1/\(k\-1\)\. The old keys and the self\-key are not exchangeable withNiN\_\{i\}, so they may hold the maximum; that is why the unconditional value is the smaller one\.□\\square

*Underdetermination, not cone\-emptiness\.*A rescuingWWrealizing the target on all new rows exists with positive probability \(Monte Carlo:≈20\\approx 20–35%35\\%of draws;experiments/theorem2b\_learned\_abs\_montecarlo\.py\); the selection rule is simply blind to the untrained coordinates\. The self\-keyj=ij=iis excluded becauseai​i=vi⊤​W​via\_\{ii\}=v\_\{i\}^\{\\top\}Wv\_\{i\}is a*quadratic*\(non\-exchangeable\) form with meantr​W\\mathrm\{tr\}\\,W, which is why the conditional rate is1/\(k−1\)1/\(k\-1\)rather than1/k1/k\.

### A\.4Proposition[3](https://arxiv.org/html/2610.41218#Thmproposition3)\(Sinusoidal: existence vs\. selection\)

*\(A\) Existence\.*With incommensurate geometric frequencies\{θk\}\\\{\\theta\_\{k\}\\\}, the translation\-invariant kernelg⁡\(d\)=∑kcos⁡\(θk​\(d−δ\)\)≤\#⁡\{θk\}g\(d\)=\\sum\_\{k\}\\cos\(\\theta\_\{k\}\(d\-\\delta\)\)\\leq\\\#\\\{\\theta\_\{k\}\\\}with equality iffθk​\(d−δ\)∈2​π​ℤ\\theta\_\{k\}\(d\-\\delta\)\\in 2\\pi\\mathbb\{Z\}for allkk; incommensurability forcesd=δd=\\delta, sogghas a unique global maximum at the target offset for any fixedδ\\delta\(verified numerically toδ≈5×104\\delta\\approx 5\\times 10^\{4\}, far beyond the lowest\-frequency period, refuting the intuition that a sinusoidal encoding must fail once the offset exceeds that period\)\. Hence aWWrealizing a fixed offset exists at all lengths\.*\(B\) Growing offsets\.*A translation\-invariant kernel*with a strict unique global maximizer*c⁡\(κ\)c\(\\kappa\)realizes the constant offsetc⁡\(κ\)c\(\\kappa\)on every deep row \(Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\), which can equalδL=⌈L/2⌉\\delta\_\{L\}=\\lceil L/2\\rceilonly atL∈\{2​c​\(κ\)−1,2​c​\(κ\)\}L\\in\\\{2c\(\\kappa\)\-1,2c\(\\kappa\)\\\}, and at someL≤c⁡\(κ\)L\\leq c\(\\kappa\)where the window clips onto a lower local peak; for everyL\>2​c​\(κ\)L\>2c\(\\kappa\)each scored row is missed, soαL→0\\alpha\_\{L\}\\to 0on copy/reverse\. \(The earlier “cannot hold at two lengths” phrasing was imprecise: shallow clipping does let one kernel solve copy at two*particular*lengths, e\.g\.κ⁡\[10\]=5,κ⁡\[30\]=8\\kappa\[10\]\{=\}5,\\kappa\[30\]\{=\}8atL0=20L\_\{0\}\{=\}20andL=60L\{=\}60; the correct statement is asymptotic\. Without a global maximizer the conclusion fails outright, as the monotone example after the proposition shows\.\)*Existence≠\\neqselection:*theL0L\_\{0\}\-fitW∗W^\{\*\}does not recover the fixed\-offset rescue – under the transfer operator sinusoidal prev decays \(αL=0\.45\\alpha\_\{L\}=0\.45atL=60L\{=\}60,0\.340\.34atL=128L\{=\}128\), and the sub\-unit value*is*αL\\alpha\_\{L\}quantifying the selection gap\.□\\square

### A\.5Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\(Relative\-head deep\-row characterization\)

Letai​j=κ⁡\(i−j\)a\_\{ij\}=\\kappa\(i\-j\)on causal keysj≤ij\\leq i, with a strict unique*global*maximizerc⁡\(κ\)=arg⁡maxd≥0⁡κ⁡\(d\)c\(\\kappa\)=\\arg\\max\_\{d\\geq 0\}\\kappa\(d\), under a fixed tie rule\. Write𝒮L\\mathcal\{S\}\_\{L\}for the scored rows \(the task’s query positions in\[L0,L\)\[L\_\{0\},L\)\), which is the setαL\\alpha\_\{L\}averages over\.*Deep\-row constancy \(Lemma\)\.*For any rowi≥c⁡\(κ\)i\\geq c\(\\kappa\)the window\[0,i\]\[0,i\]containsc⁡\(κ\)c\(\\kappa\), and no admitted offset beats it becausec⁡\(κ\)c\(\\kappa\)is the maximum over alld≥0d\\geq 0; so the masked argmax offset is the constantc⁡\(κ\)c\(\\kappa\)for all suchiiand allnn\(verified over40004000random kernels, both tie rules; globality is what makes this work for the unbounded\-support RoPE and ALiBi families\)\.

By the Lemma,κ\\kappahits a deep scored rowiiiffδL​\(i\)=c​\(κ\)\\delta\_\{L\}\(i\)=c\(\\kappa\), so withℋ⁡\(κ\)=𝒮L∩\[L0,c⁡\(κ\)\)\\mathcal\{H\}\(\\kappa\)=\\mathcal\{S\}\_\{L\}\\cap\[L\_\{0\},c\(\\kappa\)\), whose size is at mostc⁡\(κ\)−L0c\(\\kappa\)\-L\_\{0\}uniformly inLL\(it is empty once the query region starts abovec⁡\(κ\)c\(\\kappa\), as happens for copy and reverse\),

αL=\#\{i∈𝒮L,i≥c\(κ\):δL\(i\)=c\(κ\)\}\+r\(κ\)\#​𝒮L,0≤r\(κ\)≤\|ℋ\(κ\)\|,\\alpha\_\{L\}\\;=\\;\\frac\{\\\#\\\{i\\in\\mathcal\{S\}\_\{L\},\\ i\\geq c\(\\kappa\):\\ \\delta\_\{L\}\(i\)=c\(\\kappa\)\\\}\+r\(\\kappa\)\}\{\\\#\\mathcal\{S\}\_\{L\}\},\\qquad 0\\leq r\(\\kappa\)\\leq\|\\mathcal\{H\}\(\\kappa\)\|,\(7\)so the clipped rows contributeO⁡\(1/\#​𝒮L\)→0O\(1/\\\#\\mathcal\{S\}\_\{L\}\)\\to 0\. Note that both the theorem and \([7](https://arxiv.org/html/2610.41218#A1.E7)\) normalize by\#​𝒮L\\\#\\mathcal\{S\}\_\{L\}; this is the same convention as the implementation, and it matters, since for tasks whose query region is a strict subset of\[L0,L\)\[L\_\{0\},L\)\(copy, reverse\) the two would differ by a non\-vanishing factor\.

*\(⇐\\Leftarrow\)*Assume \([5](https://arxiv.org/html/2610.41218#S5.E5)\): the required offset equalsδ∗\\delta^\{\*\}off a density\-zero subset of𝒮L\\mathcal\{S\}\_\{L\}, and someκ∈𝒦0\\kappa\\in\\mathcal\{K\}\_\{0\}hasc⁡\(κ\)=δ∗c\(\\kappa\)=\\delta^\{\*\}\. Take thatκ\\kappa\. Old rows are correct becauseκ\\kapparealizes the task atL0L\_\{0\}, and the causal\-mask identity of Thm[1](https://arxiv.org/html/2610.41218#Thmtheorem1)preserves that correctness at lengthLL; by \([7](https://arxiv.org/html/2610.41218#A1.E7)\) the missed deep rows form a vanishing fraction and\|ℋ⁡\(κ\)\|\|\\mathcal\{H\}\(\\kappa\)\|is fixed, soαL→1\\alpha\_\{L\}\\to 1\. \(Verified forδ∗∈\{1,2,3,5,8\}\\delta^\{\*\}\\in\\\{1,2,3,5,8\\\},LLup to50005000\.\) Note both hypotheses are needed: for the canonical ALiBi familyκ⁡\(d\)=c0\+s​d\\kappa\(d\)=c\_\{0\}\+sd,s≤0s\\leq 0, the maximizer isc⁡\(κ\)=0c\(\\kappa\)=0for every admissible slope, so no ALiBi kernel realizesδ∗≥1\\delta^\{\*\}\\geq 1however constant the demanded offset is\.

*The exact conclusionαL=1\\alpha\_\{L\}=1needs every scored row, not just the deep ones\.*The clipped setℋ⁡\(κ\)\\mathcal\{H\}\(\\kappa\)never leaves𝒮L\\mathcal\{S\}\_\{L\}, so if a row in it is missed,αL<1\\alpha\_\{L\}<1at everyLL\. Example:κ⁡\(d\)=−\|d−30\|\\kappa\(d\)=\-\|d\-30\|withL0=20L\_\{0\}=20leaves the ten rows\[20,30\)\[20,30\)clipped; a task demanding offset11there and3030on deep rows satisfies exact constancy on all deep rows, yetαL=1−10/\(L−20\)\\alpha\_\{L\}=1\-10/\(L\-20\), i\.e\.0\.500,0\.875,0\.990,0\.999950\.500,0\.875,0\.990,0\.99995atL=40,100,103,2⋅105L=40,100,10^\{3\},2\\cdot 10^\{5\}: it tends to11but never equals it\. HenceαL=1\\alpha\_\{L\}=1is equivalent to the realized offset hitting the target on every scored row, deep and clipped alike\.

*\(⇒\\Rightarrow\)*SupposeαL→1\\alpha\_\{L\}\\to 1for someκ∈𝒦0\\kappa\\in\\mathcal\{K\}\_\{0\}\. Putδ∗=c⁡\(κ\)\\delta^\{\*\}=c\(\\kappa\), which is attainable in𝒦0\\mathcal\{K\}\_\{0\}by construction, giving \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(ii\)\. By \([7](https://arxiv.org/html/2610.41218#A1.E7)\) the fraction of scored rows withδL​\(i\)≠δ∗\\delta\_\{L\}\(i\)\\neq\\delta^\{\*\}equals1−αL\+o⁡\(1\)→01\-\\alpha\_\{L\}\+o\(1\)\\to 0, giving \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(i\)\. For the corollaries, noδ∗\\delta^\{\*\}can satisfy \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(i\) in either of two cases\.\(a\) The offset grows withnn\(copy,δn=⌈n/2⌉\\delta\_\{n\}=\\lceil n/2\\rceil\): for allL\>2​c​\(κ\)L\>2c\(\\kappa\)every copy row is deep with realized offsetc⁡\(κ\)c\(\\kappa\), which cannot equal⌈L/2⌉→∞\\lceil L/2\\rceil\\to\\infty, so the failing fraction tends to11andαL→0\\alpha\_\{L\}\\to 0\.\(b\) It varies within a length\(reverse,δn​\(i\)=2​i−2​⌈n/2⌉\+1\\delta\_\{n\}\(i\)=2i\-2\\lceil n/2\\rceil\+1, distinct offsets\{1,3,5,…\}\\\{1,3,5,\\dots\\\}\): a nested\-window monotonicity argument gives a contradiction already atL0=20L\_\{0\}=20\. Row1818demands offset1717and its window is\{0,…,18\}\\\{0,\\dots,18\\\}, which contains offset1515, soκ⁡\[17\]\>κ⁡\[15\]\\kappa\[17\]\>\\kappa\[15\]\. Row1717demands offset1515and its window\{0,…,17\}\\\{0,\\dots,17\\\}contains offset1717, soκ⁡\[15\]\>κ⁡\[17\]\\kappa\[15\]\>\\kappa\[17\]\. The two are incompatible, independent ofc⁡\(κ\)c\(\\kappa\)\. A2020k\-kernel sweep confirms noκ\\kappasolves reverse atL0L\_\{0\}\(iff\_constant\_offset\_lg\.py\)\.□\\square

*Why density and not exact constancy\.*See App\.[A\.6](https://arxiv.org/html/2610.41218#A1.SS6)\(a\) for the counterexample that forces condition \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(i\) to be stated in density\.

*Why deep\-row/asymptotic, not a finite\-LL“complete iff”\.*One might hope to add the necessity clause “shallow rows=min⁡\(i,δ∗\)=\\min\(i,\\delta^\{\*\}\)”, but it isfalse:κ=\[1\.46,1\.96,1\.80,6\.32\]\\kappa=\[1\.46,1\.96,1\.80,6\.32\]\(offsets00–33,c⁡\(κ\)=3c\(\\kappa\)=3\) makes deep rows take offset33but shallow rowi=2i\{=\}2argmax offset1≠min⁡\(2,3\)=21\\neq\\min\(2,3\)\{=\}2\(window values\[1\.46,1\.96,1\.80\]\[1\.46,1\.96,1\.80\]\)\. Shallow rows are a finite,κ\\kappa\-dependent set, uncharacterized by a universal formula but immaterial toαL\\alpha\_\{L\}asL→∞L\\to\\infty– hence the criterion is on deep rows\. Two further cautions: the growing\-offset clash is*asymptotic*rather than two\-length \(κ⁡\[10\]=5,κ⁡\[30\]=8\\kappa\[10\]\{=\}5,\\kappa\[30\]\{=\}8solves copy at bothL0=20L\_\{0\}\{=\}20andL=60L\{=\}60via shallow clipping\); and “bounded offset” is insufficient \(theii\-parity offsetδ⁡\(i\)∈\{1,2\}\\delta\(i\)\\in\\\{1,2\\\}scores exactly0\.50\.5,2020k\-kernel sweep\)\.*Capacity corollary:*for a training\-length distribution with unbounded support, every fixed kernel with a strict unique global maximizer incurs positive error on copy, soacctrain<1\\mathrm\{acc\}\_\{\\text\{train\}\}<1\. The restriction matters: a kernel peaked atccsolves copy atL∈\{2​c−1,2​c\}L\\in\\\{2c\-1,2c\\\}, so a distribution supported on those two lengths alone can be solved exactly\.

### A\.6Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2): necessity of the hypotheses, and scope

*Why each hypothesis is needed\.**\(a\) The condition is asymptotic\.*RequiringδL​\(i\)=δ∗\\delta\_\{L\}\(i\)=\\delta^\{\*\}on every deep row would be too strong:αL\\alpha\_\{L\}only measures a*fraction*, so a density\-zero exceptional set is compatible withαL→1\\alpha\_\{L\}\\to 1\. Concretely, letδL​\(i\)=2\\delta\_\{L\}\(i\)=2wheniiis a perfect square and11otherwise\. A kernel peaked at offset11fails exactly on the squares, so1−αL=O\(L−1/2\)→01\-\\alpha\_\{L\}=O\(L^\{\-1/2\}\)\\to 0even though the offset is not eventually constant \(αL=0\.944,0\.987,0\.996\\alpha\_\{L\}=0\.944,0\.987,0\.996atL=200,5⋅103,5⋅104L=200,5\\cdot 10^\{3\},5\\cdot 10^\{4\}\)\.*\(b\) Attainability inside the family\.*Constancy alone is not sufficient:δ∗\\delta^\{\*\}must be attainable in𝒦\\mathcal\{K\}, and one cannot simply exhibitκ⁡\(d\)=−\|d−δ∗\|\\kappa\(d\)=\-\|d\-\\delta^\{\*\}\|without checking that it belongs to the PE\-induced family\. Canonical ALiBi is exactly this case: its familyκ⁡\(d\)=c0\+s​d\\kappa\(d\)=c\_\{0\}\+sdwiths≤0s\\leq 0has its maximizer at offset00for every admissible slope, so no ALiBi kernel peaks atδ∗=1\\delta^\{\*\}\{=\}1and ALiBi cannot realize prev\-token positionally, consistent with Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)\. Condition \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(ii\) makes this explicit\.*\(c\) The existence of a global maximizer is load\-bearing\.*Translation invariance alone does not forbid tracking a*growing*offset, because the causal window admits ever larger offsets asLLgrows\. Take the unbounded ladderκ⁡\(d\)=−1\\kappa\(d\)=\-1exceptκ⁡\(20⋅4m\)=m\+2\\kappa\(20\\cdot 4^\{m\}\)=m\+2form=0,1,2,…m=0,1,2,\\dots; its values increase without bound, so it has no maximizer at all, and it attainsαL=1\\alpha\_\{L\}=1on copy along the whole sequenceL=40,160,640,2560,10240,40960L=40,160,640,2560,10240,40960\(we verified this numerically\)\. A kernel that does have a strict unique global maximizerc⁡\(κ\)c\(\\kappa\)can also solve copy at particular lengths, namely atL∈\{2​c​\(κ\)−1,2​c​\(κ\)\}L\\in\\\{2c\(\\kappa\)\-1,2c\(\\kappa\)\\\}, where the demanded offset⌈L/2⌉\\lceil L/2\\rceilequalsc⁡\(κ\)c\(\\kappa\), and possibly at someL≤c⁡\(κ\)L\\leq c\(\\kappa\)where the causal window clips the argmax onto a lower local peak \(forκ\\kappapeaked at100100with a secondary peak at2020we findαL=1\\alpha\_\{L\}=1atL=39,40,199,200L=39,40,199,200\), but it fails for everyL\>2​c​\(κ\)L\>2c\(\\kappa\), which is the content of the corollary\. The impossibility is asymptotic; finitely many lucky lengths are not a counterexample to it\. Growing\-offset impossibility is therefore a statement about kernels with a global peak, and we state it with that qualifier throughout\.*\(d\) Deep rows are defined byc⁡\(κ\)c\(\\kappa\)alone\.*One might also ask fori≥Di\\geq Dfor a support radiusDD, but that would be vacuous for the very families we care about, since RoPE and ALiBi kernels have unbounded support, soD=∞D=\\inftyand no row would be deep\. Globality of the maximizer is what the argument actually needs, and with it the clipped set isℋ⁡\(κ\)⊆\[L0,c⁡\(κ\)\)\\mathcal\{H\}\(\\kappa\)\\subseteq\[L\_\{0\},c\(\\kappa\)\)\.*Scope of the characterization\.*The criterion is on*deep*rows and the asymptoticαL\\alpha\_\{L\}; we do not claim a finite\-LL“complete iff” that also pins shallow rows tomin⁡\(i,δ∗\)\\min\(i,\\delta^\{\*\}\)– that isfalse: the kernelκ=\[1\.46,1\.96,1\.80,6\.32\]\\kappa=\[1\.46,1\.96,1\.80,6\.32\]\(peak at offset33\) makes shallow rowi=2i\{=\}2select offset11, notmin⁡\(2,3\)=2\\min\(2,3\)\{=\}2\(we reproduce this in App\.[A](https://arxiv.org/html/2610.41218#A1)\)\. Shallow rows are a finite,κ\\kappa\-dependent set that does not affect the asymptoticαL\\alpha\_\{L\}\. Two further cautions: the growing\-offset clash is*asymptotic*\(a kernel peaked at offset3030solves copy at bothL0=20L\_\{0\}\{=\}20andL=60L\{=\}60via shallow clipping – there is no two\-length contradiction\); and “bounded offset” is insufficient\. The last point is now exactly condition \([5](https://arxiv.org/html/2610.41218#S5.E5)\)\(i\): anii\-parity offset alternating in\{1,2\}\\\{1,2\\\}is bounded, but each value occurs with density1/21/2, so noδ∗\\delta^\{\*\}has a vanishing exceptional fraction andαL\\alpha\_\{L\}sits at0\.50\.5\. What matters is that one offset occurs with density one; boundedness alone is not enough\. This theorem subsumes the growing\-offset impossibility and the absolute/relative duality\. Capacity corollary: for a training\-length distribution with unbounded support, every fixed kernel with a strict unique global maximizer incurs positive error on copy, soacctrain<1\\mathrm\{acc\}\_\{\\text\{train\}\}<1\. The restriction matters: a kernel peaked atccsolves copy atL∈\{2​c−1,2​c\}L\\in\\\{2c\-1,2c\\\}, so a distribution supported on those two lengths alone can be solved exactly\. We additionally stress\-tested the corrected statement numerically, fuzzing both the kernel \(40004000random kernels under both tie rules, iniff\_constant\_offset\_lg\.py\) and the task’s offset function \(constant, perfect\-square, power\-of\-two and cube exceptional sets, parity, drifting and block\-alternating offsets, and growing offsets\), confirming thatαL→1\\alpha\_\{L\}\\to 1tracks density\-1 constancy and nothing weaker \(experiments/verify\_thm2\_density\.py\); the proof does not rely on these checks\.

### A\.7Proposition[4](https://arxiv.org/html/2610.41218#Thmproposition4)\(Absolute/relative duality\)

*Relative*κ⁡\(i−j\)\\kappa\(i\-j\): a fixed offsetkkis realized by aκ\\kappawhose strict unique global maximizer isd=kd\{=\}k, correct on every row at every length \(αL=1\\alpha\_\{L\}\{=\}1\)\. Attend\-first \(t=0t\{=\}0\) demands the growing offsetii, so by Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)noκ\\kappa*with a strict unique global maximizer*reachesαL→1\\alpha\_\{L\}\\to 1\. This is the right qualifier: a monotone increasing kernel such asκ⁡\(d\)=d\\kappa\(d\)=dhas no global maximizer and does realize attend\-first at every length \(the causal window truncates atd=id\{=\}i\), so what defeats attend\-first in practice is the sign constraint on the canonical ALiBi slope together with theL0L\_\{0\}\-fit’s failure to select such a kernel, not translation invariance by itself; the selected RoPE kernel decays asαL=0\.80,0\.40,0\.20,0\.12,0\.055,0\.039\\alpha\_\{L\}=0\.80,0\.40,0\.20,0\.12,0\.055,0\.039atL=30,40,60,128,256,512L=30,40,60,128,256,512\.*Clamped learned\-absolute*φ⁡\(i\)=vmin⁡\(i,L0−1\)\\varphi\(i\)=v\_\{\\min\(i,L\_\{0\}\-1\)\}: attend\-first is realized by the length\-independent column\-00\-hot kernel \(margin\+1\+1at all lengths\)\. A fixed relative offset moves the target across absolute positions into the collapsed range, where all keysj≥L0j\\geq L\_\{0\}share one feature and their logits tie exactly, soαL=0\\alpha\_\{L\}\{=\}0with margin00\. For the i\.i\.d\.\-extended variant the corresponding statement is the chance bound of Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\. Growing offsets defeat both\. Verified for the*clamped*object insrc/spectra\.py\(attend\-first1\.001\.00; prev0\.000\.00with margin exactly00\) and for the*i\.i\.d\.\-extended*object inexperiments/duality\_abs\_vs\_rel\.py\(prev0\.0460\.046, the chance level of Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)\)\. Induality\_abs\_vs\_rel\.pythe relative side scores0\.000\.00on attend\-first and1\.001\.00on prev; the concrete PEs behave as their own families dictate, so RoPE scores0\.800\.80on attend\-first atL=30L\{=\}30and decays withLL\(Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\(a\)\), while ALiBi scores0\.000\.00on prev \(Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)\)\. Note thatduality\_abs\_vs\_rel\.py’s attend\-first “absolute” construction is the unrestricted kernelK\[i,j\]=𝟏\[j=0\]K\[i,j\]=\\mathbf\{1\}\[j\{=\}0\], i\.e\. object \(1\) above, which is why it is reported separately from the clamped values\.□\\square

## Appendix BFailure modes ofαL\\alpha\_\{L\}

##### Failure modes ofαL\\alpha\_\{L\}\.

We list the five waysαL\\alpha\_\{L\}departs from a trained model’s length generalization\. They are the precise boundary of the single\-head positional theory, and each is something we observe and characterize\.

1. 1\.Selection gap\.A rescuing kernelWWexists, but theL0L\_\{0\}\-fit does not select it, soαL<1\\alpha\_\{L\}<1despite realizability in principle \(sinusoidal fixed\-offset:αL=0\.45\\alpha\_\{L\}\{=\}0\.45atL=60L\{=\}60; Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)\)\.*Existence≠\\neqselection\.*
2. 2\.Content rescue\.The target is carried by the*content*channel, outside the abstraction, so the positionalαL\\alpha\_\{L\}under\-predicts: LG occurs withαL=0\\alpha\_\{L\}\{=\}0\(induction, and ALiBi/NoPE prev/first at depth; App\.[H](https://arxiv.org/html/2610.41218#A8)\)\. This is the binding limitation \(S3\)\.
3. 3\.Depth / multi\-head escape\.Multiple heads or layers route around the single\-head impossibility \(an anchor head\+\+a content\-match head solves copy at depth11\), soαL\\alpha\_\{L\}can under\-predict at scale while remaining a*sufficient screen*\(§[6](https://arxiv.org/html/2610.41218#S6)\)\.
4. 4\.Low\-frequency relative under\-prediction\.Even within the positional channel and on a*content\-neutral*task,αL\\alpha\_\{L\}is not a perfect graded predictor: on content\-neutral skip\-kka frequency\-scaled \(low\-frequency\) RoPE model generalizes a target the linear\-kernel fit deems unrealizable \(Table[10](https://arxiv.org/html/2610.41218#A8.T10), App\.[I](https://arxiv.org/html/2610.41218#A9)\), soαL\\alpha\_\{L\}under\-predicts it\. The task is content\-neutral, so the discrepancy is not content rescue in the sense of \(2\); we have not isolated the mechanism further, and we report it as an unexplained case where the trained model generalizes and the fitted kernel does not\.
5. 5\.Overprediction by the fitted kernel\.The converse also occurs: the fitted kernel can have highαL\\alpha\_\{L\}while the trained model selects a different solution and fails to generalize \(rope\-firstatL=30L\{=\}30,αL=0\.80\\alpha\_\{L\}\{=\}0\.80against LG0\.130\.13;sinusoidal\-firstatL=80L\{=\}80on one seed\)\. Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)is unaffected, since it is a statement about a single realizing kernel; what fails is the empirical bridge fromαL​\(Wfit\)\\alpha\_\{L\}\(W\_\{\\mathrm\{fit\}\}\)to the behaviour ofWtrainedW\_\{\\mathrm\{trained\}\}\.

The five are distinct: \(1\) is intra\-positional selection, \(2\) is a different channel, \(3\) is a larger architecture, \(4\) is a graded\-versus\-binary gap, and \(5\) is a mismatch between the fitted and the trained kernel\. The first four leave the*sufficiency*direction \(αL→1⇒\\alpha\_\{L\}\{\\to\}1\\RightarrowLG\) intact and are failures of*necessity*, which is why we callαL\\alpha\_\{L\}a one\-sided screen; \(5\) is the one case where highαL\\alpha\_\{L\}does not co\-occur with length generalization, which is why we report the empirical link as a co\-occurrence rate\.

### B\.1Scope of the individual statements

The following limits attach to specific results and are easy to over\-read\. \(i\) No general sinusoidal LG impossibility: Conjecture[1](https://arxiv.org/html/2610.41218#Thmconjecture1)is open and the intuition that a sinusoidal encoding must fail once the required offset exceeds its lowest\-frequency period is*refuted*\(Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)\)\. \(ii\) Sinusoidal fixed\-offset transfer is not exact: existence of a rescuingWWdoes not imply its selection by theL0L\_\{0\}\-fit\. \(iii\) The learned\-absolute rate1/\(k−1\)1/\(k\-1\)is*conditional*on the argmax falling among the off\-diagonal new keys, and applies when the target is one of them; the self\-key is a non\-exchangeable quadratic competitor, which is why it is1/\(k−1\)1/\(k\-1\)and not1/k1/k, and the unconditional rate is smaller\. \(iv\) Prop\.[1](https://arxiv.org/html/2610.41218#Thmproposition1)is about the*unique positive\-margin*argmax; set\-containment is a different statement \(which fails on the measure\-zero tie locus\)\. \(v\) All impossibility results are single\-head, positional\-only; multi\-head/depth/content provably route around them\. \(vi\) Prop\.[2](https://arxiv.org/html/2610.41218#Thmproposition2)proves chance via i\.i\.d\.*random*untrained features, whereas our learned\-absolute implementation*clamps*φ⁡\(i\)=emin⁡\(i,L0−1\)\\varphi\(i\)\{=\}e\_\{\\min\(i,L\_\{0\}\-1\)\}\(matching the trained model\); both routes give chance, by exchangeability and by degeneracy respectively, but they are distinct mechanisms and we do not conflate them\.

## Appendix CExperimental details

Architecture\.Decoder\-only transformer,dmodel=64d\_\{\\text\{model\}\}=64, GELU MLP \(width4​d4d\), pre\-LN, learned token embedding \(vocab1212\)\. PEs: NoPE; sinusoidal \(additive\); learned\-absolute \(table of sizeL0L\_\{0\}, positions≥L0\\geq L\_\{0\}clamped toL0−1L\_\{0\}\-1\); RoPE \(rotary onq,kq,k\); ALiBi \(fixed geometric slopes\)\. Tiers: 1\-layer/1\-head, 2\-layer/4\-head, 4\-layer/4\-head\.Training\.AdamW \(lr3×10−43\\times 10^\{\-4\}, wd0\.010\.01\), batch6464,20002000steps, variable train lengthn∼Unif⁡\[L0/2,L0\]n\\sim\\mathrm\{Unif\}\[L\_\{0\}/2,L\_\{0\}\],L0=20L\_\{0\}=20, cross\-entropy over query positions \(ignore\-index elsewhere\)\.Tasks\(target positiontn​\(i\)t\_\{n\}\(i\), all position\-dependent via small vocab\): prev \(i−1i\{\-\}1\), skip\-kk\(i−ki\{\-\}k,k∈\{2,3,5\}k\\in\\\{2,3,5\\\}\), copy \(i−⌈n/2⌉i\{\-\}\\lceil n/2\\rceil\), reverse, first \(00\)\.Evaluation\.Accuracy on extrapolation query rows\[L0,L\)\[L\_\{0\},L\)atL∈\{30,40,50,60,80\}L\\in\\\{30,40,50,60,80\\\},55seeds\.Predictor fit\.Cross\-entropy fit of the kernel coefficients to theL0L\_\{0\}targets \(400400Adam steps\), PE\-structural constraints imposed \(RoPE relative basis; ALiBi recency slope constrained≤0\\leq 0\); strict\-unique\-argmax scoring on new rows\.Compute\.Apple M3 Pro \(MPS\); the full26252625\-measurement,55\-seed study runs in∼\\sim55 min\.

## Appendix DAdditional results

Table[4](https://arxiv.org/html/2610.41218#A4.T4)reports the depth trend \(H3\) over all three tiers; Fig\.[6](https://arxiv.org/html/2610.41218#A4.F6)plots it\. As depth grows \(1→2→41\\to 2\\to 4layers\), the learnable\-cell*graded*rank correlation drops from0\.900\.90\(CI\[0\.73,0\.95\]\[0\.73,0\.95\]\) to0\.310\.31and0\.280\.28with bootstrap CIs that include00\(Table[4](https://arxiv.org/html/2610.41218#A4.T4)\); the precision on high\-αL\\alpha\_\{L\}rows instead stays high and tight \(0\.925→0\.94→0\.970\.925\\to 0\.94\\to 0\.97\)\. The lost graded signal is explained by the content channel: the set of low\-αL\\alpha\_\{L\}cells that nonetheless generalize*grows*with depth\. Scoring cells by their means, it is\{\\\{ALiBi prev/first/skip2, NoPE prev/first, RoPE first\}\\\}at22layers and gains ALiBi skip3 at44layers, with the two marginal members strengthening as depth grows \(NoPE prev0\.53→0\.740\.53\\to 0\.74, RoPE first0\.83→1\.000\.83\\to 1\.00\)\. Depth does not break the sufficient\-indicator property; it adds more content\-channel routes around a low positionalαL\\alpha\_\{L\}, exactly the \(S3\) boundary\.

Table 4:Depth trend \(H3\), learnable cells \(train​acc\>0\.8\\mathrm\{train\\ acc\}\>0\.8\),55seeds\. The Spearman column is*cell\-level*with a bootstrap95%95\\%CI over the \(PE, task\) cells; the precision column is the ratePr⁡\[LG≥0\.7∣αL≥0\.5\]\\Pr\[\\mathrm\{LG\}\\geq 0\.7\\mid\\alpha\_\{L\}\\geq 0\.5\]with a bootstrap95%95\\%CI over the high\-αL\\alpha\_\{L\}*rows*\.αL\\alpha\_\{L\}stays a high\-precision*sufficient*indicator at every depth \(precision CIs well above chance\); the graded rank correlation, by contrast, is significant only at depth11– at depths2,42,4its CI*includes00*, consistent withαL\\alpha\_\{L\}acting as a binary screen rather than a graded predictor once content rescues low\-αL\\alpha\_\{L\}cells\.Figure 6:H3: learnable\-cell Spearmanρ⁡\(αL,LG\)\\rho\(\\alpha\_\{L\},\\mathrm\{LG\}\)\(circles\) drops with depth \(its bootstrap CI includes00at depths2,42,4\), while sufficiency precision \(squares\) stays≥0\.92\\geq 0\.92with tight CIs\.αL\\alpha\_\{L\}remains a*sufficient screen*at all depths\.##### Inductive\-bias ablation \(which fit drives the correlation\)\.

Table[5](https://arxiv.org/html/2610.41218#A4.T5)computesαL\\alpha\_\{L\}three ways on the75751L1H\(PE,task,L\)\(\\text\{PE,task\},L\)rows and correlates each with empirical LG\. The min\-norm interpolant of Theorem[1](https://arxiv.org/html/2610.41218#Thmtheorem1)yields onlyρ=0\.56\\rho\{=\}0\.56; the regularized cross\-entropy fit used throughout yields0\.910\.91, and its weight\-decay→0\\to 0limit, which approaches the max\-margin direction\([Soudry et al\., 2018](https://arxiv.org/html/2610.41218#bib.bib18)\), yields0\.920\.92\. The predictive content ofαL\\alpha\_\{L\}therefore comes from the cross\-entropy direction rather than from the min\-norm object, a gap we state here\.

Table 5:Inductive\-bias ablation \(7575rows in the 1L1H setting\)\. The cross\-entropy fit, not the min\-norm interpolant, drives theαL\\alpha\_\{L\}–LG correlation\.
##### Robustness checks\.

The learnability cutoff is inert: cell\-levelρ=0\.91/0\.90/0\.90\\rho=0\.91/0\.90/0\.90attrain\>\{0\.5,0\.7,0\.9\}\\mathrm\{train\}\>\\\{0\.5,0\.7,0\.9\\\}\(train accuracy is bimodal\)\. On the2020dropped cellsαL\\alpha\_\{L\}has std0\.020\.02\(uniformly≈0\\approx 0\), so it provides no rank signal there by construction\. Scale robustness \(vocab6464,d=128d\{=\}128\) is reported in App\.[F](https://arxiv.org/html/2610.41218#A6); training\-length robustness \(the screen AUC stays1\.01\.0and the per\-cell verdicts are unchanged atL0∈\{10,20,40\}L\_\{0\}\\in\\\{10,20,40\\\}\) in App\.[G](https://arxiv.org/html/2610.41218#A7); and PE frequency/θ\\theta\-base/dimension robustness in App\.[I](https://arxiv.org/html/2610.41218#A9)\. Interior fixed\-absolute positions, which remove the reliance on the special boundary position00, are in App\.[E](https://arxiv.org/html/2610.41218#A5)\.

## Appendix EInterior fixed\-absolute positions

The fixed\-absolute regime in the main grid is represented byfirst, whose target is position00\. That position is special: it is the causal boundary, so a monotone recency kernel can reach it without addressing an absolute position at all\. To check that the absolute side of the duality does not depend on this artefact, we evaluate two interior targets,pos2andpos5\(attend to absolute position22, resp\.55\), and ran the full study on them:55PEs×\\times22tasks×\\times33depth tiers×\\times55seeds×\\times55lengths,750750further trained\-model measurements\.

The separation is if anything cleaner than at the boundary \(Table[6](https://arxiv.org/html/2610.41218#A5.T6)\)\. The absolute encodings solve both interior positions and transfer them perfectly \(α60=1\.00\\alpha\_\{60\}=1\.00, LG≥0\.97\\geq 0\.97atL=80L=80\), while the relative encodings neither learn nor transfer them\. The screen AUC over these ten cells is1\.001\.00, withαL=1\.00\\alpha\_\{L\}=1\.00on every generalizing cell and0\.000\.00on every failing one\. The boundary artefact is visible in the comparison: RoPE scoresα60=0\.20\\alpha\_\{60\}=0\.20onfirstbut0\.000\.00on both interior positions, i\.e\. the residual score at position00came from the causal boundary rather than from any ability to address absolute positions\.

Table 6:Interior fixed\-absolute positions \(1L1H, mean over55seeds\)\. Absolute encodings realize and transfer them; relative encodings do not learn them at all\. Comparefirst, where RoPE scoresα60=0\.20\\alpha\_\{60\}=0\.20purely because position00is the causal boundary\.
## Appendix FScale robustness

BecauseαL\\alpha\_\{L\}depends only on\(Φ,tn\)\(\\Phi,t\_\{n\}\), it is*identical*across model scales; the question is whether the empirical LG it predicts is stable\. We re\-ran the full study at a5×5\\timeslarger vocabulary \(6464vs1212\) and2×2\\timeswidth \(d=128d\{=\}128vs6464\)\. The 1L1H cell\-level correlation is*unchanged*\(indeed slightly higher\):ρ=0\.92\\rho=0\.92\(vs0\.900\.90atd=64d\{=\}64\), with sufficiency precision0\.940\.94; the scale tier moves→0\.450\.36\\\!\\to\\\!0\.45\(Table[7](https://arxiv.org/html/2610.41218#A6.T7)\)\. The headline relationship is therefore not an artifact of the tiny vocabulary or width\. \(This is a robustness check within the synthetic position\-driven family, not a claim about natural\-language LLMs; see §[7](https://arxiv.org/html/2610.41218#S7)\.\)

Table 7:Scale robustness: cell\-level Spearmanρ⁡\(αL,LG\)\\rho\(\\alpha\_\{L\},\\mathrm\{LG\}\)on learnable cells, at the paper’s default size vs\. a larger model\. The predictor is identical; only the trained models change\. The scale study uses22seeds \(700700rows\), and both columns are computed on the cells the two studies have in common, so the22L44H entry \(0\.360\.36\) is over1919cells rather than the2020of Table[4](https://arxiv.org/html/2610.41218#A4.T4), where the same quantity over all2020cells is0\.310\.31\.
## Appendix GTraining\-length \(L0L\_\{0\}\) sensitivity

Is the clean separation an artifact of the default training lengthL0=20L\_\{0\}\{=\}20? We re\-ran the 1\-layer 1\-head study atL0∈\{10,20,40\}L\_\{0\}\\in\\\{10,20,40\\\}, each evaluated at3×L03\\times L\_\{0\}\(so the extrapolation factor is held fixed\),33seeds\. The screen AUC is1\.0001\.000at everyL0L\_\{0\}, and*the same four cells generalize*in each case \(rope prev/skip\-3, sinusoidal/learned\-abs first\); the Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)verdicts are unchanged \(relative bounded offsets transfer; relative growing offsets do not\)\. The separation is therefore not specific toL0=20L\_\{0\}\{=\}20\(Table[8](https://arxiv.org/html/2610.41218#A7.T8)\)\.

Table 8:L0L\_\{0\}\-sensitivity \(1L1H, predictorαL\\alpha\_\{L\}/ empirical LG atL=3​L0L\{=\}3L\_\{0\}\)\. The screen AUC is1\.01\.0at every training length and the per\-cell verdicts are stable; the separation is not an artifact ofL0=20L\_\{0\}\{=\}20\. Generated byexperiments/l0\_sensitivity\.py\.
## Appendix HA content\-aware extension:αL\+\\alpha\_\{L\}^\{\+\}\(with a demonstration\)

The binding limitation is \(S3\): where the content channel carries the target, the positionalαL\\alpha\_\{L\}under\-predicts\. We show the one\-sidedness is closable, using a content\-driven task, and state what is and is not recovered\.

##### Setup\.

Augment the realizable space from the positional kernel𝒦n\\mathcal\{K\}\_\{n\}to𝒦n\+𝒞\\mathcal\{K\}\_\{n\}\+\\mathcal\{C\}, where𝒞\\mathcal\{C\}is the content\-matching channele​\(xi\)⊤​Wc​e​\(xj\)e\(x\_\{i\}\)^\{\\top\}W\_\{c\}e\(x\_\{j\}\)\. DefineαL\+=max⁡\(αL,αLcontent\)\\alpha\_\{L\}^\{\+\}=\\max\(\\alpha\_\{L\},\\ \\alpha\_\{L\}^\{\\mathrm\{content\}\}\), whereαLcontent=1\\alpha\_\{L\}^\{\\mathrm\{content\}\}=1iff the target is realizable by a*length\-stationary*content rule \(a token match composed with a fixed shift\)\. The canonical such task isinduction: the target atiiis the token following the previous occurrence ofxix\_\{i\}– content\-determined, henceαL=0\\alpha\_\{L\}\{=\}0for every PE \(no length\-dependent positional rule realizes it\), whileαLcontent=1\\alpha\_\{L\}^\{\\mathrm\{content\}\}\{=\}1\.

##### Demonstration \(Table[9](https://arxiv.org/html/2610.41218#A8.T9)\)\.

We train induction across PEs\. At11layer the circuit cannot form \(train acc≈0\.25\\approx 0\.25\); at22layers it is learned \(train0\.760\.76–1\.01\.0\) and*generalizes*, strongly for ALiBi \(LG=0\.84=0\.84\), partially for RoPE/NoPE \(0\.43/0\.340\.43/0\.34\)\. So positionalαL=0\\alpha\_\{L\}=0*mispredicts*a cell with LG0\.840\.84– a real one\-sidedness failure – whichαL\+\\alpha\_\{L\}^\{\+\}correctly flags as realizable through the content channel\.Caveat:induction LG is PE\-*modulated*\(0\.260\.26–0\.840\.84: the content circuit still needs a working prev\-token head, which absolute PEs support poorly\), so the*binary*αLcontent\\alpha\_\{L\}^\{\\mathrm\{content\}\}over\-predicts for sinusoidal/learned\-abs\. A graded content channel weightingαLcontent\\alpha\_\{L\}^\{\\mathrm\{content\}\}by the PE’s support for the match circuit – and the multi\-head extension that provably escapes Theorem[2](https://arxiv.org/html/2610.41218#Thmtheorem2)\(anchor head\+\+content\-match head solves copy at depth 1\) – are the natural next steps\. The point is narrow: the positional indicator’s blind spot is identifiable, and the content channel recovers a cell it misses\.

##### A frequency sweep for a discriminating cell\.

One may reasonably ask for one \(PE, task\) cell whereαL\\alpha\_\{L\}beats the relative/absolute taxonomy\. We searched the most promising direction: a*frequency\-scaled RoPE*\(still relative\) on the*content\-neutral*skip\-kktask \(the query token is independent of the targets⁡\[i−k\]s\[i\{\-\}k\], so content cannot supply the answer\), sweeping the frequency scale to provoke a too smooth to separate nearby offsets failureαL\\alpha\_\{L\}would catch\. The result is negative \(Table[10](https://arxiv.org/html/2610.41218#A8.T10)\): standard and moderately\-low frequencies resolve the offset \(αL=1\\alpha\_\{L\}\{=\}1, LG≈1\\approx 1\); at scale0\.030\.03the trained head*still*generalizes \(LG=0\.88=0\.88\) whileαL=0\\alpha\_\{L\}\{=\}0– anαL\\alpha\_\{L\}*false negative*, with the trained model generalizing a target the linear\-kernel fit deems unrealizable; and at scale0\.010\.01the model fails to learn even in\-distribution \(excluded by the learnability filter\)\.No discriminating cell exists in this family, and the probe instead reveals thatαL\\alpha\_\{L\}under\-predicts low\-frequency relative PEs\.Combined with the row\-level over\-predictions \(rope\-first\), the verdict is thatαL\\alpha\_\{L\}*matches but does not beat*the taxonomy as a predictor; its contribution is the theory that*explains*the taxonomy, not a superior predictor\. \(The low\-frequency case is not content rescue, since skip\-kkis content\-neutral; we do not claim to have identified the mechanism behind it\.\)

Table 9:Content\-driven induction \(target==token after the previous occurrence ofxix\_\{i\}\): positionalαL=0\\alpha\_\{L\}=0for all PEs, yet22\-layer models generalize \(LG up to0\.840\.84\)\.αL\+\\alpha\_\{L\}^\{\+\}flags the realizable through the content channel regimeαL\\alpha\_\{L\}misses; residual PE\-modulation of content LG is future work\.Table 10:RoPE frequency sweep on content\-neutral skip\-3 \(1L1H\)\. No discriminating cell: where the model generalizes the taxonomy is right; at scale0\.030\.03αL\\alpha\_\{L\}under\-predicts \(false negative\); at0\.010\.01the model cannot learn at all\.

## Appendix IFrequency and dimension sensitivity ofαL\\alpha\_\{L\}

SinceαL\\alpha\_\{L\}is computed from the PE feature map, it is fair to ask whether its verdict is an artifact of our default frequency count,θ\\theta\-base, or feature dimension\. We sweep each \(predictor\-only, no training;L0=→L=60L\_\{0\}\{=\}20\\\!\\to\\\!L\{=\}60\) and find a clean separation: the realizability*verdict*is stable wherever the theory makes a clean claim, and the only cells whose realizability verdict moves are exactly the sinusoidal bounded\-offset cells the theory flags as a selection gap \(Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)\); a few other cells show small sub\-threshold raw movement that never flips the verdict\.

Table 11:αL\\alpha\_\{L\}for RoPE under varying frequency countnum\\mathrm\{num\}\(feature dimensionm=2​numm\{=\}2\\,\\mathrm\{num\}\) andθ\\theta\-base\. The verdict is invariant: fixed relative offsets transfer \(prev,skip3=1\.00\\mathrm\{prev,skip3\}\{=\}1\.00everywhere\), growing offsets never transfer \(copy/reverse≈0\{\\approx\}0\), and the kernel theL0L\_\{0\}\-fit selects does not hit a fixed absolute position \(first stays at chance; a monotone kernel in the family could, but the fit does not select it, see the discussion after Prop\.[4](https://arxiv.org/html/2610.41218#Thmproposition4)\)\. For low\-frequency*scaling*see Table[10](https://arxiv.org/html/2610.41218#A8.T10)\.Table 12:αL\\alpha\_\{L\}for sinusoidal under the same sweep\. The absolute verdict is invariant \(first≈1\{\\approx\}1everywhere; growing offsets≈0\{\\approx\}0\)\. The bounded\-offset cells \(prev/skip3\) are the only entries whose verdict moves with frequency – they*degrade*as frequencies are added \(num:8→32\\mathrm\{num\}\{:\}8\{\\to\}32drives prev0\.47→0\.000\.47\{\\to\}0\.00\); the other cells’ rawαL\\alpha\_\{L\}shifts stay sub\-threshold \(e\.g\. first0\.93→1\.000\.93\{\\to\}1\.00\) and never flip the verdict\. This is what the theory predicts: enlarging the realizable space does not help theL0L\_\{0\}\-fit*select*a transferring kernel \(existence≠\\neqselection, Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)\)\. Verdict movement appears precisely at the selection gap and nowhere else\.Takeaway\.Across frequency count,θ\\theta\-base, and feature dimension,αL\\alpha\_\{L\}’s*binary realizability verdict*is stable for every regime with a clean theoretical prediction; the residual frequency dependence is confined to the sinusoidal bounded\-offset selection gap, which is the behaviour Prop\.[3](https://arxiv.org/html/2610.41218#Thmproposition3)predicts\. The screen is therefore not an artifact of the default PE hyperparameters\. \(Generated byexperiments/freq\_dim\_sensitivity\.py\.\)

## Appendix JReproducibility checklist

The full source \(predictor, models, tasks, experiments and theory\-verification scripts\) is released at[https://github\.com/superajgklasjglaks/positional\-realizability](https://github.com/superajgklasjglaks/positional-realizability)\.

- •Predictor\.αL\\alpha\_\{L\}is insrc/spectra\.py\(simulate\(pe,task,L0L\_\{0\},LL\)\); it uses no end\-to\-end Transformer training, no data samples, and no learned Transformer parameters – only a deterministic kernel\-selection on the PE basis\. Models:src/model\.py\(the five PEs\); tasks:src/data\.py\.
- •Seeds\.The main study uses seeds\{0,1,2,3,4\}\\\{0,1,2,3,4\\\}\(fixed inrun\_study\.py\); bootstrap CIs resample cells for the correlations and high\-αL\\alpha\_\{L\}rows for the precisions\. All randomness is seeded\.
- •Configuration\.The design grid \(PEs×\\timestasks×\\timeslengths×\\timesseeds×\\timesdepth tiers\) is Table[2](https://arxiv.org/html/2610.41218#S6.T2); architecture/optimizer/training details are in App\.[C](https://arxiv.org/html/2610.41218#A3)\. Tiers and tags are env\-overridable inrun\_study\.py\(SEEDS,DMODEL,TIERS\_ONLY,STUDY\_TAG\); the run is resumable\.
- •Data behind every number\.results/study\_rows\.json\(all26252625rows, including runs the model failed to learn atL0L\_\{0\}– no failed runs are silently dropped; the learnability filtertrain\_acc\>0\.8\{\>\}0\.8is applied explicitly and reported alongside the unconditioned numbers\)\. Summaries inresults/study\_summary\.json\.
- •Figures/tables→\\toscripts\.Figs\.[3](https://arxiv.org/html/2610.41218#S6.F3)–[6](https://arxiv.org/html/2610.41218#A4.F6)and the heatmaps:make\_figures\.py\. Headline statistics \(AUC, permutationpp, Spearman \+ bootstrap CIs, sufficiency precision\):review\_stats\.py\. Scale robustness \(Table[7](https://arxiv.org/html/2610.41218#A6.T7)\):run\_study\.pywithSTUDY\_TAG=scale\.L0L\_\{0\}\-sensitivity \(Table[8](https://arxiv.org/html/2610.41218#A7.T8)\):l0\_sensitivity\.py\. Frequency/dimension sensitivity \(Tables[11](https://arxiv.org/html/2610.41218#A9.T11)–[12](https://arxiv.org/html/2610.41218#A9.T12)\):freq\_dim\_sensitivity\.py\. Content extension / induction \(Table[9](https://arxiv.org/html/2610.41218#A8.T9)\) and the RoPE frequency sweep \(Table[10](https://arxiv.org/html/2610.41218#A8.T10)\):ablation\_fit\.pyand the content study\.
- •Theory verification\.The following claims have executable checks; the list is exhaustive, and claims not named here are supported by their proofs alone: relative growing\-offset impossibility \(theorem\_relative\_copy\_impossibility\.py\), the deep\-row iff and its counterexampleκ=\[1\.46,1\.96,1\.80,6\.32\]\\kappa\{=\}\[1\.46,1\.96,1\.80,6\.32\]\(iff\_constant\_offset\_lg\.py\), learned\-absolute chance1/\(k−1\)1/\(k\{\-\}1\)\(verify\_learned\_abs\_fixed\_position\_iff\.py,theorem2b\_learned\_abs\_montecarlo\.py\), the absolute/relative duality \(duality\_abs\_vs\_rel\.py\), and the Conjecture[1](https://arxiv.org/html/2610.41218#Thmconjecture1)cone/selection probes \(underdetermined\_conj1\_\*\.py\)\.
- •Compute\.The full study runs in≈55\{\\approx\}55min on one Apple M3 Pro \(MPS\); the predictor and all sensitivity sweeps run in seconds on CPU\.