Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention

# Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention
Source: [https://arxiv.org/html/2608.19203](https://arxiv.org/html/2608.19203)
###### Abstract

Standard multi\-head attention \(MHA\) gives every head the same full causal context span, although heads can serve different contextual roles\. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer\-range relations such as entity interactions, discourse links, or state changes\. We presentAsymmetricAttentionHeads \(AAH\), a head\-wise context\-allocation framework that treats context length as an explicit per\-head or per\-group allocation variable\. AAH groups heads using feature\-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface\. In 4096\-token seed\-0 experiments, several AAH\-style local\-allocation variants achieve lower validation loss than pure full attention\. Short\-budget ablations show that stable local allocation and head\-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy\. We interpret AAH as a structured head\-wise context\-allocation mechanism for quality and analysis, with Attention Coverage Ratio \(ACR\) reported as a selected\-window routing diagnostic\.

\(a\) Standard MHAInput hidden statesShared Q/K/V projectionHeadHeadHeadHeadFull causal spanfor every headStandard causalattentionFlat head concatOutput projectionStandard block outputUniform full\-span execution\(b\) AAH\-v3Input hidden statesShared Q/K/V projectionControl branchQ/K/V headfeaturesEMA smoothingHierarchyHeadHeadHeadHeadGroupGroupGroupUpper hierarchyWide jointsibling scorerFinal per\-headwindow indicesExecution branchGroup heads byselected windowDenseMaskFlexAttentionFlashAttention[1](https://arxiv.org/html/2608.19203#S5.SS10)[2](https://arxiv.org/html/2608.19203#S5.SS10)[3](https://arxiv.org/html/2608.19203#S5.SS10)Scatter outputsto head orderFlat head concatOutput projectionStandard block outputWindow executionselected windows

Figure 1:Standard MHA versus AAH\-v3\. Standard MHA applies one full causal span to all heads before the flat output merge\. AAH\-v3 keeps the shared Q/K/V projection and flat Transformer interface, but adds a control branch that assigns per\-head windows; the execution branch buckets heads by selected window and can use dense\-masked, FlexAttention, or FlashAttention execution\.## 1Introduction

Transformers rely on multi\-head attention \(MHA\)\[[1](https://arxiv.org/html/2608.19203#bib.bib1)\], where each head is typically executed with a uniform attention pattern and full attended span\. This design is simple and robust, but it also hides a structural question: different heads may not need the same amount of context to make useful contributions\. This motivates the central question in this paper:can a Transformer allocate context windows unevenly across heads while preserving, or sometimes improving, language\-modeling quality?

We use the termAsymmetric Attention Heads \(AAH\)for this design direction\. Attention heads remain part of the standard MHA interface, but they are allowed to receive unequal context windows\. AAH\-v3 controls MHA execution across heads and head groups by selecting discrete causal local windows, while preserving the standard Transformer block interface and flat output merge\. The method changes thehead\-wise context\-allocation policy, rather than the output topology of the Transformer block\.

Our evaluation scope is intentionally constrained\. We treatAttention Coverage Ratio \(ACR\)111See[Measuring Attention Coverage and Hardware FLOPs](https://arxiv.org/html/2608.19203#S5.SS10)for the formal ACR definition and FLOPs boundary\.as a selected\-window coverage proxy, not a measured FLOPs metric\. The dense\-masked path does not establish skipped backend work; therefore the main rows omit backend work\-accounting columns\. True GPU FLOPs are measured only by Nsight\-derived GPU floating\-point\-operation counters, and the current Nsight evidence does not support a FLOPs\-reduction claim for the implementations in this paper\.

AAH is backend\-agnostic in the narrow sense that it chooses per\-head context windows and then hands those windows to an execution path\. The current evidence shows that this routing structure can affect quality and selected\-window diagnostics, but it does not show lower measured GPU FLOPs\. We therefore treat AAH primarily as a head\-wise context\-allocation mechanism for quality and interpretability, with systems profiling reported as negative or boundary evidence rather than as a systems\-performance claim\.

Earlier internal variants explored static and preliminary dynamic asymmetry, but are not part of the main paper narrative\.AAH\-v3 is the sole main method studied in this paper\.The present claim is structural: AAH exposes non\-uniform head\-wise context allocation, and the reported experiments test how that structure affects validation quality and routing diagnostics under a fixed 4096\-token setting\.

Our contributions are as follows:

1. 1\.Method\.We present AAH\-v3 as an MHA\-level, cross\-head context\-allocation mechanism using EMA\-smoothed head features, mixed hierarchy construction, enriched controller inputs, wide joint sibling scoring, top\-down parent constraints, and grouped causal local attention\.
2. 2\.Structured allocation diagnostics\.We report ACR as a selected\-window routing diagnostic for head\-wise context allocation, while separating it from measured GPU FLOPs\.
3. 3\.Quality and structure evidence\.We report seed\-0 4096\-token training/inference results and a 3000\-step screening ablation showing that several structured/local allocation policies achieve lower validation loss than pure full attention, while fixed/random controls caution against attributing the gains to adaptive hierarchy alone\.
4. 4\.Claim boundaries\.We report Nsight and FLOPs\-lab diagnostics showing that the current implementations do not establish measured GPU\-FLOPs savings; final confirmation requires multi\-seed and matched longer\-budget controls\.

## 2Motivation and Problem Diagnosis

### 2\.1Uniform head execution as context\-allocation rigidity

Standard MHA assigns a comparable attention pattern and full causal span to every head in a layer\. This is robust, but it can be structurally rigid when heads differ in specialization, redundancy, and context\-dependent utility\. The potential failure mode is not only cost: a uniform full\-span rule may also prevent the model from expressing stable head\-specific context roles\.

AAH targets this diagnosis directly: head\-level context budgets should be allocatable unevenly while preserving standard Transformer block semantics\. The goal is therefore not generic sparsification or a demonstrated GPU\-FLOPs reduction, but*quality\-aware context\-budget redistribution within the multi\-head interface*\.

### 2\.2Why AAH is needed

Early static asymmetry confirms that unequal allocation is possible, but static policies are too rigid across context and training phases\. Preliminary dynamic asymmetry improves flexibility but is sensitive to instability and control noise\.

AAH\-v3 addresses this by using grouped dynamic control with enriched hierarchy\-aware features, joint sibling scoring, and constrained resolution decisions\. This design keeps the external Transformer interface unchanged while exposing a stable MHA\-level context\-allocation policy across heads\.

### 2\.3Routing diagnostics versus hardware profiling

A second diagnosis is methodological: selected\-window coverage is not the same as measured hardware work\. Masking path, kernel behavior, memory traffic, launch overhead, and block scheduling can decouple routing diagnostics from Nsight\-measured GPU floating\-point operations\. AAH reduces selected attention coverage under the routing proxy, but the current implementations do not translate this proxy into lower measured GPU FLOPs\. We therefore treat ACR as a structure diagnostic and evaluate AAH primarily as a head\-wise context\-allocation mechanism for quality and interpretability\.

## 3Related Work

### 3\.1Standard MHA and head redundancy

The Transformer introduced scaled dot\-product attention and multi\-head attention as the core sequence\-mixing mechanism\[[1](https://arxiv.org/html/2608.19203#bib.bib1)\]\. Standard MHA allocates comparable execution patterns and attention span across all heads in a layer, but empirical head\-pruning and head\-specialization analyses show that heads are not equally useful, equally specialized, or equally necessary at inference time\[[2](https://arxiv.org/html/2608.19203#bib.bib2),[3](https://arxiv.org/html/2608.19203#bib.bib3)\]\. AAH builds on this observation but does not prune heads away; it keeps the standard MHA interface and asks whether execution span can be allocated unevenly across heads while preserving quality\.

### 3\.2Sparse, local, and routed attention

A large body of work reduces attention cost by changing the attention pattern itself, including sparse factorization, local/global sparse patterns, and content\-based routing\[[4](https://arxiv.org/html/2608.19203#bib.bib4),[5](https://arxiv.org/html/2608.19203#bib.bib5),[6](https://arxiv.org/html/2608.19203#bib.bib6),[7](https://arxiv.org/html/2608.19203#bib.bib7)\]\. AAH is positioned differently\. It isnota sparse\-attention replacement andnota topology\-changing output design\. Instead, AAH provideshead\-level execution control within standard Transformer block semantics: the output still comes from the usual flat head concatenation and output projection, while the execution branch assigns different causal local window spans to heads or head groups\.

### 3\.3K/V sharing and grouped\-query attention

Multi\-query attention \(MQA\) and grouped\-query attention \(GQA\) reduce decoder\-inference cost by sharing or reducing K/V heads across query heads\[[9](https://arxiv.org/html/2608.19203#bib.bib9),[10](https://arxiv.org/html/2608.19203#bib.bib10)\]\. AAH\-v3 is complementary rather than identical: it keeps the full attention\-head set and changes the selected execution span per head or group\. Thus, AAH targets the attended region and effective attention elements, whereas MQA/GQA primarily target K/V\-cache bandwidth and K/V\-head sharing\.

### 3\.4Adaptive computation and routing

Adaptive computation methods condition the amount or location of computation on the input, from adaptive recurrent computation time to routed sparse expert models\[[11](https://arxiv.org/html/2608.19203#bib.bib11),[12](https://arxiv.org/html/2608.19203#bib.bib12)\]\. Routing Transformer similarly learns content\-dependent sparse attention patterns\[[7](https://arxiv.org/html/2608.19203#bib.bib7)\]\. AAH shares the broad goal of conditional allocation, but the routing unit is the MHA head or head group and the decision space is a discrete set of causal window spans under hierarchy constraints\. In the reported hard\-policy runs, the executed window policy is best understood as feature\-conditioned and statistic\-driven rather than as a proven differentiably learned router\.

Adaptive Attention Span learns per\-head maximum attention spans with a differentiable masking mechanism and a span\-length penalty to reduce memory and computation for long\-context Transformers\[[8](https://arxiv.org/html/2608.19203#bib.bib8)\]\. AAH\-v3 is related in spirit because it assigns unequal attention spans across heads, but differs in using a separate hierarchy\-aware control branch, group\-level decisions, parent constraints, and grouped local execution while preserving the standard flat MHA output interface\.

### 3\.5Systems\-aware attention execution and profiling

IO\-aware exact\-attention kernels such as FlashAttention and FlashAttention\-2 show that attention runtime depends strongly on memory traffic, tiling, parallelism, work partitioning, and kernel implementation, not only on nominal attention elements\[[13](https://arxiv.org/html/2608.19203#bib.bib13),[14](https://arxiv.org/html/2608.19203#bib.bib14)\]\. Programmable local\-attention backends such as PyTorch FlexAttention provide a complementary route for expressing sparse or local attention patterns in backend\-executable form\[[15](https://arxiv.org/html/2608.19203#bib.bib15)\]\. AAH is not a kernel\-optimization method\. FlashAttention/FlexAttention and AAH address different layers of the stack: Flash/Flex optimize or express attention execution, while AAH selects asymmetric per\-head context windows\. We therefore report ACR as a routing diagnostic and reserve GPU\-FLOPs claims for Nsight\-derived floating\-point\-operation counters\.

## 4Background: Standard Multi\-Head Attention

LetX∈ℝT×dX\\in\\mathbb\{R\}^\{T\\times d\}denote the input sequence representation, whereTTis the sequence length, or maximum causal span, andddis model dimension\. For each attention headh∈\{1,…,H\}h\\in\\\{1,\\dots,H\\\}, whereHHis the number of heads,

Qh=X​WhQ,Kh=X​WhK,Vh=X​WhV\.Q\_\{h\}=XW\_\{h\}^\{Q\},\\qquad K\_\{h\}=XW\_\{h\}^\{K\},\\qquad V\_\{h\}=XW\_\{h\}^\{V\}\.HereQhQ\_\{h\},KhK\_\{h\}, andVhV\_\{h\}are the query, key, and value projections for headhh, andWhQW\_\{h\}^\{Q\},WhKW\_\{h\}^\{K\}, andWhVW\_\{h\}^\{V\}are the corresponding projection matrices\.

The head outputOhO\_\{h\}is

Oh=softmax⁡\(Qh​Kh⊤dh\+M\)​Vh,O\_\{h\}=\\operatorname\{softmax\}\\\!\\left\(\\frac\{Q\_\{h\}K\_\{h\}^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\+M\\right\)V\_\{h\},wheredhd\_\{h\}is the head dimension andMMis the causal mask\. The final multi\-head attention outputOOis

O=Concat⁡\(O1,…,OH\)​WO,O=\\operatorname\{Concat\}\(O\_\{1\},\\dots,O\_\{H\}\)W^\{O\},whereWOW^\{O\}is the output projection matrix\.

This formulation has two important properties\. First, all heads operate in parallel on the same layer input\. Second, all heads use the same full\-width attention pattern over the sequence\. In other words, standard MHA allocatesuniform attention computation across heads\.

If each head attends over the full causal context, the dominant attention\-score computation scales proportionally to

∑h=1HT2​dh,\\sum\_\{h=1\}^\{H\}T^\{2\}d\_\{h\},ignoring constant factors and non\-attention terms\. This quadratic dependence on sequence length makes attention a natural target for studying execution\-aware context allocation\.

## 5Method: AAH\-v3

### 5\.1Method overview

AAH\-v3 augments standard multi\-head attention with hierarchical context allocation over heads and head groups\. It preserves the Transformer block contract: the model still formsQ/K/VQ/K/Vprojections, computes scaled dot\-product attention under a causal mask, concatenates head outputs, and applies the usual output projection\. The change is the MHA context policy: different heads may use different causal local windows selected by a hierarchy\-aware controller\. In the reported hard\-policy implementation, the controller/scorer state is fixed from seeded AAH state and operates on feature statistics; the language\-modeling loss trains the executed Transformer path, not a differentiable routing objective for the hard window choices\.

Conceptually, AAH\-v3 has two main parts\.Head groupingsummarizes currentQ/K/VQ/K/Vbehavior and recent attention diagnostics, smooths these signals over time, forms level\-0 head groups, and builds or reuses the upper hierarchy that determines which heads are controlled together\.Window controlthen uses that grouped hierarchy to choose and execute per\-head context windows\. Its selection stage constructs enriched controller inputs, applies joint sibling scoring, enforces parent constraints, and maps group decisions back to heads\. Its execution stage buckets heads by the selected window and runs the chosen attention backend before the usual scatter, flat merge, and output projection\.

The central controller shift from earlier drafts is the move from independent group scoring tojoint sibling scoring\. Rather than asking each group to choose a window in isolation, AAH\-v3 scores sibling groups together using enriched representations and directional differences\. This makes the sibling contrast part of the decision problem itself; the resulting logits replace the paired siblings’ independent logits, and the selected discrete windows are propagated top\-down before head\-level execution\.

This organization separates objects that are easy to conflate: level\-0 head grouping, upper\-level hierarchy construction, hierarchy\-aware decision scoring, group\-to\-head window mapping, and the backend that physically executes the selected windows\. The main 4096\-token experiments use the current dense\-masked, grouped local execution path\. Later, Subsection[7\.7](https://arxiv.org/html/2608.19203#S7.SS7)reports backend and Nsight diagnostics that test whether the selected\-window proxy becomes lower measured GPU FLOPs; the current answer is no\.

### 5\.2Formal bridge: from uniform spans to head\-wise context allocation

Standard MHA can be viewed as the special case in which every head receives the same causal context window,

Wh=Tfor every head​h,W\_\{h\}=T\\qquad\\text\{for every head \}h,whereWhW\_\{h\}is the causal window assigned to headhhandTTis the sequence length\. AAH changes this allocation rule\. Instead of fixing allWhW\_\{h\}to the maximum span, the controller selects a discrete indexchc\_\{h\}for each head and maps it to a causal window,

Wh∈𝒲,𝒲=\{512,1024,2048,4096\}​in the main 4096\-token setting\.W\_\{h\}\\in\\mathcal\{W\},\\qquad\\mathcal\{W\}=\\\{512,1024,2048,4096\\\}\\ \\text\{in the main 4096\-token setting\}\.This makes AAH a head\-wise context\-allocation mechanism rather than a new output topology: the model still projects Q/K/V, computes causal attention, concatenates heads, and applies the standard output projection\.

### 5\.3Head features and EMA smoothing

For each headhh, wherehhindexes an attention head, AAH\-v3 forms a per\-head base featurexhx\_\{h\}from currentQ/K/VQ/K/Vstatistics and previous attention diagnostics:

xh=\[μ​\(\|qh\|\),σ​\(qh\),μ​\(\|kh\|\),σ​\(kh\),μ​\(\|vh\|\),σ​\(vh\),eh,nh,ρh\]\.x\_\{h\}=\[\\mu\(\|q\_\{h\}\|\),\\ \\sigma\(q\_\{h\}\),\\ \\mu\(\|k\_\{h\}\|\),\\ \\sigma\(k\_\{h\}\),\\ \\mu\(\|v\_\{h\}\|\),\\ \\sigma\(v\_\{h\}\),\\ e\_\{h\},\\ n\_\{h\},\\ \\rho\_\{h\}\]\.Hereqh,kh,vhq\_\{h\},k\_\{h\},v\_\{h\}denote the current query, key, and value activations for headhh\. The statistics are scalar reductions over the batch, sequence, and per\-head feature axes for that head\. The entriesμ​\(\|qh\|\)\\mu\(\|q\_\{h\}\|\),μ​\(\|kh\|\)\\mu\(\|k\_\{h\}\|\), andμ​\(\|vh\|\)\\mu\(\|v\_\{h\}\|\)are mean absolute activations reduced over batch, sequence, and head\-feature dimensions while preserving the head index\. The implementation does not compute\|μ​\(qh\)\|\|\\mu\(q\_\{h\}\)\|,\|μ​\(kh\)\|\|\\mu\(k\_\{h\}\)\|, or\|μ​\(vh\)\|\|\\mu\(v\_\{h\}\)\|\. Group\-level Q/K/V magnitude entries use the same mean\-absolute convention, with the reduction additionally averaging over the heads inside the group\. The operatorσ​\(⋅\)\\sigma\(\\cdot\)denotes the corresponding scalar standard deviation\. The terms\(eh,nh,ρh\)\(e\_\{h\},n\_\{h\},\\rho\_\{h\}\)are previous attention diagnostics: entropy, output norm, and usage from the most recent stored attention statistics\.

This base featurexhx\_\{h\}is not yet the controller input\. It is first smoothed and aggregated into group features; the enriched\-controller stage later adds hierarchy\-level, parent, global, and size information\.

Head features are temporally smoothed before hierarchy construction:

x¯h,t=α​x¯h,t−1\+\(1−α\)​xh,t\.\\bar\{x\}\_\{h,t\}=\\alpha\\bar\{x\}\_\{h,t\-1\}\+\(1\-\\alpha\)x\_\{h,t\}\.Herettis the control\-update step,x¯h,t\\bar\{x\}\_\{h,t\}is the EMA\-smoothed feature for headhh, andα\\alphais the feature EMA coefficient\. In the main 4096\-token protocol, control updates occur every five steps andα=0\.9\\alpha=0\.9\. Between control updates, the cached per\-head window indices are reused\. At the first control update, the feature EMA is initialized from the current observed head feature,x¯h,0=xh,0\\bar\{x\}\_\{h,0\}=x\_\{h,0\}\. Missing previous\-attention diagnostics\(eh,nh,ρh\)\(e\_\{h\},n\_\{h\},\\rho\_\{h\}\)are initialized to zero\.

### 5\.4Hierarchy construction

AAH\-v3 deliberately separateslevel\-0 head groupingfromupper\-level group hierarchy\. The hierarchy therefore does not use one global similarity rule across all levels\.

Level 0: cosine head grouping with fallback\.The bottom level clusters EMA\-smoothed head features using cosine\-based centroid\-threshold clustering\. If this clustering collapses to a single group, AAH\-v3 applies a deterministic forced\-bipartition fallback\. This fallback is not random and not PCA\-based: it selects the least\-similar anchor pair, then assigns the remaining heads according to their relative similarity to the two anchors\. The fallback is a robustness mechanism that ensures a nontrivial level\-0 partition; it is not the main contribution\.

LetGiG\_\{i\}denote the set of heads assigned to level\-0 groupii\. The level\-0 group feature is the mean of the EMA\-smoothed head features in that group:

gi\(0\)=1\|Gi\|​∑h∈Gix¯h\.g\_\{i\}^\{\(0\)\}=\\frac\{1\}\{\|G\_\{i\}\|\}\\sum\_\{h\\in G\_\{i\}\}\\bar\{x\}\_\{h\}\.For upper levels, letCi\(r\)C\_\{i\}^\{\(r\)\}denote the child items from levelr−1r\-1that are merged into itemiiat levelrr\. The upper\-level feature is

gi\(r\)=1\|Ci\(r\)\|​∑j∈Ci\(r\)gj\(r−1\)\.g\_\{i\}^\{\(r\)\}=\\frac\{1\}\{\|C\_\{i\}^\{\(r\)\}\|\}\\sum\_\{j\\in C\_\{i\}^\{\(r\)\}\}g\_\{j\}^\{\(r\-1\)\}\.The size fraction supplied to the enriched controller is

si\(r\)=\|Ci\(r\)\|∑j\|Cj\(r\)\|,s\_\{i\}^\{\(r\)\}=\\frac\{\|C\_\{i\}^\{\(r\)\}\|\}\{\\sum\_\{j\}\|C\_\{j\}^\{\(r\)\}\|\},withCi\(0\)C\_\{i\}^\{\(0\)\}interpreted asGiG\_\{i\}at level 0\. Upper\-level aggregation is intentionally child\-weighted: each child item contributes equally regardless of how many descendant heads it represents\. Accordingly,si\(r\)s\_\{i\}^\{\(r\)\}is an immediate\-child fraction at upper levels, not a descendant\-head fraction\.

Upper levels:cosine\_normdiffgroup hierarchy\.At hierarchy levelrr, whererrdenotes the level index andi,ji,jindex candidate groups or items at that level, letgi\(r\)g\_\{i\}^\{\(r\)\}be the aggregated base feature for itemii\. Upper levels are built using cosine similarity penalized by relative norm difference:

sim\(r\)⁡\(i,j\)=cos⁡\(gi\(r\),gj\(r\)\)−λ⋅\|∥gi\(r\)∥−∥gj\(r\)∥\|max⁡\(∥gi\(r\)∥,∥gj\(r\)∥,ϵ\)\.\\operatorname\{sim\}^\{\(r\)\}\(i,j\)=\\cos\(g\_\{i\}^\{\(r\)\},g\_\{j\}^\{\(r\)\}\)\-\\lambda\\cdot\\frac\{\\left\|\\lVert g\_\{i\}^\{\(r\)\}\\rVert\-\\lVert g\_\{j\}^\{\(r\)\}\\rVert\\right\|\}\{\\max\(\\lVert g\_\{i\}^\{\(r\)\}\\rVert,\\lVert g\_\{j\}^\{\(r\)\}\\rVert,\\epsilon\)\}\.Hereλ\\lambdaweights the norm\-difference penalty andϵ\\epsilonprevents division by zero; the final configurations useλ=16\.0\\lambda=16\.0andϵ=10−6\\epsilon=10^\{\-6\}\. Thus, level 0 uses cosine clustering with forced bipartition allowed, while upper levels use the modified norm\-aware similarity\. This distinction should be read as part of the method definition, not as an incidental implementation detail\.

For a hierarchy withRhierR\_\{\\mathrm\{hier\}\}levels andr∈\{0,…,Rhier−1\}r\\in\\\{0,\\dots,R\_\{\\mathrm\{hier\}\}\-1\\\}, the normalized level coordinateηr\\eta\_\{r\}is

ηr=rmax⁡\(1,Rhier−1\)\.\\eta\_\{r\}=\\frac\{r\}\{\\max\(1,R\_\{\\mathrm\{hier\}\}\-1\)\}\.This design also makes hierarchy depth an explicit experimental variable: shallow feature\-derived\[2\]\[2\]and deeper practical\[2,2,2,2\]\[2,2,2,2\]regimes can be compared without changing the Transformer block interface\. A hierarchy shape\[2\]\[2\]means one binary split from heads into two level\-0 groups\. A shape\[2,2,2,2\]\[2,2,2,2\]means repeated binary grouping across four hierarchy construction stages, producing progressively coarser parent groups until the top\-level controller decision\.

### 5\.5Enriched controller inputs

The controller operates on enriched group representations, not raw pooled group features\. For itemiiat hierarchy levelrr, the enriched controller inputui\(r\)u\_\{i\}^\{\(r\)\}is

ui\(r\)=\[gi\(r\),ηr,gi\(r\)−pi\(r\),gi\(r\)−g¯\(r\),si\(r\)\]\.u\_\{i\}^\{\(r\)\}=\[g\_\{i\}^\{\(r\)\},\\ \\eta\_\{r\},\\ g\_\{i\}^\{\(r\)\}\-p\_\{i\}^\{\(r\)\},\\ g\_\{i\}^\{\(r\)\}\-\\bar\{g\}^\{\(r\)\},\\ s\_\{i\}^\{\(r\)\}\]\.Heregi\(r\)g\_\{i\}^\{\(r\)\}is the base group feature defined above,ηr\\eta\_\{r\}identifies the normalized hierarchy level,pi\(r\)p\_\{i\}^\{\(r\)\}is the parent feature for itemii,g¯\(r\)\\bar\{g\}^\{\(r\)\}is the mean feature over items at levelrr, andsi\(r\)s\_\{i\}^\{\(r\)\}is the group\-size fraction\. At the top hierarchy level, where no parent exists, the implementation sets the parent\-difference featuregi\(r\)−pi\(r\)g\_\{i\}^\{\(r\)\}\-p\_\{i\}^\{\(r\)\}to the zero vector by usingpi\(r\)=gi\(r\)p\_\{i\}^\{\(r\)\}=g\_\{i\}^\{\(r\)\}, preserving the fixed input dimension\.

With the 9\-dimensional base feature, enriched mode has dimension9\+1\+9\+9\+1=299\+1\+9\+9\+1=29\. This enriched representation is the input to both the base controller logits and the joint sibling scorer\.

### 5\.6Joint sibling scoring

Joint sibling scoring is the final AAH\-v3 controller design and the main controller change relative to earlier independent\-scoring variants\. Under independent scoring, sibling groups are evaluated separately, as shown in Figure[2](https://arxiv.org/html/2608.19203#S5.F2)\(a\)\. When sibling features are very similar, a shared scorer can produce nearly identical rankings over candidate windows even when different assignments would be preferable\. AAH\-v3 addresses this by scoring the ordered sibling pair jointly, making the contrast between siblings part of the decision, as shown in Figure[2](https://arxiv.org/html/2608.19203#S5.F2)\(b\)\. In the figure and formulas below,aaandbbindex an ordered sibling pair at hierarchy levelrr,ua\(r\)u\_\{a\}^\{\(r\)\}andub\(r\)u\_\{b\}^\{\(r\)\}are their enriched inputs,ja,b\(r\)j\_\{a,b\}^\{\(r\)\}is the joint input built from those inputs and their directional differences,fbasef\_\{\\mathrm\{base\}\}is the independent scorer,fjointf\_\{\\mathrm\{joint\}\}is the joint scorer,za\(r\)z\_\{a\}^\{\(r\)\}andzb\(r\)z\_\{b\}^\{\(r\)\}are window\-logit vectors, andc^a\(r\)\\hat\{c\}\_\{a\}^\{\(r\)\}andc^b\(r\)\\hat\{c\}\_\{b\}^\{\(r\)\}are raw selected window indices before any parent constraint\.

uau\_\{a\}ub≈uau\_\{b\}\\approx u\_\{a\}fbasef\_\{\\mathrm\{base\}\}fbasef\_\{\\mathrm\{base\}\}zaz\_\{a\}nearly same rankingzbz\_\{b\}nearly same ranking\(a\) independent scoringja,b\(r\)j^\{\(r\)\}\_\{a,b\}\[ua\(r\),ub\(r\),ua\(r\)−ub\(r\),ub\(r\)−ua\(r\)\]\[u^\{\(r\)\}\_\{a\},u^\{\(r\)\}\_\{b\},u^\{\(r\)\}\_\{a\}\-u^\{\(r\)\}\_\{b\},u^\{\(r\)\}\_\{b\}\-u^\{\(r\)\}\_\{a\}\]fjointf\_\{\\mathrm\{joint\}\}\[za\(r\),zb\(r\)\]\[z^\{\(r\)\}\_\{a\},z^\{\(r\)\}\_\{b\}\]two output slotsc^a\(r\)\\hat\{c\}^\{\(r\)\}\_\{a\}smaller windowc^b\(r\)\\hat\{c\}^\{\(r\)\}\_\{b\}larger window\(b\) Joint sibling scoring

Figure 2:Independent sibling scoring versus joint sibling scoring\. Panel \(a\) shows sibling groups scored separately by the base scorer\. Panel \(b\) shows the ordered sibling pair scored jointly at hierarchy levelrrusing enriched inputsua\(r\)u\_\{a\}^\{\(r\)\}andub\(r\)u\_\{b\}^\{\(r\)\}, joint inputja,b\(r\)j\_\{a,b\}^\{\(r\)\}, logit vectorsza\(r\)z\_\{a\}^\{\(r\)\}andzb\(r\)z\_\{b\}^\{\(r\)\}, and raw window\-index choicesc^a\(r\)\\hat\{c\}\_\{a\}^\{\(r\)\}andc^b\(r\)\\hat\{c\}\_\{b\}^\{\(r\)\}\.For sibling pair\(a,b\)\(a,b\)at hierarchy levelrr, AAH\-v3 builds the order\-sensitive joint inputja,b\(r\)j^\{\(r\)\}\_\{a,b\}:

ja,b\(r\)=\[ua\(r\),ub\(r\),ua\(r\)−ub\(r\),ub\(r\)−ua\(r\)\]\.j^\{\(r\)\}\_\{a,b\}=\[u^\{\(r\)\}\_\{a\},\\ u^\{\(r\)\}\_\{b\},\\ u^\{\(r\)\}\_\{a\}\-u^\{\(r\)\}\_\{b\},\\ u^\{\(r\)\}\_\{b\}\-u^\{\(r\)\}\_\{a\}\]\.This representation is not symmetric under swappingaaandbb: the first output slot corresponds to the first sibling and the second output slot corresponds to the second sibling\. Because the scorer observes bothua\(r\)−ub\(r\)u^\{\(r\)\}\_\{a\}\-u^\{\(r\)\}\_\{b\}andub\(r\)−ua\(r\)u^\{\(r\)\}\_\{b\}\-u^\{\(r\)\}\_\{a\}, it can assign different logits and complementary windows to siblings with otherwise similar base features\. The released implementation uses the same joint\-scorer parameterization across hierarchy levels; level information enters through the enriched input features\.

The wide joint sibling scorerfjointf\_\{\\mathrm\{joint\}\}outputs two logit vectors at once:

\[za\(r\),zb\(r\)\]=reshape⁡\(fjoint​\(ja,b\(r\)\),2,K\),\[z^\{\(r\)\}\_\{a\},z^\{\(r\)\}\_\{b\}\]=\\operatorname\{reshape\}\(f\_\{\\mathrm\{joint\}\}\(j^\{\(r\)\}\_\{a,b\}\),2,K\),whereK=\|𝒲\|K=\|\\mathcal\{W\}\|is the number of candidate window options\. The main 4096\-token runs use𝒲=\[512,1024,2048,4096\]\\mathcal\{W\}=\[512,1024,2048,4096\]\. The sibling\-specific raw choices are then

c^a\(r\)=arg⁡maxk⁡za,k\(r\),c^b\(r\)=arg⁡maxk⁡zb,k\(r\)\.\\hat\{c\}\_\{a\}^\{\(r\)\}=\\arg\\max\_\{k\}z^\{\(r\)\}\_\{a,k\},\\qquad\\hat\{c\}\_\{b\}^\{\(r\)\}=\\arg\\max\_\{k\}z^\{\(r\)\}\_\{b,k\}\.Herec^a\(r\)\\hat\{c\}\_\{a\}^\{\(r\)\}andc^b\(r\)\\hat\{c\}\_\{b\}^\{\(r\)\}are raw selected indices before parent constraints, andk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}indexes candidate windows\. For paired siblings, these joint logits replace the corresponding independently computed logits rather than acting as an additive correction or residual bias\. Equivalently, the base controller may produce logits first, but the joint sibling scorer overwrites those logits for paired siblings before the argmax window decision\.

### 5\.7Window selection and parent index constraint

For non\-paired items, or after paired logits have been replaced, the same raw\-choice rule can be written generically for itemiiat levelrras

c^i\(r\)=arg⁡maxk⁡zi,k\(r\)\.\\hat\{c\}\_\{i\}^\{\(r\)\}=\\arg\\max\_\{k\}z\_\{i,k\}^\{\(r\)\}\.Herec^i\(r\)\\hat\{c\}\_\{i\}^\{\(r\)\}is the raw selected window index before parent constraint, andzi,k\(r\)z\_\{i,k\}^\{\(r\)\}is the logit for candidate windowkk\. This raw choice is distinct from the parent\-constrained selected indexci\(r\)c\_\{i\}^\{\(r\)\}\. The candidate window list𝒲\\mathcal\{W\}is ordered by increasing span; in the 4096\-token runs,𝒲=\[512,1024,2048,4096\]\\mathcal\{W\}=\[512,1024,2048,4096\]\. Thus, larger indices correspond to physically larger causal local windows\. At the top level of the hierarchy, the raw choice is used directly; lower levels apply a parent index constraint:

ci\(r\)=min⁡\(c^i\(r\),cparent⁡\(i\)\(r\+1\)\)\.c\_\{i\}^\{\(r\)\}=\\min\(\\hat\{c\}\_\{i\}^\{\(r\)\},\\ c\_\{\\operatorname\{parent\}\(i\)\}^\{\(r\+1\)\}\)\.Hereparent⁡\(i\)\\operatorname\{parent\}\(i\)denotes itemii’s parent at levelr\+1r\+1\. This operation is an index\-level top\-down constraint, not the later execution\-window bounding operation\. Because𝒲\\mathcal\{W\}is ordered by increasing span, the minimum operation has a direct physical meaning\. A child group cannot select a larger attention\-window span than its parent group\. It does not force the child to copy the parent; a child can still choose a smaller window than its parent\.

For the one\-level Shallow freeze\[2\]\[2\]topology, the parent\-index clamp is vacuous because the level\-0 groups are already top\-level items; nontrivial parent\-child clamping is exercised only in multi\-level hierarchies such as\[2,2,2,2\]\[2,2,2,2\]\.

The controller decision is intentionally discrete at execution time\. On each control update, the controller computes window logits, selects hard window indices witharg⁡max\\arg\\max, applies the top\-down parent clamp above, maps level\-0 group decisions back to heads, optionally applies the post\-warmup ramp if that stabilization path is enabled, and then applies the final per\-head resolution EMA before grouped local execution\. The main reported 4096\-token runs use hard selected windows during the measured forward path; between control updates, cached per\-head window indices are reused\. Thus the reported ACR reflects the executed hard window policy, not a soft expected\-window mixture\. The implementation treats these index operations as non\-differentiable execution\-policy decisions and does not rely on backpropagating gradients througharg⁡max\\arg\\max, the parentmin\\min, ramping, orround\\operatorname\{round\}in the final executed index\.

Controller optimization\.The reported hard\-policy runs use the standard next\-token language\-modeling loss as the only training objective; there is no straight\-through estimator, policy\-gradient term, soft expected\-window mixture, supervised routing target, or auxiliary controller loss\. Consequently, the loss updates the ordinary Transformer weights and any differentiable AAH adapter and projection weights on the executed attention path, while the discrete window assignment itself is driven by features and statistics\. The level\-0 grouping and upper topology have no gradient\-updated parameters: they are recomputed or reused from EMA\-smoothed head features according to the regime\. The scorer modulesfbasef\_\{\\mathrm\{base\}\}andfjointf\_\{\\mathrm\{joint\}\}are checkpointed AAH\-state components that produce logits from those features; in the hard execution path, the language\-modeling loss does not provide direct credit assignment to their selected indices becausearg⁡max\\arg\\max, the parentmin\\min, ramping, and finalround\\operatorname\{round\}are treated as stop\-gradient policy operations\. Thus the measured controller policy should be read primarily as a statistic\-driven, checkpointed execution policy rather than as a fully differentiable router\.

Controller\-state lifecycle\.In the reported hard\-policy runs, the scorer MLP weights are fixed seeded AAH\-state components; “learned” in the local config denotes the controller\-choice module type and should not be read as gradient\-trained routing\. Concretely,fbasef\_\{\\mathrm\{base\}\}andfjointf\_\{\\mathrm\{joint\}\}are initialized with the same run seed as the Transformer checkpoint, stored in the checkpointed AAH state, and loaded from the same seed\-0, 10000\-step checkpoint for the final rows; their selected hard indices are not directly optimized by the language\-modeling loss\. The local result export records this module undermodel\.aah\_v3\_controller\_choice\_mode=learned, but the measured policy should be interpreted as a fixed seeded scorer policy operating on feature statistics\. Reproducing the statistic\-driven execution policy requires the checkpointed scorer weights, topology cache, resolution\-EMA state, candidate\-window list, configuration, tensor/load names, and expected load paths\.

### 5\.8Head mapping and execution

After parent\-constrained level\-0 decisions are available, group decisions are mapped back to individual heads at control updatett:

ch,t=cgroup⁡\(h\)\(0\)\.c\_\{h,t\}=c\_\{\\operatorname\{group\}\(h\)\}^\{\(0\)\}\.Herech,tc\_\{h,t\}is the selected window index for headhh, andgroup⁡\(h\)\\operatorname\{group\}\(h\)maps headhhto its level\-0 group\. This is the explicit bridge between hierarchy\-level decisions and per\-head MHA execution\.

AAH\-v3 smooths final per\-head resolution indices before execution\. For headhhat control\-update steptt, the smoothed selected indexc~h,t\\tilde\{c\}\_\{h,t\}is

c~h,t=round⁡\(β​c~h,t−1\+\(1−β\)​ch,t\)\.\\tilde\{c\}\_\{h,t\}=\\operatorname\{round\}\\left\(\\beta\\tilde\{c\}\_\{h,t\-1\}\+\(1\-\\beta\)c\_\{h,t\}\\right\)\.This resolution EMA smooths the selected window index over time after hierarchy scoring and group\-to\-head mapping, withβ\\betadenoting the resolution EMA coefficient\. It is not topology reuse, and it is not hierarchy smoothing: it acts only on the final per\-head discrete resolution index\. The main 4096\-token protocol includes this smoothing withβ=0\.15\\beta=0\.15\. The resolution EMA index is initialized to the full\-window candidate index, so the initial fallback execution is ordinary full\-span causal attention\. When the update step is clear, we writec~h\\tilde\{c\}\_\{h\}for the current smoothed index\.

Post\-warmup ramping is a separate optional stabilization mechanism in the implementation\. It is inactive in the main reported regimes, so the active path is: hierarchy scoring, parent index constraint, group\-to\-head mapping, resolution EMA, execution\-window mapping, and local attention\.

The final execution windowWhW\_\{h\}for headhhis then mapped from the current smoothed selected indexc~h\\tilde\{c\}\_\{h\}and clamped to valid execution bounds:

Wh=clamp⁡\(𝒲​\[c~h\],Wmin,T\)\.W\_\{h\}=\\operatorname\{clamp\}\(\\mathcal\{W\}\[\\tilde\{c\}\_\{h\}\],\\ W\_\{\\min\},\\ T\)\.HereWminW\_\{\\min\}is the minimum allowed local\-window span andTTis the sequence length, or maximum causal span\. For the main 4096\-token setting,Wmin=512W\_\{\\min\}=512andT=4096T=4096\. Before window lookup, smoothed indices are clamped to the valid candidate range\. Mathematical notation uses one\-based indexing for𝒲\\mathcal\{W\}; the implementation uses zero\-based indexing\. This final execution\-window clamp is distinct from the parent index constraint above\.

The current execution path used in the reported 4096\-token experiments is a dense\-masked, grouped local implementation\. Heads with the same selected window spanWWare grouped and executed together under causal local masking\. For each selected window, the local causal attention maskMWM\_\{W\}allows only key positionstkt\_\{k\}satisfyingtk≤tqt\_\{k\}\\leq t\_\{q\}andtk≥tq−W\+1t\_\{k\}\\geq t\_\{q\}\-W\+1for query positiontqt\_\{q\}\. With bucketed projectionsQQ,KK, andVV, attention is then

A=softmax⁡\(Q​K⊤dh\+MW\),Y=A​V\.A=\\operatorname\{softmax\}\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\+M\_\{W\}\\right\),\\quad Y=AV\.Heredhd\_\{h\}is the head dimension,AAis the attention matrix, andYYis the bucket output\. A full\-window choiceW=TW=Treduces to standard causal attention\. This execution stage is intentionally separable from the policy that selectedWW: the same per\-head window indices are handed to backend\-realized local\-attention kernels in the separate backend\-realized suite\. The supported interpretation for the current dense\-masked experiments is therefore head\-wise context\-allocation behavior and validation\-quality comparison, not kernel\-level FLOPs reduction or universal wall\-clock speedup\.

### 5\.9What AAH\-v3 changes relative to baseline

Baseline MHA applies the same full causal attention pattern to all heads and then merges head outputs as usual:

O=Concat⁡\(O1,…,OH\)​WO\.O=\\operatorname\{Concat\}\(O\_\{1\},\\dots,O\_\{H\}\)W^\{O\}\.HereOhO\_\{h\}is the output of headhh,HHis the number of heads, andWOW^\{O\}is the output projection matrix\. AAH\-v3 keeps this output topology but changes the execution policy before the merge\. Its real changes are feature\-derived head grouping, enriched controller inputs, wide joint sibling scoring, top\-down constrained discrete window decisions, and grouped local causal execution in the current backend\. The wide joint sibling scorer is the key controller change: AAH\-v3 is not an independent per\-group scorer and not a mechanism operating inside a single attention head\.

### 5\.10Measuring Attention Coverage and Hardware FLOPs

Because the main question is structural context allocation, we separate selected\-window routing diagnostics, measured GPU FLOPs, and wall\-clock speed\.

We define theAttention Coverage Ratio \(ACR\)as the implementation’s rectangular query\-by\-window selected\-span proxy:

ACR=∑m=1Nlayer∑h=1HTq\(m,h\)​Tk\(m,h\)Nlayer​H​T2,\\operatorname\{ACR\}=\\frac\{\\sum\_\{m=1\}^\{N\_\{\\mathrm\{layer\}\}\}\\sum\_\{h=1\}^\{H\}T\_\{q\}^\{\(m,h\)\}T\_\{k\}^\{\(m,h\)\}\}\{N\_\{\\mathrm\{layer\}\}HT^\{2\}\},HereNlayerN\_\{\\mathrm\{layer\}\}is the number of Transformer layers,HHis the number of attention heads, andTTis the full context length\. The termsTq\(m,h\)T\_\{q\}^\{\(m,h\)\}andTk\(m,h\)T\_\{k\}^\{\(m,h\)\}are the effective query and key spans selected for headhhin layermm\.

In the decoder\-only setting used here,Tq\(m,h\)=TT\_\{q\}^\{\(m,h\)\}=T, so

ACR=∑m=1Nlayer∑h=1HT⋅Tk\(m,h\)Nlayer​H​T2\.\\operatorname\{ACR\}=\\frac\{\\sum\_\{m=1\}^\{N\_\{\\mathrm\{layer\}\}\}\\sum\_\{h=1\}^\{H\}T\\cdot T\_\{k\}^\{\(m,h\)\}\}\{N\_\{\\mathrm\{layer\}\}HT^\{2\}\}\.This metric corresponds to the loggedattn\_ratioon training rows andACRon final inference rows\. ACR follows the implementation’s rectangular query\-by\-window accounting: it measures selected coverage structure rather than exact lower\-triangular causal\-pair count\. For example, a local causal window of sizeWWover context lengthTThas exact allowed\-pair count∑t=1Tmin⁡\(t,W\)\\sum\_\{t=1\}^\{T\}\\min\(t,W\), whereas ACR intentionally logs the simpler rectangular span proxyT⋅WT\\cdot W\. It should therefore be interpreted as a routing diagnostic, not as backend\-executed work, kernel\-level FLOPs, exact causal\-pair count, or runtime\.

#### Dense\-masked backend status\.

The current dense\-masked path uses selected windows to define the attention policy but does not establish skipped backend query–key work\. Therefore the main training, inference, and downstream\-context rows do not include backend work\-accounting columns\. Backend artifacts may contain span diagnostics such asbackend\_realized\_ACR\_est; these are implementation logs rather than paper metrics and should not be read as hardware FLOPs measurements\.

#### Measured GPU FLOPs\.

True GPU FLOPs are measured only by profiler counters that report GPU floating\-point operations under matched input batch, sequence length, precision, checkpoint, hardware, and software backend\. In this draft, the relevant hardware evidence is Nsight\-derived GPU floating\-point\-operation accounting\. The current Nsight and FLOPs\-lab evidence does not support a GPU\-FLOPs\-reduction claim for AAH\.

For a methodrr, let𝒫​\(r,x\)\\mathcal\{P\}\(r,x\)denote a GPU profiling run on the same input batchxx, sequence length, precision, checkpoint, hardware, and software backend as the corresponding pure\-attention baseline\. LetΦ​\(k\)\\Phi\(k\)be the number of GPU floating\-point operations reported by the profiler for kernelkk\. For inference we profile one forward pass after warmup\.

The measured total GPU FLOPs for methodrrare

GPUFPtotal​\(r\)=mediani=1N​∑k∈𝒦total\(i\)​\(r\)Φ​\(k\),\\mathrm\{GPUFP\}\_\{\\mathrm\{total\}\}\(r\)=\\operatorname\{median\}\_\{i=1\}^\{N\}\\sum\_\{k\\in\\mathcal\{K\}^\{\(i\)\}\_\{\\mathrm\{total\}\}\(r\)\}\\Phi\(k\),where𝒦total\(i\)​\(r\)\\mathcal\{K\}^\{\(i\)\}\_\{\\mathrm\{total\}\}\(r\)is the set of GPU kernels executed inside the profiled forward\-pass region on profiling repeatii\. The measured total FLOPs Ratio is

Measured​Total​FLOPs​Ratio​\(r\)=GPUFPtotal​\(r\)GPUFPtotal​\(Pure\)\.\\mathrm\{Measured\\ Total\\ FLOPs\\ Ratio\}\(r\)=\\frac\{\\mathrm\{GPUFP\}\_\{\\mathrm\{total\}\}\(r\)\}\{\\mathrm\{GPUFP\}\_\{\\mathrm\{total\}\}\(\\mathrm\{Pure\}\)\}\.For attention\-only accounting, we additionally profile kernels inside explicit attention execution ranges:

GPUFPattn​\(r\)=mediani=1N​∑k∈𝒦attn\(i\)​\(r\)Φ​\(k\),\\mathrm\{GPUFP\}\_\{\\mathrm\{attn\}\}\(r\)=\\operatorname\{median\}\_\{i=1\}^\{N\}\\sum\_\{k\\in\\mathcal\{K\}^\{\(i\)\}\_\{\\mathrm\{attn\}\}\(r\)\}\\Phi\(k\),and define

Measured​Attention​FLOPs​Ratio​\(r\)=GPUFPattn​\(r\)GPUFPattn​\(Pure\)\.\\mathrm\{Measured\\ Attention\\ FLOPs\\ Ratio\}\(r\)=\\frac\{\\mathrm\{GPUFP\}\_\{\\mathrm\{attn\}\}\(r\)\}\{\\mathrm\{GPUFP\}\_\{\\mathrm\{attn\}\}\(\\mathrm\{Pure\}\)\}\.These profiler\-measured FLOPs/FLOPs quantities are not derived from ACR, selected window sizes, analytic attention\-element counts, backend span logs, or token throughput\.

#### Dense\-masked reporting in the current experiments\.

The 1B / 4096\-token tables report ACR, validation quality, throughput, and memory where available; they do not report measured FLOPs Ratios\. For the current dense\-masked implementation, selected shorter windows validate the AAH routing policy and expose head\-wise context structure, but they do not establish skipped backend work or physical GPU FLOPs savings\. Because matched Nsight GPU floating\-point\-operation totals were not collected for the dense\-masked training rows, the paper does not report a measured FLOPs Ratio column for those rows\.

Table[1](https://arxiv.org/html/2608.19203#S5.T1)summarizes the metric separation used throughout the paper\. ACR is the selected\-window routing metric, and FLOPs Ratios are reserved for profiler\-measured GPU floating\-point operations\.

Table 1:Relationship between ACR and measured FLOPs Ratios\. ACR is a selected\-window routing diagnostic\. Measured FLOPs Ratios require matched profiler counters and are not inferred from selected windows\.

## 6Systems Diagnostics and FLOPs Boundary

AAH separates the structural routing mechanism from the systems outcome\. The routing mechanism selects unequal head\-wise context windows\. The systems outcome depends on whether the runtime stack turns those windows into lower measured GPU floating\-point\-operation counts\.

Coverage diagnostic and backend status\.ACR is a selected\-window coverage proxy, defined in Subsection[5\.10](https://arxiv.org/html/2608.19203#S5.SS10)\. It is not a measured FLOPs metric\. In the current dense\-masked rows, the backend has not been shown to skip the corresponding dense attention work, so backend work accounting is omitted from the main tables\.

Measured FLOPs boundary\.True GPU FLOPs are measured only by Nsight\-derived or equivalent GPU floating\-point\-operation counters collected under matched hardware, software, precision, checkpoint, and input settings\. The latest systems evidence is negative for a FLOPs\-reduction claim: in the backend 4096 suite, FlashAttention AAH rows are about 1\.59x–1\.61x the pure FlashAttention measured FLOPs; later lower\-overhead FLOPs\-lab probes approach but do not beat pure FlashAttention or pure dense MHA; and dense\-framework probes also approach 1\.0 but remain above it\. These results mean the current implementations do not establish measured GPU\-FLOPs savings\.

Reported diagnostics\.To keep the claim interpretable, the main comparison reports method, grouping, hierarchy, joint\-scorer setting, context length, validation loss, ACR, tokens per second, and memory when the run records it\. ACR should be read as a selected\-window structure diagnostic\. Token/s and memory are reported as secondary implementation observations, not as evidence that AAH reduces hardware work\.

## 7Experiments

### 7\.1Experimental setup

The main 4096\-token suite uses five seed\-0 runs: Full attention, Grouping off, Full adaptive, Shallow freeze, and Deep practical reuse\. Unless otherwise stated, each run trains for 10000 optimizer steps with batch size 1, and the local experiment configuration holds the tokenizer, data path, optimizer schedule, batch construction, model width/depth, seed, and standard Transformer output interface fixed by experiment intent\. For the AAH rows, the candidate window set is𝒲=\[512,1024,2048,4096\]\\mathcal\{W\}=\[512,1024,2048,4096\], control updates occur every five steps, the feature EMA coefficient is 0\.9 \(see Subsection[5\.3](https://arxiv.org/html/2608.19203#S5.SS3)\), and the final resolution EMA coefficient is 0\.15\. Here,contextdenotes the per\-sample model sequence length; at context length 4096, the Full attention reference attends overT=4096T=4096\. Checkpoints are saved at optimizer steps 1000, 5000, and 10000, and evaluation runs every 1000 optimizer steps\. The W&B logged\-step axis in the training\-curve figures is a logging index: the final plotted validation point at logged step 250 corresponds to optimizer step 10000 under the 40\-optimizer\-step logging interval\. The main suite contains five training runs, one per principal regime, all under the same seed\-0 protocol by local experiment intent\. The subsections below proceed from training results to mechanism diagnostics, hierarchy and controller checks, inference evaluation, backend diagnostics, and then a final summary of claim boundaries\. All W&B\-derived training, inference, ACR, throughput, and peak allocated memory values are checked against the authoritativewandb\_results\_new/export; older bundled result summaries should be ignored if they conflict with that folder\. For Full attention rows, ACR is set to 1\.0000 by definition when AAH\-specific ACR fields are absent\. For AAH rows, table rows are selected from completed seed\-0 runs with final checkpoint step 10000 and non\-null validation, ACR or branch\-usage data, throughput, and peak allocated memory fields when available\. Table[2](https://arxiv.org/html/2608.19203#S7.T2)summarizes the main\-run protocol and the W&B export metadata used for row selection\. Appendix[A\.5](https://arxiv.org/html/2608.19203#A1.SS5)lists additional packaging metadata for public artifact release\.

The main 4096\-token dense\-masked suite evaluates AAH as a context\-allocation mechanism\. Section[7\.7](https://arxiv.org/html/2608.19203#S7.SS7)separates this routing result from backend and Nsight diagnostics, which show that the current implementations do not establish measured GPU\-FLOPs savings\.

Table 2:Main 1B / 4096\-token protocol fields and W&B row\-selection rule\. Additional artifact\-packaging metadata is listed in Appendix[A\.5](https://arxiv.org/html/2608.19203#A1.SS5)\.The evaluated regimes are defined as follows:

- •Full attention: standard full\-attention reference with ordinary causal MHA\.
- •Grouping off: disables feature\-derived grouping while retaining the surrounding execution\-control framework, used as a package\-level comparison regime without feature\-derived grouping or paired joint scoring rather than as a pure one\-factor ablation of grouping alone\. In this regime, each head is treated as its own level\-0 execution item, the base window scorer remains active, paired joint sibling scoring is disabled, and upper\-level parent constraints are bypassed because no learned or feature\-derived upper hierarchy is used\. The final resolution EMA is initialized at the full\-window candidate index and cached per\-head window state is still maintained between control updates, matching the other AAH execution\-control rows\.
- •Full adaptive: the fully adaptive AAH\-v3 regime, which refreshes feature\-derived grouping and hierarchy on control\-update steps while retaining joint sibling scoring and grouped local execution\.
- •Shallow freeze: uses a feature\-derived shallow\[2\]\[2\]topology that is then frozen, while group features, controller inputs, joint sibling decisions, raw or vacuously constrained choices, and final head windows continue to update\.
- •Deep practical reuse: reuses the cached level\-0 head partition associated with the deep\[2,2,2,2\]\[2,2,2,2\]regime while continuing to recompute upper hierarchy and controller decisions\.

Additional diagnostic regimes are reserved for appendix follow\-up at the same 4096\-token context with 10000 steps and one seed unless explicitly promoted into the main table:control\_off, fixed random grouping, freeze\-after\-warmup passthrough, independent scoring, no\-parent\-constraint, no\-feature\-EMA, and simple deterministic or random\-window policy variants\. These seven configurations define the intended diagnostic package, but the main text treats them as mechanism checks rather than headline evidence\.

Table 3:Configuration differences among the main 1B / 4096\-token regimes\. Performance metrics are reported separately in Tables[4](https://arxiv.org/html/2608.19203#S7.T4)and[6](https://arxiv.org/html/2608.19203#S7.T6)\.
### 7\.2Training experiments

Training is the primary quality\-constrained comparison because it uses the shared optimization protocol and records validation quality, attention coverage, post\-warmup throughput, and memory for each principal regime\. Table[4](https://arxiv.org/html/2608.19203#S7.T4)reports the final training\-loop row for the standard Full attention and the AAH\-v3 execution regimes\. The memory column is the exported peak allocated GPU memory, computed fromperf/gpu\_alloc\_max\_mb/1024, not the current\-allocation or peak\-reserved memory field\.

Provenance note\.The W&B\-derived export records two commit prefixes across the five main rows:4b660ccfor Full attention, Grouping off, and Full adaptive, andaa92c47for Shallow freeze and Deep practical reuse\. We therefore treat the table as a locally controlled comparison from the exported experiment set; the appendix lists the per\-row manifest fields that should accompany a public artifact package\.

Table 4:Main 1B / 4096\-token training comparison for the Full attention reference and principal AAH\-v3 execution regimes\. ACR is the selected attention\-coverage policy proxy\. The dense\-masked path does not establish skipped backend work, so backend work accounting is omitted\. Memory is peak allocated GPU memory fromperf/gpu\_alloc\_max\_mb/1024\. Bold indicates the best value in each metric column\.The training table separates three outcomes\. First, the AAH\-v3 regimes choose shorter selected windows than Full attention under the ACR routing proxy: the feature\-derived AAH\-v3 rows reach ACR values between 0\.2891 and 0\.3724, while Grouping off reaches 0\.5814\. Second, Shallow freeze gives the lowest final training validation loss in this single\-seed suite, 6\.5367 versus 6\.5672 for Full attention and 6\.5655 for Grouping off; this should be read as a seed\-0 outcome rather than a multi\-seed significance result\. Third, selected\-window changes do not translate into measured training throughput gains in this implementation: all AAH rows have lower token/s than Full attention and modestly higher exported peak allocated memory\. The dense\-masked backend does not establish skipped backend work or physical GPU FLOPs savings\. The training evidence therefore supports a quality/structure interpretation of head\-wise context allocation, not an end\-to\-end training speedup claim\.

### 7\.33000\-step quality/structure screening ablation

The latest Phase 1 quality/structure lab is a short\-budget screening ablation, not the final flagship paper table\. It nevertheless adds important controls for interpreting AAH\. Table[5](https://arxiv.org/html/2608.19203#S7.T5)reports 3000\-step validation loss for a pure baseline, shallow AAH variants, fixed/random controls, and shuffle\-post\-select controls\.

Table 5:Phase 1 3000\-step quality/structure screening ablation at 4096\-token context\. Lower validation loss is better\. These short\-budget rows test whether head\-window assignment structure and routing stability matter; they are not a replacement for matched 10000\-step paper\-grade confirmation\.The screening result supports a conservative structure claim\. AAH\-style local structured allocation improves short\-budget quality relative to the pure baseline, and shuffle\-post\-select controls suggest that stable head\-window assignment structure matters\. However, the strong fixed random grouping and fixed\-1024 controls mean the paper should not claim that fully adaptive hierarchy alone is proven causal\. Final paper\-grade confirmation would require a matched 10000\-step suite for pure baseline,shallow\_freeze,shallow\-control\-interval10,fixed\-1024, andfixed\-random\-grouping\.

Figure[3](https://arxiv.org/html/2608.19203#S7.F3)shows how the existing 10000\-step rows emerge over the logged training trajectory\. The validation\-loss panel checks whether the final quality differences are late\-training artifacts, while the ACR and hierarchy\-level panels show how aggressively each regime changes its selected context span during training\. The inference endpoints are reported separately in Table[6](https://arxiv.org/html/2608.19203#S7.T6); they are not mixed into this training figure\.

1751751801801851851901901951952002002052052102102152152202202252252302302352352402402452452502506\.526\.526\.566\.566\.66\.66\.646\.646\.686\.68logged stepval\. lossValidation loss \(late\-training zoom\)Full attentionGrouping offFull adaptiveShallow freezeDeep practical reuse

−20\-200202040406060808010010012012014014016016018018020020022022024024026026000\.50\.511logged stepattn\_ratioAttention coverage ratio

−20\-2002020404060608080100100120120140140160160180180200200220220240240260260011223344logged steplevels usedHierarchy levels used

Figure 3:Training dynamics for the 1B / 4096\-token seed\-0 suite\. The x\-axis is the W&B logged\-step index rather than raw optimizer step; logged step 250 corresponds to optimizer step 10000\. The top panel zooms the late\-training validation\-loss range on a linear y\-axis so the differences between runs remain visible; the lower panels track attention\-coverage ratio and hierarchy levels used\. Curves are drawn with smooth interpolation for readability\. Validation points use their logged W&B coordinates\. For the ACR and hierarchy panels, the final diagnostic rows exported at W&B step 249 are visually aligned to the 10000\-step endpoint at x=250 so the final diagnostic values correspond to the same selected final checkpoint as the validation curve\.
### 7\.4Aggregate window\-bucket heatmap

Placed immediately after the training curves, the window\-bucket diagnostic explains how the final compute proxies arise\. Figure[4](https://arxiv.org/html/2608.19203#S7.F4)summarizes the distribution of selected execution windows over\[512,1024,2048,4096\]\[512,1024,2048,4096\], aggregated across final layer–head assignments: each row is a regime and each cell is the fraction of final assignments in that window bucket\.

32\.7%8\.8%13\.6%44\.8%63\.5%18\.5%11\.4%6\.6%51\.3%17\.0%16\.8%15\.0%50\.3%20\.1%9\.4%20\.1%512102420484096Grouping offFull adaptiveShallow freezeDeep practical reuseselected windowregimeFinal selected\-window distribution0101020203030404050506060Figure 4:Aggregate selected\-window bucket heatmap for the 1B / 4096\-token seed\-0 regimes at the 10000\-step checkpoint\. The heatmap shows aggregate bucket fractions over the candidate execution windows, not a per\-layer/per\-head diagnostic heatmap; warmer and brighter cells indicate more final assignments in that window bucket\.The bucket fractions explain the final inference ACR ordering\. The training\-loop ACR ordering differs because training ACR is logged on the training path, while this heatmap summarizes final inference window\-bucket assignments\. Full adaptive assigns the largest mass to the 512\-token bucket and the smallest mass to the full 4096\-token bucket, consistent with its lowest final\-inference coverage proxy\. Shallow freeze keeps more mass in the 2048\- and 4096\-token buckets, which raises final\-inference ACR relative to Full adaptive but coincides with the best validation quality in this seed\-0 suite\. Deep practical reuse also favors the 512\-token bucket, but retains a larger full\-window fraction than Full adaptive\.

### 7\.5Hierarchy\-depth comparison and mechanism interpretation

After the training and window\-allocation diagnostics, the hierarchy\-depth comparison defines what can and cannot be inferred about AAH\-v3 components\. The main 10000\-step depth comparison is shallow\[2\]\[2\]frozen feature\-derived topology versus deep\[2,2,2,2\]\[2,2,2,2\]practical reuse\. The Phase 1 screening ablation in Subsection[7\.3](https://arxiv.org/html/2608.19203#S7.SS3)adds short\-budget controls for routing stability, fixed windows, random grouping, and shuffle\-post\-select assignment structure\.

These controls sharpen the causal interpretation\. The current suite evaluates AAH\-v3 as a complete context\-allocation package\. Phase 1 shows that local allocation and stable head\-window assignment structure matter, but it also cautions against claiming that fully adaptive hierarchy alone is proven causal\. A planned mechanism\-check manifest is kept in Appendix[A\.2](https://arxiv.org/html/2608.19203#A1.SS2)rather than used as main\-text evidence\.

### 7\.6Inference experiments

Inference experiments evaluate the final 10000\-step checkpoints under the same quality\-versus\-routing\-diagnostics framing, but with runtime observed on the inference path rather than the training loop\. Quality is measured by validation loss and validation perplexity\. ACR \(Subsection[5\.10](https://arxiv.org/html/2608.19203#S5.SS10)\) is the selected\-window routing diagnostic, while post\-warmup tokens per second is a secondary implementation observation\. The local result export does not contain matched profiler GPU FP\-operation totals or attention\-range FP\-operation totals, so measured FLOPs ratios are not reported in these rows\. A separate inference peak\-memory field is not recorded for these rows, so memory remains a training\-table diagnostic rather than an inference\-table column\.

Table[6](https://arxiv.org/html/2608.19203#S7.T6)mirrors the main training comparison at the final 10000\-step checkpoint\. The added perplexity column makes the inference quality constraint explicit, while the throughput column tests whether the current bucketed local\-attention implementation converts lower selected attention coverage into measured speed\.

Table 6:Main 1B / 4096\-token inference comparison for the Full attention reference and principal AAH\-v3 execution regimes\. Inference ACR is reconstructed from final inference branch\-usage frequencies as∑WfW​W/T\\sum\_\{W\}f\_\{W\}W/TwithT=4096T=4096\. The dense\-masked path does not establish skipped backend work; no measured FLOPs Ratio is reported without matched profiler GPU FP\-operation totals\. Bold indicates the best value in each metric column\.At inference time, the same distinction between routing diagnostics and systems outcome remains visible\. Shallow freeze has the best final validation quality, with loss 6\.5293 and perplexity 684\.93, while Full adaptive has the lowest inference ACR at 0\.2484\. Compared with Grouping off, the feature\-derived AAH\-v3 package improves validation quality while selecting shorter context windows in this seed\-0 suite, but the dense\-masked path does not establish skipped backend work and all AAH rows are slower than Full attention in measured token/s\. Thus the inference results reinforce the context\-allocation interpretation while making the current runtime overhead explicit\.

### 7\.7Backend\-realized local attention and FLOPs diagnostics

Dense\-masked AAH validates the window\-selection policy, but it does not prove backend\-level or hardware\-level work reduction\. We therefore use the backend and FLOPs\-lab summaries as systems diagnostics for the current implementation boundary\. FlexAttention and FlashAttention can consume selected local windows, but measured GPU FLOPs must be evaluated with profiler counters rather than inferred from ACR or backend span logs\.

Table[7](https://arxiv.org/html/2608.19203#S7.T7)summarizes the current systems evidence\. The backend\-realized attention suite is useful for checking whether selected windows can be represented as backend spans and for recording backend span diagnostics\. However, the Nsight\-derived and FLOPs\-lab results do not support a GPU\-FLOPs\-reduction claim for the current implementations\.

Table 7:Systems diagnostics for the measured\-GPU\-FLOPs claim boundary\. ACR is a routing diagnostic; measured FLOPs claims require Nsight\-derived GPU floating\-point\-operation counters\.For a FlexAttention path, the implementation uses cached BlockMasks for each local causal window bucket\. The local causal predicate for windowWWis

kidx≤qidxandkidx≥qidx−\(W−1\),k\_\{\\mathrm\{idx\}\}\\leq q\_\{\\mathrm\{idx\}\}\\quad\\text\{and\}\\quad k\_\{\\mathrm\{idx\}\}\\geq q\_\{\\mathrm\{idx\}\}\-\(W\-1\),so the backend can represent disallowed query/key\-value blocks rather than merely using the same dense mask\. For a FlashAttention path, each selected\-window bucket uses sliding\-window causal attention where the installed backend exposes that mode, with local window settingwindow\_size=\(W−1,0\)\\texttt\{window\\\_size\}=\(W\-1,0\)\. If a backend cannot execute a requested bucket natively, that bucket should be reported as a fallback rather than counted as backend\-realized local attention\.

The claim boundary is therefore compact\. The 4096\-token dense\-masked suite supports AAH policy behavior, hierarchy/window selection, validation\-quality comparisons, and ACR routing diagnostics\. The Qwen3 compatibility check supports only pretrained compatibility under the AAH patch\. The backend and FLOPs\-lab diagnostics show that the current implementations do not translate selected\-window reductions into lower measured GPU FLOPs\. AAH should therefore be read as a head\-wise context\-allocation and structure mechanism, not as a demonstrated GPU\-FLOPs\-reduction method\.

### 7\.8Main results

Taken together, the experiments support a bounded head\-wise context\-allocation result\. Compared with the Grouping off package\-level comparison, the feature\-derived AAH\-v3 package has lower validation loss while selecting shorter context windows under the ACR routing proxy in this seed\-0 suite\. The dense\-masked path does not establish skipped backend work, so these tables do not claim physical GPU FLOPs reduction\. Shallow freeze gives the lowest validation loss in the 10000\-step run, while Full adaptive gives the shortest final\-inference selected\-window proxy\. Deep practical reuse gives the lowest training\-loop ACR, but is not dominant at final inference\. The aggregate window\-bucket heatmap makes this tradeoff visible by showing how much probability mass each regime places on short versus long execution windows\. The present evidence evaluates the complete AAH\-v3 execution package, not the isolated causal contribution of the fixed wide joint sibling scorer\.

The Phase 1 screening ablation strengthens the structural interpretation while weakening any single\-mechanism causal claim\. Shallow control interval 10 is the best 3000\-step row, but fixed random grouping and fixed 1024 are also strong\. Shuffle\-post\-select controls are worse, especially for Full adaptive, suggesting that stable head\-window assignments matter\. The supported claim is therefore that structured/local head\-wise allocation can improve quality in these seed\-0 4096\-token experiments, not that fully adaptive hierarchy alone is proven causal or that the current implementation lowers measured GPU FLOPs\.

## 8Discussion and Limitations

AAH\-v3 demonstrates that MHA\-level, cross\-head context allocation can be implemented without changing the Transformer block interface\. The most important methodological change from earlier drafts is the final controller design: enriched controller inputs and wide joint sibling scoring replace the old independent\-group\-scoring narrative\.

The evidence supports a conservative structural conclusion\. The main 4096\-token, seed\-0 suite contrasts the feature\-derived AAH\-v3 package with Grouping off, then compares three feature\-derived topology regimes: Full adaptive, Shallow freeze with a feature\-derived\[2\]\[2\]topology, and Deep practical reuse with\[2,2,2,2\]\[2,2,2,2\]reuse\. The Phase 1 screening ablation adds short\-budget controls showing that shallow routing stability, fixed random grouping, fixed local windows, and shuffle\-post\-select assignment structure materially affect validation loss\. These controls support the importance of head\-wise context allocation but caution against overclaiming that fully adaptive hierarchy alone is the causal driver\.

The practical regimes are not all fully adaptive\. Deep practical reuse reuses the level\-0 head partition after it is cached, while continuing to recompute group features, upper hierarchy, parent maps, enriched inputs, joint scorer decisions, and final windows\. Shallow freeze freezes the feature\-derived shallow topology, not the entire decision process\. These distinctions are central to interpreting the experiments\.

The single\-seed protocol is intentionally narrow\. It is sufficient for the current runnable comparison suite in Table[4](https://arxiv.org/html/2608.19203#S7.T4)and the Phase 1 screening table, but broader claims require additional seeds, longer\-context stress tests, stability diagnostics, different hierarchy or controller choices, full benchmark evaluations, and matched longer\-budget controls\. Final paper\-grade confirmation of the Phase 1 interpretation would require a matched 10000\-step suite for pure baseline,shallow\_freeze,shallow\-control\-interval10,fixed\-1024, andfixed\-random\-grouping\.

The appendix\-only Qwen3 smoke test is intentionally capped: it uses deterministic fixed subsets to check pretrained compatibility, not to report official full benchmark scores\. Finally, ACR remains a structure diagnostic\. Measured Total FLOPs Ratio and Measured Attention FLOPs Ratio require profiler\-collected GPU floating\-point\-operation totals and attention\-range totals; they are not derived from ACR or analytic window\-span logs\. The current Nsight and FLOPs\-lab evidence does not support a measured GPU\-FLOPs\-reduction claim\.

What is not claimed\.AAH\-v3 is a single\-seed, 4096\-token head\-wise context\-allocation study\. The evidence supports the complete AAH\-v3 execution package and several short\-budget structured/local controls improving validation quality relative to the pure baseline\. It does not yet isolate the wide joint scorer or adaptive hierarchy as the causal driver, prove multi\-seed significance, provide end\-to\-end speedup, report measured GPU FLOPs reduction, or report official full\-benchmark improvements\.

## 9Conclusion

We presented AAH\-v3, an implemented Asymmetric Attention Heads method for structured head\-wise context allocation in multi\-head attention\. The final design uses EMA\-smoothed head features, level\-specific hierarchy construction, enriched controller inputs, wide joint sibling replacement scoring, top\-down parent constraints, group\-to\-head window mapping, resolution EMA smoothing, and grouped causal local attention\. The revised runnable protocol evaluates Full attention, Full adaptive, Deep practical reuse, Shallow freeze, and Grouping off at the main 1B / 4096 / 10000\-step / seed\-0 budget, while the Phase 1 screening ablation adds short\-budget fixed/random/shuffle controls\. Across these experiments, the supported conclusion is that structured/local head\-wise allocation can improve validation quality in this seed\-0 setting and exposes interpretable context\-allocation structure; the evidence does not prove adaptive hierarchy alone\. ACR is therefore reported only as a selected\-window routing diagnostic, while hardware claims are reserved for profiler\-measured FLOPs\. Multi\-seed confirmation, matched 10000\-step controls, and longer\-context stress testing remain future work\.

## References

- \[1\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Lukasz Kaiser, and Illia Polosukhin\. Attention Is All You Need\.*Advances in Neural Information Processing Systems*, 2017\. arXiv:1706\.03762\.
- \[2\]Paul Michel, Omer Levy, and Graham Neubig\. Are Sixteen Heads Really Better than One?*Advances in Neural Information Processing Systems*, 2019\. arXiv:1905\.10650\.
- \[3\]Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov\. Analyzing Multi\-Head Self\-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned\.*Proceedings of ACL*, 2019\. arXiv:1905\.09418\.
- \[4\]Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever\. Generating Long Sequences with Sparse Transformers\. arXiv preprint arXiv:1904\.10509, 2019\.
- \[5\]Iz Beltagy, Matthew E\. Peters, and Arman Cohan\. Longformer: The Long\-Document Transformer\. arXiv preprint arXiv:2004\.05150, 2020\.
- \[6\]Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed\. Big Bird: Transformers for Longer Sequences\.*Advances in Neural Information Processing Systems*, 2020\. arXiv:2007\.14062\.
- \[7\]Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier\. Efficient Content\-Based Sparse Attention with Routing Transformers\.*Transactions of the Association for Computational Linguistics*, 2021\. arXiv:2003\.05997\.
- \[8\]Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin\. Adaptive Attention Span in Transformers\.*Proceedings of ACL*, 2019\. arXiv:1905\.07799\.
- \[9\]Noam Shazeer\. Fast Transformer Decoding: One Write\-Head is All You Need\. arXiv preprint arXiv:1911\.02150, 2019\.
- \[10\]Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\. GQA: Training Generalized Multi\-Query Transformer Models from Multi\-Head Checkpoints\.*Proceedings of EMNLP*, 2023\. arXiv:2305\.13245\.
- \[11\]Alex Graves\. Adaptive Computation Time for Recurrent Neural Networks\. arXiv preprint arXiv:1603\.08983, 2016\.
- \[12\]William Fedus, Barret Zoph, and Noam Shazeer\. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\.*Journal of Machine Learning Research*, 2022\. arXiv:2101\.03961\.
- \[13\]Tri Dao, Daniel Y\. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré\. FlashAttention: Fast and Memory\-Efficient Exact Attention with IO\-Awareness\.*Advances in Neural Information Processing Systems*, 2022\. arXiv:2205\.14135\.
- \[14\]Tri Dao\. FlashAttention\-2: Faster Attention with Better Parallelism and Work Partitioning\. arXiv preprint arXiv:2307\.08691, 2023\.
- \[15\]PyTorch Team\. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention\. PyTorch Blog, 2024\. URL:[https://pytorch\.org/blog/flexattention/](https://pytorch.org/blog/flexattention/)\. Accessed 2026\-05\-30\.
- \[16\]Qwen Team\. Qwen3 Technical Report\. arXiv preprint arXiv:2505\.09388, 2025\.
- \[17\]Qwen Team\. Qwen3\-4B\-Base Model Card\. Hugging Face, 2025\. URL:[https://huggingface\.co/Qwen/Qwen3\-4B\-Base](https://huggingface.co/Qwen/Qwen3-4B-Base)\. Accessed 2026\-05\-21; license rechecked 2026\-05\-23\.
- \[18\]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\. Measuring Massive Multitask Language Understanding\.*International Conference on Learning Representations*, 2021\. arXiv:2009\.03300\.
- \[19\]Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen\. MMLU\-Pro: A More Robust and Challenging Multi\-Task Language Understanding Benchmark\. arXiv preprint arXiv:2406\.01574, 2024\.
- \[20\]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\. GPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\. arXiv preprint arXiv:2311\.12022, 2023\.
- \[21\]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\. arXiv preprint arXiv:1803\.05457, 2018\.
- \[22\]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi\. HellaSwag: Can a Machine Really Finish Your Sentence?*Proceedings of ACL*, 2019\. arXiv:1905\.07830\.
- \[23\]Mandar Joshi, Eunsol Choi, Daniel S\. Weld, and Luke Zettlemoyer\. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\.*Proceedings of ACL*, 2017\. arXiv:1705\.03551\.
- \[24\]Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He\. C\-Eval: A Multi\-Level Multi\-Discipline Chinese Evaluation Suite for Foundation Models\.*Advances in Neural Information Processing Systems*, 2023\. arXiv:2305\.08322\.
- \[25\]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\. Training Verifiers to Solve Math Word Problems\. arXiv preprint arXiv:2110\.14168, 2021\.
- \[26\]Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert\-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N\. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba\. Evaluating Large Language Models Trained on Code\. arXiv preprint arXiv:2107\.03374, 2021\.
- \[27\]Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton\. Program Synthesis with Large Language Models\. arXiv preprint arXiv:2108\.07732, 2021\.

## Appendix AAppendix

This appendix records historical method variants, planned mechanism diagnostics, the appendix\-only Qwen3 compatibility smoke test, and artifact\-packaging notes\. It supports reproducibility and claim\-boundary checking; it does not introduce additional headline evidence beyond the main seed\-0 suite\.

### A\.1Earlier Internal Variants

AAH\-v1: static asymmetric heads\.This variant tested fixed asymmetric head roles such as static local\-window or reduced\-resolution assignments\. It is retained only as historical context because static asymmetry did not reliably preserve quality\.

AAH\-v2: preliminary dynamic control\.This variant introduced early dynamic resolution control, but stability and implementation issues prevented robust main\-text use\.

AAH\-v3: final method in this paper\.AAH\-v3 is the sole main method\. Planned mechanism checks are listed below as appendix/release\-bundle definitions and should not be read as completed main\-text evidence\.

### A\.2Planned Mechanism Diagnostics

The planned diagnostic package keeps the 1B / 4096 / 10000\-step / seed\-0 protocol fixed and changes one mechanism per run\. These rows define mechanism checks for future or release\-bundle runs; they are not completed main\-text evidence\.

Table 8:Planned appendix diagnostic runs for the fixed 1B / 4096 / 10000\-step / seed\-0 protocol\. These rows define intended mechanism checks, not completed main\-text evidence\.
### A\.3Qwen3 Compatibility Smoke Test

This appendix reports an internal pretrained compatibility smoke test rather than a new capability claim\. It asks whether the AAH execution policy can be inserted into the pretrained Qwen3\-4B\-Base Hugging Face artifact\[[16](https://arxiv.org/html/2608.19203#bib.bib16),[17](https://arxiv.org/html/2608.19203#bib.bib17)\]without large downstream degradation\. The compatibility pass is a separate transfer setting from the main 1B seed\-0 training suite\.

All scores are percentages from capped deterministic subsets for compatibility checking, not official full benchmark scores\. The evaluation uses Qwen3\-4B\-Base with the same prompts, tokenizer, context length 4096, bf16 inference, decoding settings, sample ordering, and fixed evaluation subsets across all methods\. AAH variants use the pretrained Qwen3 backbone with the corresponding adapted AAH controller/topology parameters loaded, while the Full attention row uses the original Qwen3 attention path\. The exact Qwen transfer/load mapping for these adapted controller, topology, adapter, or projection states should be included in the public artifact package\. The benchmark set covers MMLU\[[18](https://arxiv.org/html/2608.19203#bib.bib18)\], MMLU\-Pro\[[19](https://arxiv.org/html/2608.19203#bib.bib19)\], GPQA\-Diamond\[[20](https://arxiv.org/html/2608.19203#bib.bib20)\], ARC\-Challenge\[[21](https://arxiv.org/html/2608.19203#bib.bib21)\], HellaSwag\[[22](https://arxiv.org/html/2608.19203#bib.bib22)\], TriviaQA\[[23](https://arxiv.org/html/2608.19203#bib.bib23)\], C\-Eval\[[24](https://arxiv.org/html/2608.19203#bib.bib24)\], GSM8K\[[25](https://arxiv.org/html/2608.19203#bib.bib25)\], HumanEval\[[26](https://arxiv.org/html/2608.19203#bib.bib26)\], and MBPP\[[27](https://arxiv.org/html/2608.19203#bib.bib27)\]\. The exact percentages in this table are included only to check whether the AAH patch preserves the pretrained Qwen3 execution interface under a fixed local protocol\. They should not be cited as standalone benchmark results or used for external model comparisons\. The provenance notes below list the additional evaluated\-artifact, subset, prompt, scoring, execution\-harness, transfer\-load, and adapter/controller/topology fields needed for an independently packaged reproduction\.

Table 9:Appendix\-only internal pretrained compatibility smoke test on capped deterministic Qwen3\-4B\-Base subsets at 4096\-token context\. These are not official full benchmark scores and should not be cited as benchmark results or used for external model comparisons\. Best score for each benchmark is bolded, with ties bolded\.Subset sizes: MMLU 512, MMLU\-Pro 256, GPQA\-Diamond 198, ARC\-Challenge 512, HellaSwag 512, TriviaQA 512, C\-Eval 512, GSM8K 128, HumanEval 32, and MBPP 32\.

Table 10:Separate 1B\-suite routing\-diagnostic context for the AAH regimes used in the Qwen3 compatibility check\. ACR is taken from the 1B final\-checkpoint inference comparison in Table[6](https://arxiv.org/html/2608.19203#S7.T6); it is not a Qwen3 downstream\-run hardware or FLOPs measurement\. Max absolute delta is the largest benchmark\-score change, in percentage points, relative to the Full attention row in Table[9](https://arxiv.org/html/2608.19203#A1.T9)\.As shown in Table[9](https://arxiv.org/html/2608.19203#A1.T9), downstream scores remain close across the Full attention and AAH execution regimes in this internal smoke test\. Table[10](https://arxiv.org/html/2608.19203#A1.T10)pairs those small score changes with the separate 1B\-suite ACR context\. The largest absolute score change is 1\.5 percentage points on these capped subsets, and most method–benchmark differences are smaller or tied\. This pattern supports the conservative interpretation that AAH mainly changes the attention\-window assignment structure, while broad downstream task quality remains comparable under the fixed Qwen3\-4B\-Base compatibility protocol\. The table remains a capped\-subset compatibility check, not official full benchmark reporting and not evidence for external model comparisons\.

### A\.4Reproducibility Notes

The main protocol uses context length 4096, candidate windows𝒲=\[512,1024,2048,4096\]\\mathcal\{W\}=\[512,1024,2048,4096\], seed 0, batch size 1, 10000 optimizer steps, control updates every five steps, feature EMA coefficient 0\.9, and final resolution EMA coefficient 0\.15\. The paper\-facing regimes are Full attention, Grouping off, Full adaptive, Shallow freeze, and Deep practical reuse; the authoritative W&B\-derived result export is kept underwandb\_results\_new/\. The W&B\-derived export records two git commits across the five main rows: Full attention, Grouping off, and Full adaptive use commit4b660cc7f3cc629deadce30b9d93382b2e5a0f7f, while Shallow freeze and Deep practical reuse use commitaa92c473024dabd679a49bc51c2c0b3433abb441\. The paper treats these rows as the intended controlled comparison exported fromwandb\_results\_new/; a public artifact package should add a per\-row manifest with source run, checkpoint basename, checkpoint step, config basename/hash, git commit, and artifact hash, or rerun the suite under one commit\. The Qwen3\-4B\-Base downstream check uses deterministic capped subsets with the sample counts listed in Table[9](https://arxiv.org/html/2608.19203#A1.T9); these subsets are for compatibility checking rather than official full benchmark reporting\. The local provenance checklist indownstream\_provenance\_manifest\.mdrecords the model revision/license, evaluated\-artifact hash, subset item IDs or generation script, prompt templates, answer extraction, code\-task execution environment, and AAH adapter/checkpoint hashes to complete for release packaging\.

### A\.5Release Provenance Notes

For public artifact packaging, the release bundle should include a compact manifest covering the dataset source and split; tokenizer and vocabulary; layer, model, head, and FFN dimensions; optimizer and learning\-rate schedule; precision; and GPU/runtime provenance\. The current W&B\-derived export records seed, max steps, candidate windows, control/EMA settings, config paths and hashes, git commits, checkpoint step, andbf16precision; remaining checklist fields should be filled with exact values or immutable artifact identifiers before external release\.

For the Qwen3 compatibility check, the base model isQwen/Qwen3\-4B\-Baseat Hugging Face revision906bfd4b4dc7f14ee4320094d8b41684abff8539; the model\-card URL was accessed on 2026\-05\-21, and the model\-card licenseapache\-2\.0was rechecked on 2026\-05\-23\. AAH variants patch the pretrained Qwen3 attention modules, load only\.aah\_state\.\*adapter/controller/topology keys, and leave the full\-attention baseline on the original Qwen3 attention path\. The public manifest records the displayed subset counts and scoring protocol\. A public package should also include exact adapter files or artifact URLs, adapter/controller/topology hashes, dataset revisions, subset item IDs, server package versions, and code\-task execution\-environment details for independent reproduction\.

The Qwen transfer/load map is part of the artifact package rather than a headline result\. The release bundle should identify each transferred or loaded AAH component with its source artifact, source tensor or state\-dict key, target Qwen module path, conversion command or script, SHA\-256 or immutable artifact ID, expected load path, candidate\-window list, topology/cache state, controller/scorer state, adapter or projection state, and resolution\-EMA state\. Until that map and the subset/prompt/scoring provenance are complete, the downstream table should be read only as an internal compatibility smoke test, not as official benchmark evidence\.

For release, the artifact table should include file name, role, source run, checkpoint step, SHA\-256, and expected load path for each Transformer checkpoint, AAH scorer/controller state, topology/cache object, resolution EMA state, and Qwen adapter/projection state\. These hashes are method\-critical because the fixed seeded scorer policy and saved AAH state determine the executed window policy\.