TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

# Tile-Centric Mixed-Precision Attentionfor LLM Inference Acceleration
Source: [https://arxiv.org/html/2608.17336](https://arxiv.org/html/2608.17336)
## TileMix: Tile\-Centric Mixed\-Precision Attention for LLM Inference Acceleration

Qiao ZhangQinglei CaoHeng FanAffiliation:LLaVi Lab, Computer Science & Engineering, University of North TexasYan HuangAffiliation:LLaVi Lab, Computer Science & Engineering, University of North TexasKewei ShaAffiliation:Computer Science, Saint Louis University; Data Science, University of North Texas\{hanzhi\.zhang,heng\.fan,yan\.huang,kewei\.sha,yunhe\.feng\}@unt\.edu\{qiao\.zhang,qinglei\.cao\}@slu\.eduYunhe FengAffiliation:LLaVi Lab, Computer Science & Engineering, University of North Texas

###### Abstract

Long\-context prefill in large language models \(LLMs\) incurs substantial computation and memory traffic because dense self\-attention computes quadratic query\-key scores\. Existing methods either use a uniform low\-precision path or select token interactions, leaving spatial precision routing over hardware\-aligned score tiles outside fused dense attention\. We introduceTileMix, a tile\-centric precision\-routing kernel that makes numerical precision an executable spatial decision over score\-tile groups within fused dense attention\.TileMixpartitions the attention matrix into hardware\-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online\-softmax state\. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware\-aligned compute tiles and compact metadata at long contexts\. By routing all legal tile groups,TileMixpreserves dense token connectivity, requires no training, and supports grouped\-query attention, variable\-length batches, and INT8 key/value caches\. Across LongEval, LV\-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna,TileMixrecovers long\-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy\-efficiency frontier across model families\. The implementation is available at[https://github\.com/HanzhiZhang\-Ulrica/TileMix](https://github.com/HanzhiZhang-Ulrica/TileMix)\.

## 1Introduction

Transformer models increasingly rely on long contexts for document summarization[7](https://arxiv.org/html/2608.17336#bib.bib34), multi\-page question answering[30](https://arxiv.org/html/2608.17336#bib.bib35), and retrieval\-augmented generation[20](https://arxiv.org/html/2608.17336#bib.bib36), making efficient long\-sequence processing central to practical LLM inference\. During prefill, dense self\-attention computes interactions between all query and key tokens, producingO⁡\(L2\)O\(L^\{2\}\)score computation for sequence lengthLL\. This quadratic computation makes attention a primary execution bottleneck for long documents and other context\-intensive workloads\.

Figure 1:Comparison of attention efficiency strategies\. Colors denote execution states: blue for high precision, gray for low precision, and white for removed interactions\.*Quantization*reduces weight/activation memory but often keeps attention softmax and accumulation in higher precision\.*Sparse/block attention*executes a selected subset of token interactions through structured or dynamic patterns\.*TileMix*preserves all legal token interactions and routes score\-tile groups through FP16 or INT8 paths inside one fused attention kernel\.Existing acceleration methods mainly optimize numerical format, token connectivity, or IO scheduling, represented in Figure[1](https://arxiv.org/html/2608.17336#S1.F1)by quantization, sparsity, and IO\-aware fused attention\. \(1\)Low\-precision quantizationsuch as INT8[33](https://arxiv.org/html/2608.17336#bib.bib5);[38](https://arxiv.org/html/2608.17336#bib.bib26)improves arithmetic and memory efficiency across model operators including weight, activation, and attention\. Quantized attention kernels commonly use one arithmetic path per invocation or stage, leaving spatial precision routing over theL×LL\\times Lscore tiles outside the streaming loop\. \(2\)Sparsity\-based methods[35](https://arxiv.org/html/2608.17336#bib.bib37);[37](https://arxiv.org/html/2608.17336#bib.bib15);[3](https://arxiv.org/html/2608.17336#bib.bib13)reduce computation by selecting active token interactions\. \(3\)IO\-aware fused attention[11](https://arxiv.org/html/2608.17336#bib.bib9)partitions attention into hardware\-aligned tiles and fuses score computation, online softmax, and value aggregation\. These kernels use tiles for data movement and work partitioning, while retaining a uniform score\-computation path\. Together, these directions suggest using hardware\-aligned score tiles as the spatial unit of precision\. A fused kernel can preserve the complete attention graph while routing score\-tile groups through multiple arithmetic paths in one streaming computation\.

Modern fused\-attention kernels achieve high utilization through regular Tensor Core tiling and coordinated work partitioning, online softmax, and data movement across the GPU memory hierarchy[12](https://arxiv.org/html/2608.17336#bib.bib10);[25](https://arxiv.org/html/2608.17336#bib.bib11)\. Tile\-group precision routing must reconcile distinct FP16 and INT8 Tensor Core paths, including INT8 rescaling, before both paths update the shared row\-wise maximum, normalizer, and output accumulator[6](https://arxiv.org/html/2608.17336#bib.bib40);[4](https://arxiv.org/html/2608.17336#bib.bib12)\. Precision dispatch therefore falls inside the latency\-critical inner loop, making compact tile\-aligned routing essential for regular long\-context execution and creating a kernel\-design problem beyond invocation\-level precision selection\.

To address these challenges, we introduceTileMix, a tile\-centric precision\-routing kernel that integrates heterogeneous score arithmetic into a single FlashAttention\-style execution\. For each query\-tile row,TileMixloads a packed routing word, decodes each key\-tile\-group decision with constant\-time bit operations, and dispatches QK computation to FP16 or INT8 Tensor Core paths\. After rescaling, both paths update the shared online\-softmax state, preserving dense streaming execution\. Scalable precision grouping lets each routing bit govern adjacent key tiles while retaining the underlying hardware\-aligned compute tiles and compact metadata as context length grows\. The resulting kernel combines the dense connectivity of full attention with the arithmetic flexibility of mixed precision under training\-free deployment\. It also supports grouped\-query attention, variable\-length batching, and INT8 key/value caches, and exposes a controllable accuracy\-efficiency frontier between FP16 and uniform INT8 attention\.

Our contributions are as follows:

1. 1\.Tile\-Centric Precision Routing for Dense Attention:We introduce tile\-group precision as a spatial execution abstraction for fused dense attention, enabling fine\-grained FP16/INT8 allocation across all legal token interactions\.
2. 2\.Shared\-State Heterogeneous Score Execution:We design a fused kernel that aligns FP16 and INT8 score paths to a common score domain and integrates them through one online\-softmax recurrence\.
3. 3\.Compact and Scalable Kernel\-Native Routing:We develop packed bitmask routing with constant\-time inner\-loop lookup and𝒪⁡\(Hk​Tm\)\\mathcal\{O\}\(H\_\{k\}T\_\{m\}\)metadata, together with precision grouping that scales routing to long contexts while retaining hardware\-aligned compute tiles\.
4. 4\.Practical Long\-Context Inference and Evaluation:We implement grouped\-query attention, variable\-length batching, and INT8 key/value cache support, and validateTileMixthrough long\-context retrieval, question answering, prefill efficiency, and numerical analyses across LLaMA, Qwen, and Vicuna models\.

## 2Related Work

Transformer acceleration primarily follows three directions: low\-precision quantization, IO\-aware tiled attention, and structural sparsity\. These approaches reduce long\-context inference cost through numerical compression, data\-movement optimization, or selective token connectivity\.

Low\-precision quantizationreduces memory and arithmetic costs by representing weights and activations in formats such as INT8[31](https://arxiv.org/html/2608.17336#bib.bib1);[27](https://arxiv.org/html/2608.17336#bib.bib2)and INT4[40](https://arxiv.org/html/2608.17336#bib.bib3)\. Quantization\-aware and post\-training methods improve robustness through calibration, activation transformation, outlier handling, and blockwise scaling[34](https://arxiv.org/html/2608.17336#bib.bib4);[33](https://arxiv.org/html/2608.17336#bib.bib5);[24](https://arxiv.org/html/2608.17336#bib.bib7)\. Recent quantized attention kernels integrate low\-precision score computation, value aggregation, and numerical approximation into fused execution[4](https://arxiv.org/html/2608.17336#bib.bib12);[16](https://arxiv.org/html/2608.17336#bib.bib23);[38](https://arxiv.org/html/2608.17336#bib.bib26);[26](https://arxiv.org/html/2608.17336#bib.bib39)\. These designs typically assign formats at the tensor, operator, attention\-stage, or quantization\-block level, while two\-dimensional score\-tile groups follow one arithmetic path within the streaming loop[14](https://arxiv.org/html/2608.17336#bib.bib6);[31](https://arxiv.org/html/2608.17336#bib.bib1);[18](https://arxiv.org/html/2608.17336#bib.bib8)\.

IO\-aware attention kernels[11](https://arxiv.org/html/2608.17336#bib.bib9);[12](https://arxiv.org/html/2608.17336#bib.bib10);[13](https://arxiv.org/html/2608.17336#bib.bib38)process attention in SRAM\-resident tiles and fuse score computation, online softmax, and value aggregation, avoiding full attention\-matrix materialization in HBM\. These tiles govern data movement and work partitioning, while arithmetic precision is commonly fixed per kernel invocation or attention stage[25](https://arxiv.org/html/2608.17336#bib.bib11);[4](https://arxiv.org/html/2608.17336#bib.bib12)\.

Sparse attention methodsreduce computation by selecting token interactions through sliding\-window, strided, dilated, or mixed local\-global patterns[3](https://arxiv.org/html/2608.17336#bib.bib13);[5](https://arxiv.org/html/2608.17336#bib.bib14);[37](https://arxiv.org/html/2608.17336#bib.bib15);[2](https://arxiv.org/html/2608.17336#bib.bib16)\. Other systems use content\-based selection and clustering[23](https://arxiv.org/html/2608.17336#bib.bib17);[32](https://arxiv.org/html/2608.17336#bib.bib18);[17](https://arxiv.org/html/2608.17336#bib.bib19), positional mechanisms[10](https://arxiv.org/html/2608.17336#bib.bib20);[39](https://arxiv.org/html/2608.17336#bib.bib21), adaptive architectures[8](https://arxiv.org/html/2608.17336#bib.bib22), or dynamically constructed active block sets[15](https://arxiv.org/html/2608.17336#bib.bib32);[19](https://arxiv.org/html/2608.17336#bib.bib33);[35](https://arxiv.org/html/2608.17336#bib.bib37)\. These methods use spatial structure to select executed token interactions, changing attention connectivity while keeping precision outside the selection decision\.

## 3Preliminaries

We briefly review \(i\) FlashAttention\-style tiled attention with online softmax, and \(ii\) blockwise quantization for low\-precision matrix multiplication\. Their interaction defines the central kernel challenge addressed byTileMix: FP16 and INT8 score tiles follow different arithmetic paths but contribute to one shared streaming softmax state\. Appendix Table[4](https://arxiv.org/html/2608.17336#A1.T4)summarizes the notation used throughout the paper\.

### 3\.1Tiled Attention with Online Softmax

Consider one self\-attention head withQ,K,V∈ℝL×dQ,K,V\\in\\mathbb\{R\}^\{L\\times d\}, whereLLis the sequence length andddis the head dimension\. Attention computes the score matrixS∈ℝL×LS\\in\\mathbb\{R\}^\{L\\times L\}, attention weightsP∈ℝL×LP\\in\\mathbb\{R\}^\{L\\times L\}, and outputO∈ℝL×dO\\in\\mathbb\{R\}^\{L\\times d\}asS=Q​K⊤d,P=softmax⁡\(S\),O=P​V\.S=\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\}\},P=\\mathrm\{softmax\}\(S\),O=PV\.MaterializingSSandPPin GPU high\-bandwidth memory \(HBM\) requires an𝒪⁡\(L2\)\\mathcal\{O\}\(L^\{2\}\)intermediate\-memory footprint and substantial*HBM read/write traffic*at long context lengths\.

FlashAttention streams key/value tiles while keeping score and probability tiles on chip and writing only normalized outputs to HBM, providing the execution substrate forTileMix’s tile\-group precision routing\. The kernel partitionsQQintoTm=⌈L/bq⌉T\_\{m\}=\\left\\lceil L/b\_\{q\}\\right\\rceiltiles\{Qm\}m=1Tm\\\{Q\_\{m\}\\\}\_\{m=1\}^\{T\_\{m\}\}and\(K,V\)\(K,V\)intoTn=⌈L/bk​v⌉T\_\{n\}=\\left\\lceil L/b\_\{kv\}\\right\\rceiltiles\{\(Kn,Vn\)\}n=1Tn\\\{\(K\_\{n\},V\_\{n\}\)\\\}\_\{n=1\}^\{T\_\{n\}\}\. For eachQmQ\_\{m\}, it streams overn=1,…,Tnn=1,\\dots,T\_\{n\}while maintaining the shared online\-softmax state\(m~mn,z~mn,O~mn\)\\big\(\\tilde\{m\}\_\{m\}^\{\\,n\},\\tilde\{z\}\_\{m\}^\{\\,n\},\\tilde\{O\}\_\{m\}^\{\\,n\}\\big\), comprising the row\-wise maximum and normalizerm~mn,z~mn∈ℝbq\\tilde\{m\}\_\{m\}^\{\\,n\},\\tilde\{z\}\_\{m\}^\{\\,n\}\\in\\mathbb\{R\}^\{b\_\{q\}\}and unnormalized accumulatorO~mn∈ℝbq×d\\tilde\{O\}\_\{m\}^\{\\,n\}\\in\\mathbb\{R\}^\{b\_\{q\}\\times d\}\. Initialized bym~m0=−∞\\tilde\{m\}\_\{m\}^\{\\,0\}=\-\\infty,z~m0=0\\tilde\{z\}\_\{m\}^\{\\,0\}=0, andO~m0=0\\tilde\{O\}\_\{m\}^\{\\,0\}=0, the state updates as

m~mn\\displaystyle\\tilde\{m\}\_\{m\}^\{\\,n\}=max⁡\{m~mn−1,rowmax⁡\(Smn\)\},\\displaystyle=\\max\\\!\\Big\\\{\\tilde\{m\}\_\{m\}^\{\\,n\-1\},\\mathrm\{rowmax\}\(S\_\{m\}^\{\\,n\}\)\\Big\\\},z~mn\\displaystyle\\tilde\{z\}\_\{m\}^\{\\,n\}=em~mn−1−m~mn​z~mn−1\+rowsum⁡\(eSmn−m~mn\),\\displaystyle=e^\{\\tilde\{m\}\_\{m\}^\{\\,n\-1\}\-\\tilde\{m\}\_\{m\}^\{\\,n\}\}\\tilde\{z\}\_\{m\}^\{\\,n\-1\}\+\\mathrm\{rowsum\}\\\!\\left\(e^\{S\_\{m\}^\{\\,n\}\-\\tilde\{m\}\_\{m\}^\{\\,n\}\}\\right\),O~mn\\displaystyle\\tilde\{O\}\_\{m\}^\{\\,n\}=em~mn−1−m~mn​O~mn−1\+eSmn−m~mn​Vn,\\displaystyle=e^\{\\tilde\{m\}\_\{m\}^\{\\,n\-1\}\-\\tilde\{m\}\_\{m\}^\{\\,n\}\}\\tilde\{O\}\_\{m\}^\{\\,n\-1\}\+e^\{S\_\{m\}^\{\\,n\}\-\\tilde\{m\}\_\{m\}^\{\\,n\}\}V\_\{n\},Smn\\displaystyle S\_\{m\}^\{\\,n\}=Qm​Kn⊤d\.\\displaystyle=\\frac\{Q\_\{m\}K\_\{n\}^\{\\top\}\}\{\\sqrt\{d\}\}\.Here,rowmax⁡\(⋅\)\\mathrm\{rowmax\}\(\\cdot\)androwsum⁡\(⋅\)\\mathrm\{rowsum\}\(\\cdot\)reduce across thebk​vb\_\{kv\}columns ofSmn∈ℝbq×bk​vS\_\{m\}^\{\\,n\}\\in\\mathbb\{R\}^\{b\_\{q\}\\times b\_\{kv\}\}\. After tileTnT\_\{n\}, row\-wise division ofO~mTn\\tilde\{O\}\_\{m\}^\{\\,T\_\{n\}\}byz~mTn\\tilde\{z\}\_\{m\}^\{\\,T\_\{n\}\}yields the output forQmQ\_\{m\}\.

### 3\.2Blockwise Quantization

For a matrix productC=A​BC=AB, blockwise quantization appliesψ⁡\(⋅\)\\psi\(\\cdot\)independently to operand blocks, producing low\-precision representations and their scale factors:

\(A^,δA\)=ψ⁡\(A\),\(B^,δB\)=ψ⁡\(B\),C≈δA​δB​\(A^​B^\)\.\(\\hat\{A\},\\delta\_\{A\}\)=\\psi\(A\),\\;\(\\hat\{B\},\\delta\_\{B\}\)=\\psi\(B\),\\;C\\approx\\delta\_\{A\}\\delta\_\{B\}\(\\hat\{A\}\\hat\{B\}\)\.The representationsA^\\hat\{A\}andB^\\hat\{B\}may use INT8, FP8, or other low\-precision formats\. We instantiateψ⁡\(⋅\)\\psi\(\\cdot\)withINT8because NVIDIA A100 GPUs provide optimizedINT8 Tensor Coreprimitives for high\-throughput matrix multiplication\. These Tensor Cores executeMMA\(*matrix multiply\-accumulate*\) instructions that multiply INT8 fragments and accumulate partial sums in INT32 registers\. Aligning the quantization blocks with attention tiles allows the corresponding scalesδA\\delta\_\{A\}andδB\\delta\_\{B\}to be applied efficiently during fused execution\.

FP16 and INT8 score tiles exhibit different rounding, accumulation, and rescaling behavior\. Because every score tile contributes to the shared state\(m~mn,z~mn,O~mn\)\(\\tilde\{m\}\_\{m\}^\{\\,n\},\\tilde\{z\}\_\{m\}^\{\\,n\},\\tilde\{O\}\_\{m\}^\{\\,n\}\), path\-specific numerical effects propagate through running\-maximum tracking, normalization, and output accumulation\. Mixed\-precision attention is therefore a shared\-state kernel problem: heterogeneous score paths must enter a common score domain before updating the same running maximum, normalizer, and output accumulator\.

Figure 2:TileMixfused attention with tile\-group precision routing\.*Left:*TheLq×LkL\_\{q\}\\times L\_\{k\}attention matrix is processed inBLOCKM×BLOCKN\\mathrm\{BLOCK\}\_\{M\}\\times\\mathrm\{BLOCK\}\_\{N\}compute tiles\. For each query\-tile rowmm\(outer loop\), the kernel streams key/value tilesnn\(inner loop\) from HBM to on\-chip SRAM and registers, dispatches the score tileSmn=Qm​Kn⊤/dS\_\{m\}^\{\\,n\}=Q\_\{m\}K\_\{n\}^\{\\top\}/\\sqrt\{d\}to either FP16 matmul or INT8 Tensor Core MMA with INT32 accumulation and rescaling, and updates a shared FP16 online\-softmax state to produceOmO\_\{m\}without materializing the full score matrix\.*Right:*A binary routing map is grouped along the key\-tile dimension with widthBLOCKNmask=g​BLOCKN\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}=g\\,\\mathrm\{BLOCK\}\_\{N\}and packed per\(hk,m\)\(h\_\{k\},m\)into a 64\-bit maskbhk,mb\_\{h\_\{k\},m\}\. The bit at positiongj=⌊n/g⌋g\_\{j\}=\\lfloor n/g\\rfloorselects the arithmetic path for the corresponding key\-tile group \(1: INT8, 0: FP16\)\. Kernel\-side legality masks, including causality and boundary conditions, are enforced independently of the routing map\.

## 4TileMix: Tile\-Group Precision Routing

TileMixrealizes tile\-group precision routing*without retraining*by combining: \(i\) a precision policy that assigns each legal score\-tile group toFP16orINT8score computation, \(ii\) packed bitmasks that convert the policy into constant\-time inner\-loop dispatch, and \(iii\) a FlashAttention\-style fused kernel in which both score paths update one shared online\-softmax state\. For each KV head and query\-tile row, the inner loop reads one packed routing word and extracts the arithmetic\-path decision for each key\-tile group\. The fused kernel natively supports grouped\-query attention \(GQA\) and variable\-length batching through prefix\-sum metadata, while the implementation provides an INT8 key/value cache interface for decode\.

### 4\.1TileMixOverview

TileMixpartitions theLq×LkL\_\{q\}\\times L\_\{k\}attention into hardware\-aligned two\-dimensional score tiles of sizeBLOCKM×BLOCKN\\mathrm\{BLOCK\}\_\{M\}\\times\\mathrm\{BLOCK\}\_\{N\}\(Figure[2](https://arxiv.org/html/2608.17336#S3.F2)\), withBLOCKM≡bq\\mathrm\{BLOCK\}\_\{M\}\\equiv b\_\{q\}andBLOCKN≡bk​v\\mathrm\{BLOCK\}\_\{N\}\\equiv b\_\{kv\}\. The compute tile remains the execution unit, while one routing group may govern multiple adjacent compute tiles along the key dimension\. The fused kernel follows the two\-level structure of IO\-aware attention:

Outer loop \(over query tiles\)\.The kernel iterates over query tiles\{Qm\}m=1Tm\\\{Q\_\{m\}\\\}\_\{m=1\}^\{T\_\{m\}\}, whereTm=⌈Lq/BLOCKM⌉T\_\{m\}=\\lceil L\_\{q\}/\\mathrm\{BLOCK\}\_\{M\}\\rceil\. Each Triton program instance owns one query\-tile rowmm\.

Inner loop \(streaming key/value tiles\)\.For eachQmQ\_\{m\}, the kernel streams key/value tiles\{\(Kn,Vn\)\}n=1Tn\\\{\(K\_\{n\},V\_\{n\}\)\\\}\_\{n=1\}^\{T\_\{n\}\}from HBM into SRAM and registers, whereTn=⌈Lk/BLOCKN⌉T\_\{n\}=\\lceil L\_\{k\}/\\mathrm\{BLOCK\}\_\{N\}\\rceil\. Across this inner loop, the kernel updates one online\-softmax state and retains the output accumulator on chip until normalization after the final tile\.

For each compute tile\(m,n\)\(m,n\), the routing group containing key\-tile indexnnselects theFP16orINT8score\-computation path\. Kernel\-side causal and boundary masks determine legal interactions, while the routing policy determines the arithmetic path of each legal tile group\.

TileMixsupports grouped\-query attention \(GQA\) and variable\-length batching through one routing interface\. LetHqH\_\{q\}andHkH\_\{k\}denote the numbers of query and KV heads\. Each query headhq∈\{0,…,Hq−1\}h\_\{q\}\\in\\\{0,\\dots,H\_\{q\}\-1\\\}maps to a KV headhk=⌊hqHq/Hk⌋,h\_\{k\}=\\left\\lfloor\\frac\{h\_\{q\}\}\{H\_\{q\}/H\_\{k\}\}\\right\\rfloor,which indexes routing lookup\. Query heads mapped to the same KV head share tile\-group routing decisions\. For variable\-length batching, flattened inputs usecu​\_​seqlens\\mathrm\{cu\\\_seqlens\}and prefix\-sum metadata for padding\-free routing and execution\.

### 4\.2Routing Policy and Bitmask Encoding

Fused attention kernels derive tile execution from pointers, strides, legality masks, and a shared arithmetic configuration\.TileMixintroduces binary routing metadata that selects the score\-computation path inside the inner loop\.

Policy definition\.TileMixaccepts a binary tile\-group routing mapRRindexed by KV headhkh\_\{k\}, query\-tile rowmm, and key\-tile groupgjg\_\{j\}\. In our evaluation,*static, data\-free*structured templates instantiateRR, distributing FP16\-routed groups spatially under configurable INT8 coverage budgets\. The decisionRhk,m,gj=1R\_\{h\_\{k\},m,g\_\{j\}\}=1dispatches the corresponding group toINT8, whileRhk,m,gj=0R\_\{h\_\{k\},m,g\_\{j\}\}=0dispatches it toFP16\. \(See Appendix[B](https://arxiv.org/html/2608.17336#A2)for the full list of precision layouts and their definitions\.\)

Figure 3:Key\-tile grouping and bitmask encoding for one query\-tile rowmm\. Adjacent compute tiles along the key dimension form routing groupsgjg\_\{j\}, each controlled by one bit in the packed maskbhk,mb\_\{h\_\{k\},m\}\(1: INT8, 0: FP16\)\.Why grouping is needed\.TileMixtargets constant\-time routing lookup inside the attention inner loop\. As Figure[3](https://arxiv.org/html/2608.17336#S4.F3)shows, we pack up to 64 key\-tile\-group decisions for each query\-tile row into a single 64\-bit word\. To extend the same routing word across longer key sequences while retaining hardware\-aligned compute tiles,TileMixdefines the routing\-group widthBLOCKNmask\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}through a grouping factorg∈ℤ\+g\\in\\mathbb\{Z\}^\{\+\}:

BLOCKNmask=g⋅BLOCKN,Tmask=⌈LkBLOCKNmask⌉≤64\.\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}=g\\cdot\\mathrm\{BLOCK\}\_\{N\},\\;T\_\{\\text\{mask\}\}=\\left\\lceil\\frac\{L\_\{k\}\}\{\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}\}\\right\\rceil\\leq 64\.
Wheng\>1g\>1,one routing bit controls a contiguous group ofggadjacent key tiles\(Figure[3](https://arxiv.org/html/2608.17336#S4.F3)\)\. Each decision therefore spansg⋅BLOCKNg\\cdot\\mathrm\{BLOCK\}\_\{N\}key tokens\. This grouping extends one routing word across long key sequences while preserving the hardware\-aligned compute\-tile structure\.

Figure 4:Constant\-time shift\-and\-mask routing lookup\. Shifting bitgjg\_\{j\}to the least\-significant position yields the arithmetic\-path decisionRhk,m,gjR\_\{h\_\{k\},m,g\_\{j\}\}for the corresponding key\-tile group\.Bitmask packing\.After grouping, the routing mapRRis indexed by KV headhkh\_\{k\}, query\-tile rowmm, and key\-tile groupgj∈\{0,…,Tmask−1\}g\_\{j\}\\in\\\{0,\\dots,T\_\{\\text\{mask\}\}\-1\\\}\. For each\(hk,m\)\(h\_\{k\},m\), we pack the group decisions into a 64\-bit integer, or packed*bitmask word*,

bhk,m=∑gj=0Tmask−1Rhk,m,gj​2gj,Rhk,m,gj∈\{0,1\}\.b\_\{h\_\{k\},m\}=\\sum\_\{g\_\{j\}=0\}^\{T\_\{\\text\{mask\}\}\-1\}R\_\{h\_\{k\},m,g\_\{j\}\}\\,2^\{g\_\{j\}\},\\;R\_\{h\_\{k\},m,g\_\{j\}\}\\in\\\{0,1\\\}\.
Constant\-time lookup\.Inside the kernel, one shift\-and\-mask operation retrieves the routing decision for groupgjg\_\{j\}\(Figure[4](https://arxiv.org/html/2608.17336#S4.F4)\):

Rhk,m,gj=\(\(bhk,m≫gj\)&1\)\.R\_\{h\_\{k\},m,g\_\{j\}\}=\\left\(\(b\_\{h\_\{k\},m\}\\gg g\_\{j\}\)\\;\\&\\;1\\right\)\.The general representation requires one 64\-bit word per\(hk,m\)\(h\_\{k\},m\), totaling𝒪⁡\(Hk​Tm\)\\mathcal\{O\}\(H\_\{k\}T\_\{m\}\)routing metadata\. Packed bitmasks therefore provide compact kernel\-native routing metadata consumed directly by the streaming inner loop\.

### 4\.3Tile\-Group Mixed\-Precision Attention

TileMixpreserves the FlashAttention\-style online\-softmax recurrence defined in Section[3](https://arxiv.org/html/2608.17336#S3); routing changes only the arithmetic path used to construct each score tileSmnS\_\{m\}^\{\\,n\}\.

The FP16 and INT8 paths contribute distinct rounding, accumulation, and rescaling behavior\. TheFP16andINT8paths use different multiplication, accumulation, and scale\-restoration procedures\. After INT8 scale restoration, score tiles from both paths enter a common floating\-point domain before updating the shared running maximum, normalizer, and output accumulator\.TileMixmaintains this shared online\-softmax state in FP16, while the routing mapRRassignsFP16orINT8score computation to each legal tile group\. Both arithmetic paths apply the same1/d1/\\sqrt\{d\}scaling and exponentiation implementation before contributing to the shared normalization\.

Tile\-group dispatch\.For query\-tile rowmmand key\-tile indexnn, the routing\-group index isgj=⌊n/g⌋g\_\{j\}=\\lfloor n/g\\rfloor, withBLOCKNmask=g​BLOCKN\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}=g\\,\\mathrm\{BLOCK\}\_\{N\}\. The corresponding routing bit defines the routed score tileSmnS\_\{m\}^\{\\,n\}:

Smn=\{Rescalem,n⁡\(Q8,m​K8,n⊤\)/d,Rhk,m,gj=1,Qm​Kn⊤/d,Rhk,m,gj=0\.S\_\{m\}^\{\\,n\}=\\begin\{cases\}\\operatorname\{Rescale\}\_\{m,n\}\\\!\\left\(Q\_\{8,m\}K\_\{8,n\}^\{\\top\}\\right\)/\\sqrt\{d\},&R\_\{h\_\{k\},m,g\_\{j\}\}=1,\\\\\[2\.0pt\] Q\_\{m\}K\_\{n\}^\{\\top\}/\\sqrt\{d\},&R\_\{h\_\{k\},m,g\_\{j\}\}=0\.\\end\{cases\}
INT8 path and scale alignment\.For the primary evaluated score\-routing path,TileMixquantizesQQandKKblockwise to produceQ8Q\_\{8\}andK8K\_\{8\}together with per\-block scales, whileVVand theP​VPVcomputation remain in FP16\. The quantize\-once mode preparesQ8Q\_\{8\},K8K\_\{8\}, and their scales once per attention call and reuses them across all INT8\-routed tile groups\.

Figure 5:Line\-level retrieval accuracy on the LongEval benchmark forLLaMA 3\.2 3Bunder different tile\-group routing layouts, evaluated across prompt lengths from 3\.1k to 38\.7k tokens\. Line plots \(left y\-axis\) report exact\-match retrieval accuracy, while bar plots \(right y\-axis,Δ\\Deltaaccuracy vs\. FP16\) show differences relative to the FP16 attention baseline\. Each panel corresponds to a routing layout with 25%, 50%, or 75% of tile groups routed to INT8\. Colors indicate routing settings:greendenotes the original FP16 attention baseline,reddenotesOne \(all legal score\-tile groups routed to INT8\), andbluedenotes mixed configurations with partially INT8\-routed tile groups\.The INT8 path computesQ8,m​K8,n⊤Q\_\{8,m\}K\_\{8,n\}^\{\\top\}with INT8 MMA and INT32 accumulation, then applies block scales and1/d1/\\sqrt\{d\}to produceSmnS\_\{m\}^\{\\,n\}\. FP16\-routed tiles computeQm​Kn⊤/dQ\_\{m\}K\_\{n\}^\{\\top\}/\\sqrt\{d\}through the FP16 path and update the same online\-softmax state\. Variable\-length execution stores compact per\-block scales indexed through the fused kernel’s prefix\-sum metadata\. Appendix[D](https://arxiv.org/html/2608.17336#A4)distinguishes this cache interface from the prefill path and details the tensor, scale, temporary\-buffer, and data\-movement layouts\.

Finally, tile\-group policies become inner\-loop dispatch through constant\-time bitmask lookup\. The selectedFP16andINT8score paths update a shared FP16 online\-softmax state\. Together, compact metadata, constant\-time lookup, and shared\-state execution integrate heterogeneous score arithmetic into one dense fused attention computation\.

## 5Experiments

### 5\.1Experimental Setup

We evaluateTileMixfrom four perspectives: \(i\) long\-context retrieval, \(ii\) long\-context question answering, \(iii\) prefill efficiency, and \(iv\) numerical behavior under routed precision\. The main evaluation uses LLaMA 3\.2 3B[1](https://arxiv.org/html/2608.17336#bib.bib24)\. Appendix[E](https://arxiv.org/html/2608.17336#A5)extends quality evaluation to Vicuna\-7B, Qwen\-2\-7B[28](https://arxiv.org/html/2608.17336#bib.bib27), and Qwen\-2\.5\-7B[29](https://arxiv.org/html/2608.17336#bib.bib28), while Appendix[F](https://arxiv.org/html/2608.17336#A6)reports efficiency across the same model families\.

Dataset and Metrics\.We evaluate long\-context question answering on LV\-Eval[36](https://arxiv.org/html/2608.17336#bib.bib29), which covers 11 English and Chinese datasets at context lengths from 16k to 64k tokens\. We use LongEval[21](https://arxiv.org/html/2608.17336#bib.bib25)to evaluate line\-level retrieval with exact\-match accuracy\. Each LongEval example embeds a uniquely labeled target line in a sequence of up to 54k tokens and queries the model for its associated content\.

Baselines\.We compare against \(i\) dense FP16 attention as the full\-precision reference, \(ii\)One, which routes all legal score\-tile groups to INT8 using the sameTileMixkernel substrate, \(iii\) FlashAttention as the IO\-aware FP16 execution baseline, \(iv\) MInference[15](https://arxiv.org/html/2608.17336#bib.bib32)and FlexPrefill[19](https://arxiv.org/html/2608.17336#bib.bib33)as sparse long\-context baselines, and \(v\) SageAttention[38](https://arxiv.org/html/2608.17336#bib.bib26)as a representative INT8 attention kernel\.

Hardware and Measurement\.All experiments run on NVIDIA A100 40GB GPUs\.TileMix’s offline autotuner selects an A100 kernel configuration that remains fixed throughout evaluation\. Throughput evaluation uses batch size 8, three warmup iterations, and five timed iterations with the same hardware and model wrapper for all methods\. Timings include quantization, scale restoration, routing, memory staging, and kernel\-scheduling costs\. We therefore report end\-to\-end prefill throughput rather than isolated MMA throughput\.

Dataset\|Len\|FP16\|One\|SpTrans25SpTrans50SpTrans75\|BigBird25BigBird50BigBird75\|MInference\|FlexPrefill\|SageAttn1\|16k \|32\.04 \|28\.78 \|31\.7532\.1830\.25 \|31\.2729\.6429\.75 \|26\.93 \|27\.11 \|29\.79\|32k \|15\.08 \|11\.62 \|15\.5614\.7714\.37 \|15\.2214\.0213\.55 \|11\.55 \|11\.74 \|13\.50\|64k \|7\.75 \|5\.42 \|8\.017\.446\.83 \|8\.016\.475\.72 \|5\.31 \|5\.39 \|5\.772\|16k \|18\.49 \|15\.62 \|18\.6418\.0816\.78 \|18\.6116\.9816\.62 \|14\.61 \|14\.92 \|16\.67\|32k \|15\.12 \|12\.02 \|15\.2714\.6914\.18 \|15\.1413\.5313\.28 \|12\.01 \|11\.64 \|14\.59\|64k \|11\.84 \|7\.79 \|12\.0611\.3411\.04 \|12\.0410\.2810\.03 \|7\.96 \|6\.81 \|11\.223\|16k \|6\.72 \|4\.45 \|21\.0420\.5321\.65 \|6\.296\.974\.57 \|5\.41 \|5\.01 \|5\.88\|32k \|3\.78 \|1\.95 \|3\.352\.782\.68 \|2\.351\.811\.20 \|2\.75 \|2\.51 \|3\.15\|64k \|3\.38 \|1\.42 \|3\.242\.672\.47 \|3\.241\.701\.37 \|2\.13 \|1\.91 \|2\.734\|16k \|6\.00 \|2\.27 \|6\.005\.435\.04 \|6\.004\.464\.49 \|4\.77 \|4\.53 \|5\.19\|32k \|18\.00 \|13\.98 \|18\.0017\.4317\.54 \|18\.5016\.4615\.99 \|12\.99 \|11\.99 \|15\.08\|64k \|14\.50 \|10\.07 \|15\.0014\.4314\.04 \|15\.0013\.4612\.49 \|9\.03 \|8\.31 \|11\.555\|16k \|17\.51 \|12\.86 \|17\.6617\.0916\.88 \|17\.6615\.1714\.84 \|13\.97 \|13\.16 \|15\.27\|32k \|10\.80 \|8\.38 \|11\.0110\.4410\.06 \|10\.619\.079\.00 \|7\.81 \|7\.20 \|9\.11\|64k \|6\.82 \|5\.32 \|7\.636\.255\.97 \|7\.636\.095\.62 \|4\.26 \|3\.86 \|5\.506\|16k \|12\.33 \|8\.51 \|12\.3111\.7411\.03 \|12\.3810\.4010\.30 \|9\.82 \|9\.28 \|10\.69\|32k \|6\.73 \|3\.22 \|7\.086\.515\.95 \|6\.525\.184\.45 \|4\.88 \|4\.48 \|5\.69\|64k \|1\.84 \|0\.74 \|1\.971\.371\.22 \|1\.911\.351\.13 \|1\.12 \|1\.07 \|1\.447\|16k \|21\.75 \|15\.69 \|22\.0721\.1621\.05 \|21\.9620\.8120\.09 \|15\.59 \|14\.05 \|19\.94\|32k \|19\.89 \|12\.11 \|19\.9119\.3718\.56 \|20\.1718\.4117\.79 \|12\.23 \|10\.95 \|15\.92\|64k \|14\.18 \|6\.93 \|14\.2313\.6613\.34 \|14\.0912\.3711\.97 \|7\.85 \|6\.67 \|11\.248\|16k \|18\.24 \|12\.72 \|17\.5316\.9917\.46 \|17\.5716\.4515\.96 \|12\.32 \|11\.39 \|13\.93\|32k \|14\.25 \|9\.12 \|13\.6313\.1813\.00 \|13\.7312\.7312\.29 \|9\.26 \|7\.40 \|10\.54\|64k \|11\.08 \|7\.54 \|11\.3110\.6510\.38 \|11\.059\.609\.04 \|7\.24 \|5\.08 \|9\.459\|16k \|45\.68 \|34\.70 \|45\.7744\.6044\.88 \|45\.1343\.6643\.86 \|38\.16 \|35\.24 \|42\.12\|32k \|26\.84 \|18\.17 \|26\.0525\.9425\.11 \|26\.4724\.9423\.78 \|20\.09 \|21\.40 \|22\.50\|64k \|16\.36 \|9\.19 \|16\.0315\.2514\.71 \|15\.7214\.1814\.04 \|10\.24 \|10\.69 \|13\.2110\|16k \|28\.88 \|22\.37 \|29\.1028\.3127\.23 \|28\.9527\.5726\.33 \|23\.28 \|23\.43 \|25\.18\|32k \|18\.33 \|13\.66 \|18\.2617\.6517\.41 \|18\.1716\.7016\.88 \|12\.70 \|10\.28 \|13\.94\|64k \|15\.75 \|12\.81 \|15\.9415\.3215\.39 \|15\.9314\.2213\.88 \|9\.42 \|7\.68 \|13\.3711\|16k \|22\.44 \|16\.97 \|22\.9422\.3721\.74 \|22\.4420\.2420\.76 \|15\.99 \|12\.64 \|18\.19\|32k \|14\.85 \|12\.02 \|15\.6115\.0415\.06 \|16\.1114\.5713\.34 \|11\.33 \|10\.25 \|14\.57\|64k \|9\.26 \|3\.41 \|8\.928\.357\.40 \|8\.928\.387\.91 \|3\.10 \|4\.02 \|5\.79

Table 1:LV\-Eval long\-context question answering accuracy forLLaMA 3\.2 3B\. We compare FP16 attention,One \(100% INT8\), representativeTileMixrouting layouts, sparse long\-context baselines \(MInference and FlexPrefill\), and an INT8 attention baseline \(SageAttention\)\. Sparse baselines reduce computation by pruning token interactions, whereasTileMixpreserves dense legal connectivity and routes legal tile groups to different score\-computation paths\. Cells are color\-coded by comparison toOneand FP16:redindicates performance belowOne,blueindicates performance betweenOneand FP16, andgreenindicates performance above FP16\.
### 5\.2Model Performance

Long\-context Retrieval\.LongEval evaluates exact retrieval from sequences of labeled lines\. Each input contains entries such as “line wacky\-cob: CONTENT”, and the query specifies a label\. The model returns the associated content, with exact\-match accuracy measuring retrieval success\.

Figure[5](https://arxiv.org/html/2608.17336#S4.F5)reports LLaMA 3\.2 3B retrieval accuracy from 3\.1k to 38\.7k tokens\. FP16 andOneprovide the full\-precision and matched uniform\-INT8 references, respectively\. Each panel shows one spatial routing layout at 25%, 50%, and 75% INT8 tile\-group coverage\. Retrieval quality depends on FP16 placement in addition to nominal INT8 coverage\.row\_randandsptransretain stronger accuracy as INT8 coverage increases, whilealign\_sparse,band, andglobalretain more quality under conservative coverage\. Together, routing layout determines where FP16 computation is retained, while coverage controls INT8 execution\.

Long\-context Question Answering\.LV\-Eval covers 11 English and Chinese long\-context QA datasets under confusing\-fact insertion \(CFI\), keyword and phrase replacement \(KPR\), and keyword\-recall \(AK\) settings\. Datasets1cmrc\_mixupand2dureader\_mixupcover Chinese machine reading comprehension and question answering, and datasets3factrecall\_enand4factrecall\_zhcover bilingual factual recall\. Dataset5hotpotwikiqa\_mixupevaluates multi\-hop retrieval, dataset6lic\_mixupevaluates precise information localization, and datasets7loogle\_CR\_mixup,8loogle\_MIR\_mixup, and9loogle\_SD\_mixupevaluate content recall, multi\-information retrieval, and sequential dependency retrieval\. Datasets10multifieldqa\_en\_mixupand11multifieldqa\_zh\_mixupevaluate bilingual multi\-field question answering\.

Len\|Metric\|Torch\|FlashAttn\|One\|SpTrans25\|SpTrans50\|SpTrans75\|BigBird25\|BigBird50\|BigBird75\|MInference\|FlexPrefill\|SageAttn1k \|Thpt \|11\.14 \|17\.45 \|32\.27 \|27\.66 \|32\.11 \|33\.50 \|26\.92 \|31\.48 \|32\.37 \|1\.96 \|8\.01 \|19\.91\|TOPS \|41\.70 \|65\.31 \|120\.80 \|103\.50 \|120\.20 \|125\.38 \|100\.78 \|117\.85 \|121\.17 \|7\.33 \|29\.98 \|74\.542k \|Thpt \|7\.78 \|16\.48 \|32\.06 \|27\.11 \|31\.46 \|33\.92 \|26\.57 \|28\.83 \|32\.71 \|2\.79 \|12\.32 \|20\.20\|TOPS \|30\.50 \|64\.64 \|125\.70 \|106\.32 \|123\.38 \|132\.99 \|104\.23 \|113\.05 \|128\.31 \|10\.93 \|48\.30 \|79\.194k \|Thpt \|OOM\|14\.33 \|29\.80 \|27\.14 \|30\.59 \|31\.80 \|24\.69 \|26\.84 \|27\.48 \|4\.70 \|15\.45 \|19\.91\|TOPS \|OOM\|61\.31 \|127\.48 \|116\.09 \|130\.85 \|136\.03 \|105\.63 \|114\.83 \|117\.56 \|20\.13 \|66\.11 \|85\.198k \|Thpt \|OOM\|OOM\|27\.41 \|22\.81 \|25\.69 \|26\.61 \|23\.02 \|25\.17 \|25\.84 \|7\.24 \|15\.80 \|18\.79\|TOPS \|OOM\|OOM\|136\.76 \|113\.83 \|128\.21 \|132\.84 \|114\.92 \|125\.58 \|128\.90 \|36\.13 \|78\.84 \|93\.79

Table 2:Implementation\-level prefill throughput \(Thpt, K tokens/s\) and TOPS forLLaMA 3\.2 3B\-Instruct\. All methods use the same A100 40GB hardware, batch size 8, three warmup iterations, five timed iterations, and model wrapper; timings include method\-specific quantization, scale restoration, routing, memory staging, and scheduling costs\. TOPS uses a common dense\-attention operation count divided by measured end\-to\-end time\. Colors compare each row with FlashAttention andOne: red indicates performance below FlashAttention, blue indicates performance between FlashAttention andOne, and green indicates performance aboveOne; OOM entries are gray\.Table[1](https://arxiv.org/html/2608.17336#S5.T1)compares FP16, the uniform\-INT8 referenceOne, representativeTileMixconfigurations, sparse methods, and SageAttention\.Oneoften trails FP16, showing that uniform INT8 can be too coarse for long\-context QA\.TileMixnarrows this gap by assigning FP16 to selected score\-tile groups and INT8 to the rest while preserving all legal interactions\. The strongsptransresult at 16k factual recall recurs across LLaMA and Qwen models at all coverage ratios, indicating a consistent layout–task interaction\. Appendix[E](https://arxiv.org/html/2608.17336#A5)reports complete results across models and routing layouts\.

Seq Len\|0%5%10%1k \|7\.27×10−57\.27\{\\times\}10^\{\-5\}7\.47×10−47\.47\{\\times\}10^\{\-4\}1\.19×10−31\.19\{\\times\}10^\{\-3\}2k \|5\.89×10−55\.89\{\\times\}10^\{\-5\}4\.29×10−44\.29\{\\times\}10^\{\-4\}1\.16×10−31\.16\{\\times\}10^\{\-3\}4k \|4\.87×10−54\.87\{\\times\}10^\{\-5\}6\.71×10−46\.71\{\\times\}10^\{\-4\}1\.10×10−31\.10\{\\times\}10^\{\-3\}8k \|6\.84×10−56\.84\{\\times\}10^\{\-5\}8\.34×10−48\.34\{\\times\}10^\{\-4\}6\.32×10−36\.32\{\\times\}10^\{\-3\}Seq Len\|15%20%25%1k \|1\.56×10−31\.56\{\\times\}10^\{\-3\}1\.67×10−31\.67\{\\times\}10^\{\-3\}2\.03×10−32\.03\{\\times\}10^\{\-3\}2k \|1\.49×10−31\.49\{\\times\}10^\{\-3\}1\.66×10−31\.66\{\\times\}10^\{\-3\}1\.84×10−31\.84\{\\times\}10^\{\-3\}4k \|1\.41×10−31\.41\{\\times\}10^\{\-3\}1\.59×10−31\.59\{\\times\}10^\{\-3\}1\.78×10−31\.78\{\\times\}10^\{\-3\}8k \|6\.59×10−36\.59\{\\times\}10^\{\-3\}6\.71×10−36\.71\{\\times\}10^\{\-3\}6\.84×10−36\.84\{\\times\}10^\{\-3\}Table 3:Mean absolute output deviation from the fixed Torch FP16 reference under different score\-tile\-group INT8 coverage ratios; lower values indicate closer agreement with the reference implementation\.
### 5\.3Efficiency

Table[2](https://arxiv.org/html/2608.17336#S5.T2)reports prefill throughput and TOPS for LLaMA 3\.2 3B\-Instruct from 1k to 8k tokens\. Under this protocol,TileMiximproves throughput over FlashAttention by routing score\-tile groups to INT8 Tensor Cores\. At 4k tokens,SpTrans75reaches 31\.80 K tokens/s, compared with 14\.33 K tokens/s for FlashAttention and 29\.80 K tokens/s forOne\.TileMixpreserves dense legal connectivity while changing selected tile\-group arithmetic paths\. End\-to\-end throughput includes quantization, scale restoration, routing, memory staging, and kernel scheduling\. The ordering betweenOneand high\-coverage mixed configurations reflects the complete pipeline, including layout\-dependent dispatch and memory behavior\. Appendix[F](https://arxiv.org/html/2608.17336#A6)reports complete throughput and TOPS across models\.

### 5\.4Attention\-Kernel Numerical Behavior

We measure mean absolute deviation from a fixed Torch FP16 reference on randomized attention inputs\. Differences reflect quantization, scale restoration, accumulation, reduction order, and rounding across routed FP16 and INT8 score paths\.

Table[3](https://arxiv.org/html/2608.17336#S5.T3)reports deviation across INT8 coverage ratios and sequence lengths\. The 0% configuration remains close to FP16, while deviation generally increases with INT8 coverage\. At 8k tokens, deviation increases markedly between 5% and 10% coverage and remains at a similar scale through 25%, showing that coverage provides a practical numerical\-control knob\. Appendix[G](https://arxiv.org/html/2608.17336#A7)extends the analysis with model\-depth and sequence\-length studies, fused\-kernel comparisons, FP16\-versus\-FP32 accumulation controls, larger\-model checks, and heavy\-hitter exposure under static routing\.

## 6Conclusion

TileMixestablishes score\-tile\-group precision routing as a kernel\-native abstraction for long\-context attention\. Its kernel assigns FP16 and INT8 paths across legal tile groups, preserving the attention graph while varying precision within one computation\. Scalable grouping lets each routing decision span adjacent key tiles while retaining hardware\-aligned compute tiles and compact metadata\. Packed bitmasks and constant\-time lookup provide inner\-loop control, while both paths update a shared online\-softmax state\. It supports training\-free deployment, grouped\-query attention, variable\-length batching, and INT8 KV caches\. Across LongEval, LV\-Eval, prefill benchmarks, and numerical analyses on LLaMA, Qwen, and Vicuna,TileMixrecovers long\-context quality from uniform INT8 while improving prefill throughput over FP16\. This establishes a controllable accuracy\-efficiency frontier and demonstrates spatial precision control as a practical dimension for dense attention kernels\.

## Limitations

TileMixtargets forward inference during long\-context prefill, the deployment setting evaluated throughout this work\. The current implementation instantiates tile\-group routing with FP16 and INT8 Tensor Core paths on NVIDIA A100 GPUs\. Other numerical formats require format\-specific scale handling and kernel scheduling\. Static routing templates provide deterministic policy construction, compact metadata, and constant\-time kernel dispatch\. The kernel interface can consume alternative static or adaptive routing policies\.

## Ethical Considerations

TileMixis a systems method for improving the efficiency of long\-context LLM inference and does not introduce new training data, human\-subject data, or model capabilities\. Its primary broader impact is computational\. Improved attention efficiency may reduce GPU time and energy consumption per supported workload, while lower inference cost may also increase aggregate deployment and total compute demand\.

## References

- AI \(2024\)M\. AILlama 3\.2 model card\.External Links:[Link](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)Cited by:[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p1.1)\.
- Ainslieet al\.\(2020\)J\. Ainslie, S\. Ontanon, C\. Alberti, V\. Cvicek, Z\. Fisher, P\. Pham, A\. Ravula, S\. Sanghai, Q\. Wang, and L\. YangETC: encoding long and structured inputs in transformers\.External Links:2004\.08483,[Link](https://arxiv.org/abs/2004.08483)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Beltagyet al\.\(2020\)I\. Beltagy, M\. E\. Peters, and A\. CohanLongformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p11.1),[Appendix B](https://arxiv.org/html/2608.17336#A2.p12.1),[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Chenet al\.\(2024\)S\. Chen, Z\. Liu, Z\. Wu, C\. Zheng, P\. Cong, Z\. Jiang, Y\. Wu, L\. Su, and T\. YangInt\-flashattention: enabling flash attention for int8 quantization\.arXiv preprint arXiv:2409\.16997\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p3.1),[§2](https://arxiv.org/html/2608.17336#S2.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p3.1)\.
- Childet al\.\(2019\)R\. Child, S\. Gray, A\. Radford, and I\. SutskeverGenerating long sequences with sparse transformers\.External Links:1904\.10509,[Link](https://arxiv.org/abs/1904.10509)Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p16.1),[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Choquetteet al\.\(2021\)J\. Choquette, W\. Gandhi, O\. Giroux, N\. Stam, and R\. KrashinskyNvidia a100 tensor core gpu: performance and innovation\.IEEE Micro41\(2\),pp\. 29–35\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p3.1)\.
- Cohanet al\.\(2018\)A\. Cohan, F\. Dernoncourt, D\. S\. Kim, T\. Bui, S\. Kim, W\. Chang, and N\. GoharianA discourse\-aware attention model for abstractive summarization of long documents\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 615–621\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p1.1)\.
- Correiaet al\.\(2019\)G\. M\. Correia, V\. Niculae, and A\. F\. T\. MartinsAdaptively sparse transformers\.External Links:1909\.00015,[Link](https://arxiv.org/abs/1909.00015)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Daiet al\.\(2019a\)Z\. Dai, Z\. Yang, Y\. Yang, J\. G\. Carbonell, Q\. Le, and R\. SalakhutdinovTransformer\-xl: attentive language models beyond a fixed\-length context\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 2978–2988\.Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p11.1)\.
- Daiet al\.\(2019b\)Z\. Dai, Z\. Yang, Y\. Yang, J\. Carbonell, Q\. V\. Le, and R\. SalakhutdinovTransformer\-xl: attentive language models beyond a fixed\-length context\.External Links:1901\.02860,[Link](https://arxiv.org/abs/1901.02860)Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p14.1),[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashAttention: fast and memory\-efficient exact attention with io\-awareness\.External Links:2205\.14135,[Link](https://arxiv.org/abs/2205.14135)Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p3.1)\.
- Dao \(2023\)T\. DaoFlashattention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p3.1),[§2](https://arxiv.org/html/2608.17336#S2.p3.1)\.
- Degeet al\.\(2025\)P\. Dege, Q\. Luo, R\. Mao, and C\. KongFlashMLA\-etap: efficient transpose attention pipeline for accelerating mla inference on nvidia h20 gpus\.InInternational Conference on Neural Information Processing,pp\. 3–17\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p3.1)\.
- Dettmerset al\.\(2022\)T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. ZettlemoyerGpt3\. int8 \(\): 8\-bit matrix multiplication for transformers at scale\.Advances in neural information processing systems35,pp\. 30318–30332\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Jianget al\.\(2024\)H\. Jiang, Y\. Li, C\. Zhang, Q\. Wu, X\. Luo, S\. Ahn, Z\. Han, A\. H\. Abdi, D\. Li, C\. Lin,et al\.Minference 1\.0: accelerating pre\-filling for long\-context llms via dynamic sparse attention\.Advances in Neural Information Processing Systems37,pp\. 52481–52515\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p3.1)\.
- Kanget al\.\(2024\)H\. Kang, S\. Bharadwaj, J\. Hensman, T\. Krishna, V\. Ruhle, and S\. RajmohanTurboAttention: efficient attention approximation for high throughputs llms\.External Links:2412\.08585,[Link](https://arxiv.org/abs/2412.08585)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Kitaevet al\.\(2020\)N\. Kitaev, Ł\. Kaiser, and A\. LevskayaReformer: the efficient transformer\.External Links:2001\.04451,[Link](https://arxiv.org/abs/2001.04451)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Kluskaet al\.\(2024\)P\. Kluska, A\. Castelló, F\. Scheidegger, A\. C\. I\. Malossi, and E\. S\. Quintana\-OrtíQattn: efficient gpu kernels for mixed\-precision vision transformers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 3648–3657\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Laiet al\.\(2025\)X\. Lai, J\. Lu, Y\. Luo, Y\. Ma, and X\. ZhouFlexprefill: a context\-aware sparse attention mechanism for efficient long\-sequence inference\.arXiv preprint arXiv:2502\.20766\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p3.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p1.1)\.
- Liet al\.\(2023\)D\. Li, R\. Shao, A\. Xie, Y\. Sheng, L\. Zheng, J\. Gonzalez, I\. Stoica, X\. Ma, and H\. ZhangHow long can context length of open\-source llms truly promise?\.InNeurIPS 2023 Workshop on Instruction Tuning and Instruction Following,Cited by:[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p2.1)\.
- Presset al\.\(2021\)O\. Press, N\. A\. Smith, and M\. LewisTrain short, test long: attention with linear biases enables input length extrapolation\.arXiv preprint arXiv:2108\.12409\.Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p14.1)\.
- Royet al\.\(2020\)A\. Roy, M\. Saffar, A\. Vaswani, and D\. GrangierEfficient content\-based sparse attention with routing transformers\.External Links:2003\.05997,[Link](https://arxiv.org/abs/2003.05997)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Saxenaet al\.\(2024\)U\. Saxena, S\. Sharify, K\. Roy, and X\. WangResq: mixed\-precision quantization of large language models with low\-rank residuals\.arXiv preprint arXiv:2412\.14363\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Shahet al\.\(2024\)J\. Shah, G\. Bikshandi, Y\. Zhang, V\. Thakkar, P\. Ramani, and T\. DaoFlashattention\-3: fast and accurate attention with asynchrony and low\-precision\.Advances in Neural Information Processing Systems37,pp\. 68658–68685\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p3.1),[§2](https://arxiv.org/html/2608.17336#S2.p3.1)\.
- Sharratt \(2026\)J\. SharrattThriftAttention: selective mixed precision for long\-context fp4 attention\.arXiv preprint arXiv:2605\.23081\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Srinivasa Kumar \(2025\)P\. K\. Srinivasa KumarEvaluating full int8 quantization and inference techniques for causal language model\.Master’s Thesis,University of Twente\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Teamet al\.\(2024\)Q\. Teamet al\.Qwen2 technical report\.arXiv preprint arXiv:2407\.106712\(3\)\.Cited by:[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p1.1)\.
- Team \(2024\)Q\. TeamQwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p1.1)\.
- Titoet al\.\(2023\)R\. Tito, D\. Karatzas, and E\. ValvenyHierarchical multimodal transformers for multipage docvqa\.Pattern Recognition144,pp\. 109834\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p1.1)\.
- Van Baalenet al\.\(2023\)M\. Van Baalen, A\. Kuzmin, S\. S\. Nair, Y\. Ren, E\. Mahurin, C\. Patel, S\. Subramanian, S\. Lee, M\. Nagel, J\. Soriaga,et al\.FP8 versus int8 for efficient deep learning inference\.arXiv preprint arXiv:2303\.17951\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Wanget al\.\(2021\)S\. Wang, L\. Zhou, Z\. Gan, Y\. Chen, Y\. Fang, S\. Sun, Y\. Cheng, and J\. LiuCluster\-former: clustering\-based sparse transformer for long\-range dependency encoding\.External Links:2009\.06097,[Link](https://arxiv.org/abs/2009.06097)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Yaoet al\.\(2022\)Z\. Yao, R\. Aminabadi, M\. Zhang, X\. Wu, C\. Li, and Y\. Z\. HeEfficient and affordable post\-training quantization for large\-scale transformers, 2022\.URL https://arxiv\. org/abs/2206\.01861\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.
- Yuanet al\.\(2026\)J\. Yuan, C\. Shinn, K\. Xu, J\. Cui, G\. Klimiashvili, G\. Xiao, P\. Zheng, B\. Li, Y\. Zhou, Z\. Ye,et al\.BLASST: dynamic blocked attention sparsity via softmax thresholding\.Proceedings of Machine Learning and Systems8,pp\. 843–859\.Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Yuanet al\.\(2024\)T\. Yuan, X\. Ning, D\. Zhou, Z\. Yang, S\. Li, M\. Zhuang, Z\. Tan, Z\. Yao, D\. Lin, B\. Li,et al\.Lv\-eval: a balanced long\-context benchmark with 5 length levels up to 256k\.arXiv preprint arXiv:2402\.05136\.Cited by:[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p2.1)\.
- Zaheeret al\.\(2021\)M\. Zaheer, G\. Guruganesh, A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontanon, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. AhmedBig bird: transformers for longer sequences\.External Links:2007\.14062,[Link](https://arxiv.org/abs/2007.14062)Cited by:[Appendix B](https://arxiv.org/html/2608.17336#A2.p12.1),[Appendix B](https://arxiv.org/html/2608.17336#A2.p13.1),[Appendix B](https://arxiv.org/html/2608.17336#A2.p15.1),[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Zhanget al\.\(2025\)J\. Zhang, J\. Wei, H\. Huang, P\. Zhang, J\. Zhu, and J\. ChenSageAttention: accurate 8\-bit attention for plug\-and\-play inference acceleration\.External Links:2410\.02367,[Link](https://arxiv.org/abs/2410.02367)Cited by:[§1](https://arxiv.org/html/2608.17336#S1.p2.1),[§2](https://arxiv.org/html/2608.17336#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.17336#S5.SS1.p3.1)\.
- Zhanget al\.\(2023\)X\. Zhang, Z\. Lv, and Q\. YangAdaptive attention for sparse\-based long\-sequence transformer\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 8602–8610\.External Links:[Link](https://aclanthology.org/2023.findings-acl.546/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.546)Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p4.1)\.
- Zhaoet al\.\(2024\)Y\. Zhao, C\. Lin, K\. Zhu, Z\. Ye, L\. Chen, S\. Zheng, L\. Ceze, A\. Krishnamurthy, T\. Chen, and B\. KasikciAtom: low\-bit quantization for efficient and accurate llm serving\.Proceedings of Machine Learning and Systems6,pp\. 196–209\.Cited by:[§2](https://arxiv.org/html/2608.17336#S2.p2.1)\.

## Appendix ANotations

For clarity and ease of reference, we group notation in Table[4](https://arxiv.org/html/2608.17336#A1.T4)by functional role rather than by order of appearance\. This organization reflects the structure of the attention computation and kernel design: core attention definitions, tiling and indexing used by IO\-aware kernels, online\-softmax state variables, blockwise quantization primitives, head mappings for grouped\-query attention, and tile\-group routing metadata\. Grouping symbols in this way allows readers to quickly locate related quantities when following the kernel execution flow and routing logic described in Sections[3](https://arxiv.org/html/2608.17336#S3)and[4](https://arxiv.org/html/2608.17336#S4)\.

CategorySymbolDescriptionAttentionQ,K,VQ,K,VQuery, key, and value matrices:Q∈ℝLq×dQ\\in\\mathbb\{R\}^\{L\_\{q\}\\times d\},K∈ℝLk×dK\\in\\mathbb\{R\}^\{L\_\{k\}\\times d\},V∈ℝLk×dV\\in\\mathbb\{R\}^\{L\_\{k\}\\times d\}Lq,LkL\_\{q\},L\_\{k\}Query and key lengths;Lq=Lk=LL\_\{q\}=L\_\{k\}=Lfor self\-attentionddQuery, key, and value head dimensionSSScaled score matrix:S=Q​K⊤/d∈ℝLq×LkS=QK^\{\\top\}/\\sqrt\{d\}\\in\\mathbb\{R\}^\{L\_\{q\}\\times L\_\{k\}\}PPAttention weights:P=softmax⁡\(S\)∈ℝLq×LkP=\\mathrm\{softmax\}\(S\)\\in\\mathbb\{R\}^\{L\_\{q\}\\times L\_\{k\}\}OOAttention output:O=P​V∈ℝLq×dO=PV\\in\\mathbb\{R\}^\{L\_\{q\}\\times d\}Tilingbq,bk​vb\_\{q\},b\_\{kv\}Query and key/value tile sizesBLOCKM,BLOCKN\\mathrm\{BLOCK\}\_\{M\},\\mathrm\{BLOCK\}\_\{N\}Hardware\-aligned compute\-tile sizes;BLOCKM≡bq\\mathrm\{BLOCK\}\_\{M\}\\equiv b\_\{q\},BLOCKN≡bk​v\\mathrm\{BLOCK\}\_\{N\}\\equiv b\_\{kv\}TmT\_\{m\}Query\-tile count:Tm=⌈Lq/BLOCKM⌉T\_\{m\}=\\lceil L\_\{q\}/\\mathrm\{BLOCK\}\_\{M\}\\rceilTnT\_\{n\}Key/value\-tile count:Tn=⌈Lk/BLOCKN⌉T\_\{n\}=\\lceil L\_\{k\}/\\mathrm\{BLOCK\}\_\{N\}\\rceili,ji,jQuery and key token indicesm,nm,nQuery and key/value tile indicesOnline Softmaxm~mn\\tilde\{m\}\_\{m\}^\{\\,n\}Row\-wise maximum for query tilemmafter tilennz~mn\\tilde\{z\}\_\{m\}^\{\\,n\}Row\-wise normalizer for query tilemmafter tilennO~mn\\tilde\{O\}\_\{m\}^\{\\,n\}Unnormalized output accumulator after tilennQuantizationψ⁡\(⋅\)\\psi\(\\cdot\)Blockwise quantization operatorA^,B^\\hat\{A\},\\hat\{B\}Low\-precision forms of operandsAAandBBδA,δB\\delta\_\{A\},\\delta\_\{B\}Block scales forA^\\hat\{A\}andB^\\hat\{B\}Q8,K8Q\_\{8\},K\_\{8\}INT8 query and key representationsHeadsHq,HkH\_\{q\},H\_\{k\}Numbers of query and KV headshq,hkh\_\{q\},h\_\{k\}Query\-head and mapped KV\-head indicesRouting PolicyBLOCKNmask\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}Routing\-group width along the key dimensionggKey\-tile grouping factor:BLOCKNmask=g​BLOCKN\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}=g\\,\\mathrm\{BLOCK\}\_\{N\}TmaskT\_\{\\text\{mask\}\}Groups per query\-tile row:Tmask=⌈Lk/BLOCKNmask⌉≤64T\_\{\\text\{mask\}\}=\\lceil L\_\{k\}/\\mathrm\{BLOCK\}\_\{N\}^\{\\text\{mask\}\}\\rceil\\leq 64gjg\_\{j\}Group index for key tilenn:gj=⌊n/g⌋g\_\{j\}=\\lfloor n/g\\rfloorRhk,m,gjR\_\{h\_\{k\},m,g\_\{j\}\}Binary route: 1 for INT8 and 0 for FP16bhk,mb\_\{h\_\{k\},m\}Packed 64\-bit routing wordρINT8\\rho\_\{\\mathrm\{INT8\}\}Fraction of legal groups routed to INT8Table 4:Notation for attention, tiling, online softmax, quantization, head mapping, and routing\.
## Appendix BPrecision Layouts

Figure[6](https://arxiv.org/html/2608.17336#A2.F6)visualizes attention maps across layers and heads for LLaMA 3\.2 3B on the Multi\-News dataset\. The maps provide qualitative motivation for structured tile\-group precision allocation by showing that attention values are distributed non\-uniformly across the query–key plane\. The evaluated routing templates are statically constructed and do not use benchmark outputs for policy selection\. InTileMix, these layouts do not remove interactions; they only determine whether each legal tile group is routed to FP16 or INT8\. These observations motivate a precision\-routing view of attention acceleration\.

![Refer to caption](https://arxiv.org/html/2608.17336v1/attention_patterns.png)Figure 6:Visualization of attention layouts across layers and heads\. The figure compares the attention values from a LLaMA 3\.2 3B Instruct model on the Multi\-News dataset\.TileMixsupports block\-level routing layouts that specify whether each legal tile group is computed through the FP16 or INT8 score path\. These layouts reuse spatial structures studied in prior sparse\-attention work as precision\-routing templates\. The resulting policies preserve dense token connectivity while assigning selected spatial regions to FP16 and the remaining legal regions to INT8\.

Policy Construction and Sharing\.For each sequence length and block geometry, a layout is instantiated as a two\-dimensional routing templateR¯m,gj∈\{0,1\},\\overline\{R\}\_\{m,g\_\{j\}\}\\in\\\{0,1\\\},wheremmis the query\-tile row andgjg\_\{j\}is the key\-tile\-group index\. The convention is

R¯m,gj=\{1,INT8 score path,0,FP16 score path\.\\overline\{R\}\_\{m,g\_\{j\}\}=\\begin\{cases\}1,&\\text\{INT8 score path\},\\\\ 0,&\\text\{FP16 score path\}\.\\end\{cases\}Causal and sequence\-boundary masks determine whether token interactions are legal independently of this precision decision\.

The same template is reused across transformer layers and batch examples and is broadcast across KV heads:

Rhk,m,gj=R¯m,gj,hk∈\{0,…,Hk−1\}\.R\_\{h\_\{k\},m,g\_\{j\}\}=\\overline\{R\}\_\{m,g\_\{j\}\},\\qquad h\_\{k\}\\in\\\{0,\\ldots,H\_\{k\}\-1\\\}\.Query heads mapped to the same KV head and read the same packed routing word\. The layer and batch indices are omitted fromRRbecause the evaluated routing values are shared along these dimensions\.

To keep one routing word sufficient at long sequence lengths, the routing\-group width is selected as a multiple of the key compute\-tile width:b64=⌈Lk64⌉,BLOCKNmask=BLOCKN​⌈max⁡\(b64,BLOCKN\)BLOCKN⌉\.b\_\{64\}=\\left\\lceil\\frac\{L\_\{k\}\}\{64\}\\right\\rceil,\\;\\mathrm\{BLOCK\}\_\{N\}^\{\\mathrm\{mask\}\}=\\mathrm\{BLOCK\}\_\{N\}\\left\\lceil\\frac\{\\max\\\!\\left\(b\_\{64\},\\mathrm\{BLOCK\}\_\{N\}\\right\)\}\{\\mathrm\{BLOCK\}\_\{N\}\}\\right\\rceil\.Consequently,g=BLOCKNmaskBLOCKN,Tmask=⌈LkBLOCKNmask⌉≤64,g=\\frac\{\\mathrm\{BLOCK\}\_\{N\}^\{\\mathrm\{mask\}\}\}\{\\mathrm\{BLOCK\}\_\{N\}\},\\;T\_\{\\mathrm\{mask\}\}=\\left\\lceil\\frac\{L\_\{k\}\}\{\\mathrm\{BLOCK\}\_\{N\}^\{\\mathrm\{mask\}\}\}\\right\\rceil\\leq 64,and key compute tilennuses routing\-group indexgj=⌊ng⌋\.g\_\{j\}=\\left\\lfloor\\frac\{n\}\{g\}\\right\\rfloor\.

For each query\-tile row, the decisions are packed intobhk,m=∑gj=0Tmask−1Rhk,m,gj​2gj\.b\_\{h\_\{k\},m\}=\\sum\_\{g\_\{j\}=0\}^\{T\_\{\\mathrm\{mask\}\}\-1\}R\_\{h\_\{k\},m,g\_\{j\}\}\\,2^\{g\_\{j\}\}\.The template is regenerated when the sequence length or selected block geometry changes and is cached otherwise\.

Coverage Accounting\.Let𝒢=\{\(hk,m,gj\):ℐm×𝒥gj\}\\mathcal\{G\}=\\left\\\{\(h\_\{k\},m,g\_\{j\}\):\\mathcal\{I\}\_\{m\}\\times\\mathcal\{J\}\_\{g\_\{j\}\}\\right\\\}denote the set of legal routing groups\. The realized INT8 coverage is

ρ^INT8=∑\(hk,m,gj\)∈𝒢Rhk,m,gj\|𝒢\|\.\\widehat\{\\rho\}\_\{\\mathrm\{INT8\}\}=\\frac\{\\sum\_\{\(h\_\{k\},m,g\_\{j\}\)\\in\\mathcal\{G\}\}R\_\{h\_\{k\},m,g\_\{j\}\}\}\{\|\\mathcal\{G\}\|\}\.The reported 25%, 50%, and 75% settings denote legal score\-tile\-group INT8 coverage\. They do not denote the fraction of total attention FLOPs executed in INT8\. For each layout, its parameters are selected at tile\-group granularity to match the requested coverage as closely as possible\. Minor differences between requested and realized coverage may arise from discrete routing groups and sequence boundaries\.

For query\-tile rowmm, let𝒢m=\{gj:\(hk,m,gj\)∈𝒢\}\\mathcal\{G\}\_\{m\}=\\left\\\{g\_\{j\}:\(h\_\{k\},m,g\_\{j\}\)\\in\\mathcal\{G\}\\right\\\}denote its legal key\-tile groups, and letℱm=\{gj∈𝒢m:Rhk,m,gj=0\}\\mathcal\{F\}\_\{m\}=\\left\\\{g\_\{j\}\\in\\mathcal\{G\}\_\{m\}:R\_\{h\_\{k\},m,g\_\{j\}\}=0\\right\\\}denote its FP16\-routed groups\. The layouts below differ in the spatial construction ofℱm\\mathcal\{F\}\_\{m\}\.

One \(All INT8\)\.All legal tile groups are routed to INT8:ℱmOne=∅,Rhk,m,gj=1\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{One\}\}=\\varnothing,\\;R\_\{h\_\{k\},m,g\_\{j\}\}=1\.This pattern corresponds to uniform INT8 score computation and serves as the matched low\-precision reference using the sameTileMixexecution substrate\.

Zero \(All FP16\)\.All legal tile groups are routed to FP16:ℱmZero=𝒢m,Rhk,m,gj=0\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Zero\}\}=\\mathcal\{G\}\_\{m\},\\;R\_\{h\_\{k\},m,g\_\{j\}\}=0\.This pattern corresponds to all\-FP16 of theTileMixkernel and serves as its full\-precision reference\.

Band \(Local Template\)\.[3](https://arxiv.org/html/2608.17336#bib.bib13);[9](https://arxiv.org/html/2608.17336#bib.bib31)Band retains a contiguous FP16 region around the query–key diagonal\. Letγ⁡\(m\)=⌊m​BLOCKMBLOCKNmask⌋\\gamma\(m\)=\\left\\lfloor\\frac\{m\\,\\mathrm\{BLOCK\}\_\{M\}\}\{\\mathrm\{BLOCK\}\_\{N\}^\{\\mathrm\{mask\}\}\}\\right\\rfloordenote the key\-group index aligned with the beginning of query\-tile rowmm\. For a half\-widthww, the FP16 set isℱmBand​\(w\)=\{gj∈𝒢m:\|gj−γ⁡\(m\)\|≤w\}\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Band\}\}\(w\)=\\left\\\{g\_\{j\}\\in\\mathcal\{G\}\_\{m\}:\|g\_\{j\}\-\\gamma\(m\)\|\\leq w\\right\\\}\.Groups inside the band are routed to FP16, while the remaining legal groups are routed to INT8\. The widthwwis selected to match the requested INT8 coverage at tile\-group granularity\. This template transfers the locality bias of sliding\-window attention into precision allocation while preserving the complete attention\.

Global\.[37](https://arxiv.org/html/2608.17336#bib.bib15);[3](https://arxiv.org/html/2608.17336#bib.bib13)Let𝒢global⊆\{0,…,Tmask−1\}\\mathcal\{G\}\_\{\\mathrm\{global\}\}\\subseteq\\\{0,\\ldots,T\_\{\\mathrm\{mask\}\}\-1\\\}denote a fixed set of key\-tile groups associated with designated global positions\. Global usesℱmGlobal=𝒢global∩𝒢m\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Global\}\}=\\mathcal\{G\}\_\{\\mathrm\{global\}\}\\cap\\mathcal\{G\}\_\{m\}\.The selected global groups remain in FP16 for every query\-tile row, while the remaining legal groups are routed to INT8\. The number of global groups is selected to match the requested INT8 coverage\. This template follows the global\-token bias used in BigBird\- and Longformer\-style mechanisms\.

Row\-Random\.[37](https://arxiv.org/html/2608.17336#bib.bib15)Each query\-tile row independently selects a fixed\-size subset of legal key\-tile groups uniformly at random for FP16 routing:ℱmRowRand∼UniformSubset⁡\(𝒢m,km\),\\mathcal\{F\}\_\{m\}^\{\\mathrm\{RowRand\}\}\\sim\\operatorname\{UniformSubset\}\\left\(\\mathcal\{G\}\_\{m\},k\_\{m\}\\right\),wherekm=round⁡\(\(1−ρINT8\)​\|𝒢m\|\)\.k\_\{m\}=\\operatorname\{round\}\\left\(\(1\-\\rho\_\{\\mathrm\{INT8\}\}\)\|\\mathcal\{G\}\_\{m\}\|\\right\)\.Sampling is performed without replacement using a fixed random seed, and the resulting routing template is reused throughout evaluation\. The remaining groups are routed to INT8\. This template follows the random component of BigBird and distributes FP16 across nearby and long\-range positions\.

Aligned Sparse\.[10](https://arxiv.org/html/2608.17336#bib.bib20);[22](https://arxiv.org/html/2608.17336#bib.bib30)Aligned Sparse assigns a right\-aligned region of legal key\-tile groups to FP16\. Leta⁡\(m\)=min⁡\(\|𝒢m\|,⌈α\+β​m⌉\)a\(m\)=\\min\\left\(\|\\mathcal\{G\}\_\{m\}\|,\\left\\lceil\\alpha\+\\beta m\\right\\rceil\\right\)denote the number of FP16 groups assigned to query\-tile rowmm, whereα\\alphacontrols the initial width andβ\\betacontrols its growth\. Ifgmmax=max⁡𝒢m,g\_\{m\}^\{\\max\}=\\max\\mathcal\{G\}\_\{m\},thenℱmAlign=\{gj∈𝒢m:gj≥gmmax−a⁡\(m\)\+1\}\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Align\}\}=\\left\\\{g\_\{j\}\\in\\mathcal\{G\}\_\{m\}:g\_\{j\}\\geq g\_\{m\}^\{\\max\}\-a\(m\)\+1\\right\\\}\.The number of FP16\-routed groups therefore increases monotonically with the query index, producing a right\-aligned template with an expanding FP16 region\. The parametersα\\alphaandβ\\betaare selected to match the requested INT8 coverage\.

BigBird\.[37](https://arxiv.org/html/2608.17336#bib.bib15)This template combines \(i\) local\-band routing, \(ii\) global\-token routing, and \(iii\) row\-wise random routing\. Its FP16 set isℱmBigBird=ℱmBand∪ℱmGlobal∪ℱmRowRand\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{BigBird\}\}=\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Band\}\}\\cup\\mathcal\{F\}\_\{m\}^\{\\mathrm\{Global\}\}\\cup\\mathcal\{F\}\_\{m\}^\{\\mathrm\{RowRand\}\}\.A legal tile group selected by any component remains on the FP16 path, while the remaining legal groups use INT8\. The local, global, and random component sizes are selected jointly to match the requested tile\-group INT8 coverage\.TileMixtherefore interprets the corresponding spatial topology as a composite precision\-routing template rather than a structural attention mask\.

Sparse Transformer \(SpTrans\)\.[5](https://arxiv.org/html/2608.17336#bib.bib14)SpTrans combines \(i\) the local stride containing the current query position and \(ii\) the finalccpositions of each preceding stride\. For stride lengthssand query positionii, letu⁡\(i\)=⌊is⌋u\(i\)=\\left\\lfloor\\frac\{i\}\{s\}\\right\\rfloordenote its current stride\. The selected FP16 key positions are𝒦iSpTrans=\[u⁡\(i\)​s,i\]∪⋃r=0u⁡\(i\)−1\[max⁡\(r​s,\(r\+1\)​s−c\),\(r\+1\)​s−1\]\.\\mathcal\{K\}\_\{i\}^\{\\mathrm\{SpTrans\}\}=\\left\[u\(i\)s,\\,i\\right\]\\cup\\bigcup\_\{r=0\}^\{u\(i\)\-1\}\\left\[\\max\\\!\\left\(rs,\(r\+1\)s\-c\\right\),\\,\(r\+1\)s\-1\\right\]\.For query\-tile rowmm, the corresponding FP16 routing set isℱmSpTrans=\{gj∈𝒢m:∃i∈ℐm​such that​𝒥gj∩𝒦iSpTrans≠∅\}\.\\mathcal\{F\}\_\{m\}^\{\\mathrm\{SpTrans\}\}=\\left\\\{g\_\{j\}\\in\\mathcal\{G\}\_\{m\}:\\exists i\\in\\mathcal\{I\}\_\{m\}\\text\{ such that \}\\mathcal\{J\}\_\{g\_\{j\}\}\\cap\\mathcal\{K\}\_\{i\}^\{\\mathrm\{SpTrans\}\}\\neq\\varnothing\\right\\\}\.These selected tile groups are routed to FP16, while the remaining legal tile groups are routed to INT8\. The stride lengthssand tail widthccdetermine the spatial structure, and the resulting tile\-group allocation is configured for the requested INT8 coverage\.

Layouts Reported in the Main Evaluation\.The main LV\-Eval table reports BigBird and SpTrans as representative structured layouts with complementary spatial organizations and empirical quality–efficiency behavior\. Complete results for Band, Global, Row\-Random, Aligned Sparse, BigBird, and SpTrans are reported in this appendix across models, context lengths, and INT8 coverage levels\.

The supported routing layouts do not introduce sparsity into the attention computation\. Instead,TileMixtranslates established spatial structures into hardware\-aligned, block\-level precision\-routing decisions\. This decouples*which interactions are legal*from*which arithmetic path computes them*, enabling training\-free heterogeneous\-precision execution with dense token connectivity\.

## Appendix CQuantization Configuration

This section specifies the numerical configuration of the primary score\-routing implementation\. Blockwise INT8 quantization is applied toQQandKK, whileVVand theP​VPVcomputation remain in FP16\. Both score paths enter one common floating\-point score domain before updating the shared online\-softmax state\.

Blockwise INT8 Quantization\.For a valid floating\-point blockX\(r\)X^\{\(r\)\},TileMixuses signed symmetric INT8 quantization with absmax scaleδX\(r\)=maxx∈X\(r\)⁡\|x\|127\.\\delta\_\{X\}^\{\(r\)\}=\\frac\{\\max\_\{x\\in X^\{\(r\)\}\}\|x\|\}\{127\}\.The quantized representation is obtained by nearest\-integer conversion:X^\(r\)=round⁡\(X\(r\)δX\(r\)\),X^\(r\)∈\{−127,…,127\}\.\\widehat\{X\}^\{\(r\)\}=\\operatorname\{round\}\\left\(\\frac\{X^\{\(r\)\}\}\{\\delta\_\{X\}^\{\(r\)\}\}\\right\),\\;\\widehat\{X\}^\{\(r\)\}\\in\\\{\-127,\\ldots,127\\\}\.Because the scale is determined by the block maximum, finite normalized values lie within the stated signed range\.

The quantization kernel converts block values to FP32 when computing the scale and quantized representation\. The resulting scale tensors are stored in the operand dtype by the production wrapper; the FP16 experiments therefore use FP16 scale tensors\.

Quantization Granularity\.The evaluated score\-routing path usesBLKQ=128,BLKK=64\.\\mathrm\{BLK\}\_\{Q\}=128,\\qquad\\mathrm\{BLK\}\_\{K\}=64\.Each query block has shape128×d,128\\times d,and each key block has shape64×d,64\\times d,with one scale per block and attention head\.

For variable\-length batches, the scale tensors have logical shapes\[∑b⌈Lq,b128⌉,Hq\]and\[∑b⌈Lk,b64⌉,Hk\]\.\\left\[\\sum\_\{b\}\\left\\lceil\\frac\{L\_\{q,b\}\}\{128\}\\right\\rceil,H\_\{q\}\\right\]\\quad\\text\{and\}\\quad\\left\[\\sum\_\{b\}\\left\\lceil\\frac\{L\_\{k,b\}\}\{64\}\\right\\rceil,H\_\{k\}\\right\]\.Prefix\-sum metadata maps each sequence and token block to its scale entry\.

INT8 Score Computation\.For query blockrrand key blockss, an INT8\-routed score tile first computesCr,sINT32=Q8\(r\)​\(K8\(s\)\)⊤C\_\{r,s\}^\{\\mathrm\{INT32\}\}=Q\_\{8\}^\{\(r\)\}\\left\(K\_\{8\}^\{\(s\)\}\\right\)^\{\\top\}using INT8 Tensor Core multiplication with INT32 accumulation\. The accumulator is converted to floating point and rescaled asSr,sINT8=δQ\(r\)​δK\(s\)​Cr,sINT32d\.S\_\{r,s\}^\{\\mathrm\{INT8\}\}=\\frac\{\\delta\_\{Q\}^\{\(r\)\}\\delta\_\{K\}^\{\(s\)\}C\_\{r,s\}^\{\\mathrm\{INT32\}\}\}\{\\sqrt\{d\}\}\.An FP16\-routed tile computesSr,sFP16=Q\(r\)​\(K\(s\)\)⊤d\.S\_\{r,s\}^\{\\mathrm\{FP16\}\}=\\frac\{Q^\{\(r\)\}\\left\(K^\{\(s\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d\}\}\.Both paths enter FP16 score domain before exponentiation and update shared online\-softmax state\.

The running maximum, normalizer, and output accumulator are stored in FP16\. Tile maxima, exponentiation, and row reductions use FP32 intermediates before their results are incorporated into the shared FP16 state\. The resulting probability tile and theP​VPVupdate use FP16\.

Quantize\-Once Execution\.The implementation\-level prefill path generatesQ8Q\_\{8\},K8K\_\{8\}, and their compact block scales once before the attention launch\. These operands are reused by all INT8\-routed score\-tile groups, while FP16\-routed groups read the original FP16QQandKKtensors\. The all\-FP16 configuration skips this quantization step\. Reported implementation\-level timings include operand quantization, scale preparation, routing, rescaling, memory staging, and fused attention\.

Additional Execution Interfaces\.The implementation also provides an on\-the\-fly operand\-generation mode and an INT8 key/value cache interface\. The cache interface storesK8K\_\{8\}andV8V\_\{8\}together with compact per\-block scales\. During decode, cachedK8K\_\{8\}participates in the routed INT8 score path; cachedV8V\_\{8\}is converted to FP16 on chip, with its block scale folded into the probability tile before theP​VPVdot product\. This interface is separate from the primary prefill configuration, which quantizes the routedQ/KQ/Kscore operands and retains FP16VVfor everyP​VPVupdate\.

## Appendix DStorage and Execution Layout

Building on the numerical configuration in Appendix[C](https://arxiv.org/html/2608.17336#A3),TileMixkeeps the dense FP16Q,K,VQ,K,Vtensors in the standard contiguous HBM layout and stores routing metadata separately as one packed 64\-bit word for each\(hk,m\)\(h\_\{k\},m\), wherehkh\_\{k\}is a KV\-head index andmmis a query\-tile row\. The fused inner loop reads this word to select the FP16 or INT8 score path for each legal tile group\. Tensor addressing, token connectivity, and tile legality retain the dense FlashAttention\-style execution structure\.

TileMixsupports two operand\-preparation modes, as illustrated in Figure[7](https://arxiv.org/html/2608.17336#A4.F7)\. Thequantize\-oncemode is used by the reported quality and implementation\-level efficiency evaluations\. It generates temporary INT8Q/KQ/Ktensors and their block scales once per attention call and reuses them across all INT8\-routed groups\. The original FP16Q/KQ/Ktensors remain available to FP16\-routed groups, whileVVandP​VPVcomputation remain FP16 throughout the primary evaluated path\.

TileMix quantize\-once TileMix fused on\-the\-fly\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\[HBM\] FP16 Q/K/V \[HBM\] FP16 Q/K/V\| \|\|\-\- pre\-quantization pass \|\-\- no separate pre\-quantization pass\| \[L2\] stream BLK x d tiles into quant kernel \|\| \[On\-SM\] compute scales, quantize to INT8 \|\| \[HBM\] write temporary QINT8/KINT8 \+ scales \|v v\[HBM\] Q/K/V \+ QINT8/KINT8 \+ scales \+ bitmask \[HBM\] Q/K/V \+ bitmask\| \|v v\[L2 / On\-SM\] \[L2 / On\-SM\]\| \|\| load Q or QINT8 once per query row \| load FP Q once per query row\| stream K/V tiles \| stream FP K/V tiles\| online\-softmax in registers \| compute scales on chip\| \| quantize only INT8\-dispatched groups\| \| online\-softmax in registers\| \|v v\[Tensor Cores\] \[Tensor Cores\]\| \|\| INT8 QK: qint8 @ kint8 \-\> int32 MMA \-\> rescale \| INT8 QK: qint8 @ kint8 \-\> int32 MMA \-\> rescale\| FP QK: q @ k \-\> fp16/fp32 MMA \| FP QK: q @ k \-\> fp16/fp32 MMA\| FP PV: p @ v \| FP PV: p @ v\| FP PV: p @ v \| FP PV: p @ v\| \|v v\[HBM\] Out \[HBM\] Out

Figure 7:Storage and execution layout of the twoTileMixoperand\-preparation modes for the primary prefill path\. Quantize\-once materializes temporary INT8Q/KQ/Ktensors and their scales once per attention call and reuses them across INT8\-routed score\-tile groups\. Fused on\-the\-fly generates INT8Q/KQ/Kfragments inside the kernel only for INT8\-routed groups\. Both modes retainVVand theP​VPVcomputation in FP16\. The separate INT8 key/value cache interface is described in the surrounding text\.Thefused on\-the\-flymode is an additional implementation mode that loads the FP16 operands and generates INT8 fragments only for groups selected by the routing map\. This mode removes the attention\-call\-localQ8/K8Q\_\{8\}/K\_\{8\}scratch tensors from HBM and performs scale computation and quantization on\-chip\. Both modes use the same packed routing policy, routed score computation, and shared online\-softmax update\.

The kernel interface additionally supports INT8 key/value caches with per\-block scale metadata\. In this cache path, cachedK8K\_\{8\}supplies the routed INT8 score computation, while cachedV8V\_\{8\}is converted to FP16 on chip and used in the floating\-pointP​VPVupdate\. This cache interface is described separately from the primary prefill path illustrated in Figure[7](https://arxiv.org/html/2608.17336#A4.F7)\. The primary evaluation uses routed FP16/INT8Q​K⊤QK^\{\\top\}score computation with FP16V/P​VV/PV, as specified in Appendix[C](https://arxiv.org/html/2608.17336#A3)\.

For clarity, Table[5](https://arxiv.org/html/2608.17336#A4.T5)reports attention\-call\-local memory and data\-movement accounting under one representative configuration:Hq=32H\_\{q\}=32,Hk=8H\_\{k\}=8,Lq=Lk=8192L\_\{q\}=L\_\{k\}=8192,d=128d=128,BLKQ=128\\mathrm\{BLK\}\_\{Q\}=128, andBLKK=64\\mathrm\{BLK\}\_\{K\}=64\. Here,HqH\_\{q\}andHkH\_\{k\}are the query\-head and KV\-head counts,LqL\_\{q\}andLkL\_\{k\}are the query and key sequence lengths,ddis the per\-head dimension, andBLKQ\\mathrm\{BLK\}\_\{Q\}andBLKK\\mathrm\{BLK\}\_\{K\}are the query and key quantization\-block sizes\.

The accounting distinguishes: \(i\) tensors resident in HBM during one attention call; \(ii\) global/L2 operand reads for one routed QK or optional PV tile event; and \(iii\) the simultaneously live on\-SM working state for one FP16 tile event\. The INT8 QK column corresponds to the primary score\-routing path, while theV8V\_\{8\}\-backed PV column separately characterizes value\-cache traffic in the INT8 key/value cache interface\. Resident HBM for theTileMixquantize\-once row reports the primary score\-only prefill configuration\. These quantities describe operator\-local storage and traffic, distinct from full\-model peak GPU memory\.

RouteResidentHBMINT8 QK tileglobal/L2 readV8V\_\{8\}\-backed PV tileglobal/L2 readFP QK tileon\-SM live setFP PV tileon\-SM live setFP QK/PVTensor Core accumulatorFlashAttention96 MB——128 KB128 KB64 KBTileMixquantize\-once136\.01 MiB16\.25 KB16\.25 KB96 KB96 KB32 KBTileMixfused on\-the\-fly96 MB32 KB32 KB96 KB96 KB32 KB

Table 5:Memory and data\-movement accounting for FlashAttention andTileMixunderHq=32H\_\{q\}=32,Hk=8H\_\{k\}=8,Lq=Lk=8192L\_\{q\}=L\_\{k\}=8192,d=128d=128,BLKQ=128\\mathrm\{BLK\}\_\{Q\}=128, andBLKK=64\\mathrm\{BLK\}\_\{K\}=64\. Resident HBM counts persistent tensors and route\-specific temporary scratch tensors during the attention call\. Global/L2 reads are reported for one INT8\-routed QK or PV tile event\. The on\-SM live set reports the simultaneously live on\-chip working state for the corresponding FP16\-routed tile event\.Under this representative configuration, quantize\-once uses 136\.01 MiB of resident HBM because it materializes reusable INT8 scratch tensors and their scales for the duration of the attention call\. This preparation reduces the operand read for one INT8\-routed QK tile event from 32 KB in fused on\-the\-fly execution to 16\.25 KB\. The same traffic relation applies toV8V\_\{8\}\-backed PV reads in the optional INT8 key/value cache interface\.

Fused on\-the\-fly retains the same 96 MB resident\-HBM footprint as the FlashAttention configuration in this accounting and generates INT8 operands from FP16 tiles on chip\. For FP16\-routed QK and PV events, theTileMixtile path uses a 96 KB on\-SM live set and a 32 KB Tensor Core accumulator, compared with 128 KB and 64 KB, respectively, under the reported FlashAttention accounting\. These measurements characterize the local storage and data\-movement behavior of the attention implementations; full\-model OOM behavior is considered separately from this operator\-level accounting\.

## Appendix EModel Performance

Long\-context Retrieval\.

Figure[8](https://arxiv.org/html/2608.17336#A5.F8)and Figure[9](https://arxiv.org/html/2608.17336#A5.F9)report LongEval results for Qwen 2\.5 7B and Qwen 2 7B across extended context lengths\. Vicuna 7B, however, supports a maximum context length of 16k tokens\. To avoid sparsely populated plots and to present results at all supported lengths clearly, we report Vicuna 7B performance in tabular form \(Table[6](https://arxiv.org/html/2608.17336#A5.T6)\) instead of figures\. The table follows the same evaluation protocol, precision layouts, and INT8 ratios as the figure\-based results, allowing direct comparison across models within their respective context limits\.

Figure 8:Line\-level retrieval accuracy on the LongEval benchmark forQwen 2\.5 7Bunder different tile\-group routing layouts, evaluated across prompt lengths from 3\.1k to 38\.7k tokens\. Line plots \(left y\-axis\) report exact\-match retrieval accuracy, while bar plots \(right y\-axis,Δ\\Deltaaccuracy vs\. FP16\) show differences relative to the FP16 attention baseline\. Each panel corresponds to a routing layout with 25%, 50%, or 75% of tile groups routed to INT8\. Colors indicate routing settings:greendenotes the original FP16 attention baseline,reddenotesOne\(all legal score\-tile groups routed to INT8\), andbluedenotes mixed configurations with partially INT8\-routed tile groups\.Figure 9:Line\-level retrieval accuracy on the LongEval benchmark forQwen 2 7Bunder different precision policy layouts, evaluated across prompt lengths from 3\.1k to 38\.7k tokens\.Line\|Len\|FP16\|INT8\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|255075\|255075\|255075\|255075\|255075\|255075200 \|4776\.33 \|0\.99 \|0\.97 \|0\.990\.991\.00 \|0\.990\.991\.00 \|1\.001\.001\.00 \|0\.991\.001\.00 \|1\.001\.001\.00 \|1\.001\.001\.00300 \|7081\.42 \|1\.00 \|0\.99 \|1\.001\.001\.00 \|1\.001\.001\.00 \|1\.001\.001\.00 \|0\.991\.001\.00 \|1\.001\.001\.00 \|1\.001\.001\.00400 \|9384\.62 \|0\.99 \|0\.98 \|0\.980\.990\.99 \|0\.980\.990\.99 \|0\.990\.990\.99 \|0\.980\.990\.99 \|0\.990\.990\.99 \|0\.990\.990\.99500 \|11691\.21 \|0\.85 \|0\.76 \|0\.820\.800\.78 \|0\.820\.810\.82 \|0\.820\.820\.83 \|0\.820\.820\.82 \|0\.820\.820\.82 \|0\.810\.820\.82600 \|13991\.58 \|0\.73 \|0\.63 \|0\.690\.650\.63 \|0\.690\.690\.68 \|0\.690\.700\.70 \|0\.690\.690\.70 \|0\.690\.690\.69 \|0\.690\.690\.67700 \|16294\.20 \|0\.52 \|0\.44 \|0\.480\.470\.45 \|0\.490\.480\.46 \|0\.490\.490\.50 \|0\.490\.490\.50 \|0\.490\.490\.48 \|0\.490\.480\.47

Table 6:Line\-level retrieval accuracy on the LongEval benchmark forVicuna 7Bunder different precision layouts inspired by sparse\-attention patterns\. The task requires the model to scan long input sequences and exactly retrieve the content associated with a queried line identifier\. Accuracy is measured as exact\-match retrieval rate, reported across increasing input lengths\. Results compare full FP16 attention, pure INT8 attention, and mixed\-precision configurations with varying precision layouts and INT8 ratios, illustrating how different precision\-allocation patterns affect robustness under low\-precision execution\.Together, Figure[8](https://arxiv.org/html/2608.17336#A5.F8), Figure[9](https://arxiv.org/html/2608.17336#A5.F9), and Table[6](https://arxiv.org/html/2608.17336#A5.T6)show similar qualitative trends across Qwen 2\.5 7B, Qwen 2 7B, and Vicuna 7B\. Across models, uniform INT8 attention \(One\) often degrades retrieval accuracy as prompt length increases, showing that a single low\-precision arithmetic path can be too aggressive for long\-context retrieval\. In contrast, mixed\-routing configurations generally recover accuracy by keeping selected tile groups on the FP16 path while routing the remaining groups to INT8\.

Across the three model families, mixed\-routing configurations recover substantial retrieval quality relative toOne, while sensitivity to layout and INT8 coverage varies with the model and prompt length\.bigbird,row\_rand, andsptransprovide the most consistent recovery across the evaluated models, particularly at higher INT8 coverage\.

Overall, these results suggest that tile\-group precision routing provides a useful precision\-budget mechanism across model families, while the best routing layout and INT8 ratio remain model\- and task\-dependent\.

Long\-context Question Answering\.Table[7](https://arxiv.org/html/2608.17336#A5.T7)isolates the comparison against sparse long\-context and INT8 attention baselines on Qwen 2 7B\. Across the reported LV\-Eval subsets, SpTrans mixed\-precision configurations close much of the gap to FP16 while outperforming or matching sparse baselines in most settings\. This supports the central distinction ofTileMix: it recovers long\-context quality through precision routing while preserving dense token connectivity\.

Tables[8](https://arxiv.org/html/2608.17336#A5.T8),[9](https://arxiv.org/html/2608.17336#A5.T9), and[10](https://arxiv.org/html/2608.17336#A5.T10)report LV\-Eval accuracy forLLaMA 3\.2 3B,Qwen 2 7B, andQwen 2\.5 7Bacross context lengths from 16k to 64k under FP16 attention \(Baseline\),One\(100% INT8 tiles\), and mixed\-precision configurations with different precision layouts and INT8 ratios \(25/50/75%\)\.

Dataset\|Len\|One\|SpTrans25\|SpTrans50\|SpTrans75\|MInferenceFlexPrefillSageAttn1\|16k \|31\.60 \|36\.00 \|34\.30 \|33\.40 \|29\.0026\.1332\.60\|32k \|14\.70 \|20\.00 \|19\.00 \|18\.50 \|15\.0011\.8916\.90\|64k \|6\.60 \|9\.70 \|8\.80 \|8\.20 \|6\.408\.989\.102\|16k \|13\.90 \|16\.70 \|16\.20 \|15\.10 \|14\.1015\.7414\.20\|32k \|11\.40 \|15\.10 \|14\.60 \|14\.00 \|11\.5015\.2413\.50\|64k \|8\.70 \|13\.20 \|12\.50 \|12\.10 \|9\.5012\.6510\.807\|16k \|12\.20 \|17\.10 \|16\.50 \|16\.30 \|14\.9017\.0915\.80\|32k \|9\.30 \|15\.30 \|14\.90 \|14\.40 \|12\.6016\.0813\.80\|64k \|5\.80 \|11\.70 \|11\.10 \|10\.80 \|9\.0010\.2911\.408\|16k \|12\.10 \|16\.90 \|16\.30 \|16\.70 \|15\.2013\.6515\.10\|32k \|8\.40 \|12\.60 \|12\.20 \|12\.00 \|10\.8012\.2610\.80\|64k \|7\.30 \|10\.80 \|10\.10 \|9\.90 \|8\.109\.469\.509\|16k \|25\.70 \|33\.80 \|33\.10 \|33\.30 \|28\.5029\.0731\.40\|32k \|15\.70 \|22\.90 \|22\.60 \|22\.10 \|18\.1020\.7121\.00\|64k \|9\.00 \|15\.90 \|15\.50 \|15\.00 \|11\.4014\.5313\.7010\|16k \|19\.30 \|25\.10 \|24\.60 \|24\.00 \|21\.2023\.8922\.60\|32k \|14\.20 \|19\.30 \|18\.60 \|18\.20 \|15\.3017\.6717\.90\|64k \|14\.00 \|17\.50 \|16\.90 \|17\.00 \|12\.8015\.9515\.9011\|16k \|25\.30 \|34\.30 \|33\.50 \|32\.30 \|28\.7022\.7431\.40\|32k \|20\.10 \|25\.30 \|24\.60 \|24\.50 \|19\.3015\.9823\.50\|64k \|6\.90 \|18\.40 \|17\.20 \|15\.30 \|13\.0015\.2216\.10Table 7:LV\-Eval long\-context question answering results on Qwen2\-7B\. We compareTileMixpure INT8 \(One\),TileMixSpTrans mixed\-precision variants, and long\-context attention baselines\. SpTrans25, SpTrans50, and SpTrans75 denote configurations with 25%, 50%, and 75% INT8\-routed tile groups\. Datasets are denoted by the same indices defined in the experimental setup\.

Dataset\|Len\|FP16\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751\|16k \|32\.04 \|28\.78 \|32\.9132\.7732\.78 \|32\.5230\.7630\.86 \|31\.2729\.6429\.75 \|31\.7530\.1829\.87 \|31\.8930\.9430\.64 \|31\.7532\.1830\.25\|32k \|15\.08 \|11\.62 \|15\.7513\.3812\.42 \|12\.2813\.1813\.61 \|15\.2214\.0213\.55 \|15\.5614\.2413\.86 \|15\.2613\.2312\.61 \|15\.5614\.7714\.37\|64k \|7\.75 \|5\.42 \|7\.816\.865\.86 \|7\.815\.845\.81 \|8\.016\.475\.72 \|8\.017\.036\.25 \|8\.016\.326\.08 \|8\.017\.446\.832\|16k \|18\.49 \|15\.62 \|18\.9517\.0816\.83 \|19\.1616\.5717\.23 \|18\.6116\.9816\.62 \|18\.6417\.6216\.88 \|18\.3916\.5816\.56 \|18\.6418\.0816\.78\|32k \|15\.12 \|12\.02 \|15\.1814\.3413\.16 \|15\.4813\.5913\.09 \|15\.1413\.5313\.28 \|15\.2714\.7813\.59 \|15\.5814\.1513\.09 \|15\.2714\.6914\.18\|64k \|11\.84 \|7\.79 \|12\.1510\.009\.12 \|12\.3010\.1710\.10 \|12\.0410\.2810\.03 \|12\.0610\.9610\.28 \|12\.1910\.3910\.13 \|12\.0611\.3411\.043\|16k \|6\.72 \|4\.45 \|6\.465\.624\.93 \|6\.5224\.374\.37 \|6\.296\.974\.57 \|6\.295\.314\.50 \|6\.574\.724\.61 \|21\.0420\.5321\.65\|32k \|3\.78 \|1\.95 \|3\.773\.072\.92 \|3\.641\.351\.49 \|2\.351\.811\.20 \|3\.352\.371\.70 \|5\.962\.042\.65 \|3\.352\.782\.68\|64k \|3\.38 \|1\.42 \|3\.432\.341\.54 \|3\.401\.421\.41 \|3\.241\.701\.37 \|3\.242\.261\.45 \|3\.401\.511\.43 \|3\.242\.672\.474\|16k \|6\.00 \|2\.27 \|7\.506\.005\.50 \|5\.504\.504\.20 \|6\.004\.464\.49 \|6\.005\.024\.21 \|6\.004\.765\.02 \|6\.005\.435\.04\|32k \|18\.00 \|13\.98 \|18\.5017\.1014\.20 \|18\.5016\.5017\.00 \|18\.5016\.4615\.99 \|18\.0017\.0216\.71 \|18\.5016\.2616\.02 \|18\.0017\.4317\.54\|64k \|14\.50 \|10\.07 \|15\.0014\.2012\.50 \|14\.0012\.0012\.00 \|15\.0013\.4612\.49 \|15\.0014\.0212\.71 \|15\.5013\.2613\.02 \|15\.0014\.4314\.045\|16k \|17\.51 \|12\.86 \|17\.2117\.2116\.19 \|17\.6915\.5115\.25 \|17\.6615\.1714\.84 \|17\.6615\.8715\.87 \|17\.6615\.9216\.49 \|17\.6617\.0916\.88\|32k \|10\.80 \|8\.38 \|11\.0110\.159\.01 \|10\.858\.798\.49 \|10\.619\.079\.00 \|11\.0110\.038\.75 \|10\.809\.289\.21 \|11\.0110\.4410\.06\|64k \|6\.82 \|5\.32 \|7\.706\.395\.70 \|6\.895\.975\.39 \|7\.636\.095\.62 \|7\.636\.655\.84 \|7\.705\.964\.91 \|7\.636\.255\.976\|16k \|12\.33 \|8\.51 \|12\.4311\.779\.79 \|11\.669\.706\.91 \|12\.3810\.4010\.30 \|12\.3111\.3310\.52 \|12\.0210\.2710\.53 \|12\.3111\.7411\.03\|32k \|6\.73 \|3\.22 \|6\.125\.763\.88 \|7\.044\.844\.90 \|6\.525\.184\.45 \|7\.086\.104\.73 \|6\.665\.194\.56 \|7\.086\.515\.95\|64k \|1\.84 \|0\.74 \|1\.991\.871\.02 \|0\.840\.860\.86 \|1\.911\.351\.13 \|1\.871\.591\.05 \|1\.991\.891\.24 \|1\.971\.371\.227\|16k \|21\.75 \|15\.69 \|21\.8822\.3620\.47 \|27\.8419\.8619\.03 \|21\.9620\.8120\.09 \|22\.0721\.0820\.23 \|22\.4120\.1020\.08 \|22\.0721\.1621\.05\|32k \|19\.89 \|12\.11 \|19\.1617\.3514\.23 \|19\.3917\.3616\.63 \|20\.1718\.4117\.79 \|19\.9119\.0717\.99 \|19\.8817\.9617\.47 \|19\.9119\.3718\.56\|64k \|14\.18 \|6\.93 \|14\.1313\.9810\.91 \|14\.1312\.4511\.25 \|14\.0912\.3711\.97 \|14\.2313\.3112\.23 \|14\.1412\.2711\.79 \|14\.2313\.6613\.348\|16k \|18\.24 \|12\.72 \|18\.1616\.1213\.13 \|18\.5016\.6515\.53 \|17\.5716\.4515\.96 \|17\.5316\.9715\.75 \|18\.1216\.2716\.28 \|17\.5316\.9917\.46\|32k \|14\.25 \|9\.12 \|13\.9812\.1311\.43 \|14\.4412\.6411\.40 \|13\.7312\.7312\.29 \|13\.6313\.0312\.21 \|14\.1912\.3512\.17 \|13\.6313\.1813\.00\|64k \|11\.08 \|7\.54 \|11\.069\.118\.28 \|11\.229\.238\.07 \|11\.059\.609\.04 \|11\.3110\.099\.56 \|11\.089\.188\.73 \|11\.3110\.6510\.389\|16k \|45\.68 \|34\.70 \|46\.0243\.1939\.69 \|45\.9143\.6042\.61 \|45\.1343\.6643\.86 \|45\.7744\.7943\.52 \|45\.5243\.9344\.04 \|45\.7744\.6044\.88\|32k \|26\.84 \|18\.17 \|25\.7623\.2021\.69 \|26\.0824\.3623\.41 \|26\.4724\.9423\.78 \|26\.0525\.4924\.71 \|26\.9025\.0524\.58 \|26\.0525\.9425\.11\|64k \|16\.36 \|9\.19 \|15\.6014\.5811\.96 \|16\.3014\.2112\.45 \|15\.7214\.1814\.04 \|16\.0314\.8514\.24 \|16\.1414\.2613\.78 \|16\.0315\.2514\.7110\|16k \|28\.88 \|22\.37 \|28\.2626\.2125\.80 \|28\.8027\.0625\.79 \|28\.9527\.5726\.33 \|29\.1028\.1327\.15 \|28\.1726\.3126\.42 \|29\.1028\.3127\.23\|32k \|18\.33 \|13\.66 \|18\.0315\.9915\.98 \|17\.8416\.2015\.13 \|18\.1716\.7016\.88 \|18\.2617\.3516\.44 \|18\.2516\.5516\.51 \|18\.2617\.6517\.41\|64k \|15\.75 \|12\.81 \|16\.0815\.9114\.98 \|16\.1814\.1313\.18 \|15\.9314\.2213\.88 \|15\.9414\.9414\.13 \|15\.8514\.7714\.41 \|15\.9415\.3215\.3911\|16k \|22\.44 \|16\.97 \|23\.5622\.6420\.75 \|22\.3621\.0319\.89 \|22\.4420\.2420\.76 \|20\.2921\.4626\.06 \|21\.7420\.3421\.62 \|22\.9422\.3721\.74\|32k \|14\.85 \|12\.02 \|15\.5315\.2814\.49 \|15\.2813\.2814\.18 \|16\.1114\.5713\.34 \|15\.6114\.6313\.56 \|15\.8113\.5713\.54 \|15\.6115\.0415\.06\|64k \|9\.26 \|3\.41 \|8\.367\.365\.57 \|8\.366\.365\.36 \|8\.928\.387\.91 \|8\.927\.947\.13 \|8\.927\.126\.38 \|8\.928\.357\.40

Table 8:LV\-Eval long\-context question answering forLLaMA 3\.2 3B\. Cells are color\-coded by comparison toOneandBaseline:redindicates performance belowOne,blueindicates performance betweenOneand theBaseline, andgreenindicates performance above theBaseline\.

Across datasets and lengths,Onegenerally underperforms the FP16 baseline, indicating that uniform INT8 attention can be too coarse for long\-context QA\. Mixed\-routing configurations usually narrow this gap, especially at moderate INT8 ratios\. The effect of increasing INT8 coverage is layout dependent:align\_sparse,band, andglobaloften benefit from more conservative INT8 ratios, whereasbigbird,row\_rand, andsptransoften tolerate higher INT8 coverage\. A consistent layout–task interaction appears on the 16kfactrecall\_ensetting\. Across SpTrans25, SpTrans50, and SpTrans75, the scores reach 21\.04/20\.53/21\.65 on LLaMA 3\.2 3B, 50\.6/49\.7/52\.0 on Qwen 2 7B, and 31\.4/30\.7/32\.2 on Qwen 2\.5 7B, compared with the corresponding FP16 scores of 6\.72, 16\.39, and 10\.22\. The recurrence across model families and all three routing ratios shows a cross\-model consistent layout–task interaction at this setting, while the corresponding SpTrans results at 32k and 64k return to the usual quality range\. Overall, these tables support the use of tile\-group routing as a controllable precision\-budget mechanism for long\-context QA, while the best routing layout depends on the model, task, and context length\.

## Appendix FEfficiency

Tables[11](https://arxiv.org/html/2608.17336#A6.T11),[12](https://arxiv.org/html/2608.17336#A6.T12),[13](https://arxiv.org/html/2608.17336#A6.T13), and[14](https://arxiv.org/html/2608.17336#A6.T14)report implementation\-level prefill efficiency across LLaMA 3\.2 3B, Qwen 2\.5 7B, Qwen 2 7B, and Vicuna 7B using throughput \(Thpt, K tokens/s\) and TOPS from 1k to 8k where executable, with model\-dependent maximum sequence lengths \(batch size 8; 3 warmup iterations; 5 measurement iterations\)\. We compare the standard Torch implementation, FlashAttention as an IO\-aware FP16 baseline,One\(uniform all legal score\-tile groups routed to INT8\) as an efficiency\-oriented reference, andTileMixmixed\-routing variants under different routing layouts and INT8 ratios \(25/50/75%\)\.

Dataset\|Len\|FP16\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751\|16k \|34\.97 \|31\.6 \|36\.035\.635\.5 \|35\.133\.633\.8 \|34\.432\.732\.9 \|35\.033\.633\.1 \|35\.834\.734\.1 \|36\.034\.333\.4\|32k \|19\.14 \|14\.7 \|20\.017\.015\.9 \|15\.616\.717\.0 \|19\.417\.817\.3 \|19\.818\.117\.6 \|19\.517\.016\.4 \|20\.019\.018\.5\|64k \|9\.34 \|6\.6 \|9\.68\.47\.3 \|9\.57\.27\.1 \|9\.77\.87\.1 \|9\.68\.57\.6 \|9\.87\.77\.4 \|9\.78\.88\.22\|16k \|16\.39 \|13\.9 \|16\.815\.415\.1 \|17\.115\.015\.5 \|16\.415\.014\.8 \|16\.615\.715\.0 \|16\.414\.914\.9 \|16\.716\.215\.1\|32k \|14\.42 \|11\.4 \|14\.313\.412\.6 \|14\.612\.912\.4 \|14\.513\.012\.6 \|14\.614\.012\.9 \|14\.813\.412\.7 \|14\.614\.013\.5\|64k \|12\.97 \|8\.7 \|13\.311\.010\.3 \|13\.411\.211\.0 \|13\.211\.410\.9 \|13\.312\.011\.2 \|13\.511\.611\.2 \|13\.212\.512\.13\|16k \|16\.39 \|10\.7 \|15\.813\.912\.2 \|15\.95\.910\.9 \|15\.416\.611\.0 \|15\.413\.211\.4 \|16\.111\.511\.2 \|50\.649\.752\.0\|32k \|10\.12 \|5\.2 \|9\.98\.07\.6 \|9\.63\.53\.9 \|6\.24\.83\.1 \|9\.26\.54\.7 \|15\.75\.46\.8 \|9\.17\.47\.1\|64k \|5\.33 \|2\.2 \|5\.43\.92\.6 \|5\.42\.42\.4 \|5\.22\.72\.3 \|5\.23\.52\.4 \|5\.52\.52\.4 \|5\.34\.34\.05\|16k \|14\.21 \|10\.6 \|14\.214\.113\.2 \|14\.412\.812\.5 \|14\.412\.412\.1 \|14\.412\.912\.8 \|14\.312\.713\.5 \|14\.514\.013\.8\|32k \|8\.16 \|6\.3 \|8\.37\.66\.7 \|8\.16\.66\.4 \|8\.06\.96\.8 \|8\.37\.66\.7 \|8\.27\.17\.0 \|8\.37\.87\.5\|64k \|6\.81 \|5\.3 \|7\.66\.35\.6 \|7\.46\.45\.7 \|7\.76\.66\.1 \|7\.76\.76\.0 \|7\.86\.15\.1 \|7\.76\.36\.06\|16k \|23\.28 \|16\.0 \|23\.322\.118\.6 \|21\.918\.213\.3 \|23\.219\.719\.4 \|23\.021\.219\.8 \|22\.619\.319\.9 \|23\.222\.121\.3\|32k \|13\.73 \|6\.6 \|13\.112\.38\.3 \|14\.49\.910\.0 \|13\.410\.79\.1 \|14\.612\.69\.7 \|13\.810\.79\.4 \|14\.613\.512\.3\|64k \|6\.91 \|2\.9 \|7\.47\.03\.8 \|3\.23\.33\.3 \|7\.25\.14\.3 \|7\.06\.03\.9 \|7\.47\.04\.7 \|7\.35\.24\.67\|16k \|16\.96 \|12\.2 \|17\.117\.315\.9 \|21\.115\.114\.6 \|17\.015\.915\.4 \|17\.216\.415\.8 \|17\.315\.615\.5 \|17\.116\.516\.3\|32k \|15\.09 \|9\.3 \|14\.713\.611\.1 \|14\.913\.312\.7 \|15\.414\.113\.5 \|15\.214\.613\.7 \|15\.113\.813\.4 \|15\.314\.914\.4\|64k \|11\.62 \|5\.8 \|11\.611\.28\.7 \|11\.510\.19\.2 \|11\.610\.29\.8 \|11\.610\.910\.2 \|11\.610\.19\.6 \|11\.711\.110\.88\|16k \|17\.36 \|12\.1 \|17\.215\.312\.5 \|17\.716\.014\.9 \|16\.815\.715\.2 \|16\.616\.115\.1 \|17\.215\.615\.7 \|16\.916\.316\.7\|32k \|13\.03 \|8\.4 \|12\.711\.010\.2 \|13\.311\.610\.5 \|12\.611\.611\.2 \|12\.411\.811\.1 \|13\.111\.511\.3 \|12\.612\.212\.0\|64k \|10\.73 \|7\.3 \|10\.89\.18\.2 \|11\.09\.38\.2 \|10\.79\.28\.7 \|10\.99\.89\.3 \|10\.79\.38\.9 \|10\.810\.19\.99\|16k \|33\.66 \|25\.7 \|33\.731\.729\.1 \|33\.832\.231\.5 \|33\.432\.232\.4 \|33\.733\.032\.1 \|33\.632\.432\.6 \|33\.833\.133\.3\|32k \|23\.30 \|15\.7 \|22\.620\.519\.0 \|22\.821\.320\.6 \|23\.221\.920\.9 \|22\.922\.421\.6 \|23\.622\.021\.5 \|22\.922\.622\.1\|64k \|15\.95 \|9\.0 \|16\.014\.812\.1 \|15\.913\.912\.5 \|15\.714\.214\.1 \|16\.014\.914\.3 \|16\.114\.413\.9 \|15\.915\.515\.010\|16k \|24\.88 \|19\.3 \|24\.422\.922\.5 \|25\.023\.322\.2 \|25\.223\.923\.0 \|25\.324\.523\.7 \|24\.723\.123\.3 \|25\.124\.624\.0\|32k \|19\.22 \|14\.2 \|18\.816\.716\.5 \|18\.817\.116\.1 \|19\.117\.617\.3 \|19\.318\.417\.5 \|19\.217\.517\.3 \|19\.318\.618\.2\|64k \|17\.20 \|14\.0 \|17\.717\.216\.3 \|17\.615\.414\.6 \|17\.415\.715\.4 \|17\.516\.415\.6 \|17\.316\.215\.9 \|17\.516\.917\.011\|16k \|33\.70 \|25\.3 \|35\.333\.931\.2 \|33\.531\.529\.8 \|33\.730\.331\.0 \|33\.932\.431\.2 \|33\.631\.032\.1 \|34\.333\.532\.3\|32k \|24\.82 \|20\.1 \|25\.625\.023\.7 \|25\.222\.023\.6 \|26\.223\.721\.8 \|25\.423\.822\.1 \|25\.922\.322\.2 \|25\.324\.624\.5\|64k \|18\.51 \|6\.9 \|17\.115\.311\.6 \|17\.113\.111\.1 \|18\.317\.216\.3 \|18\.316\.414\.9 \|18\.214\.613\.1 \|18\.417\.215\.3Table 9:LV\-Eval forQwen 2 7B\. Cells are color\-coded by comparison toOneandBaseline:redindicates performance belowOne,blueindicates performance betweenOneand theBaseline, andgreenindicates performance above theBaseline\.

Len\|Metric\|Torch\|Flash\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751k \|Thpt \|11\.14 \|17\.45 \|32\.27 \|29\.2232\.0233\.48 \|27\.0629\.3034\.56 \|26\.9231\.4832\.37 \|26\.7129\.5932\.03 \|28\.6632\.0934\.95 \|27\.6632\.1133\.50\|TOPS \|41\.70 \|65\.31 \|120\.80 \|109\.38119\.86125\.29 \|101\.31109\.71129\.42 \|100\.78117\.85121\.17 \|99\.98110\.75119\.92 \|107\.25120\.12130\.80 \|103\.50120\.20125\.382k \|Thpt \|7\.78 \|16\.48 \|32\.06 \|29\.5032\.4133\.84 \|26\.7728\.8934\.30 \|26\.5728\.8332\.71 \|26\.4529\.2931\.74 \|26\.3431\.3534\.88 \|27\.1131\.4633\.92\|TOPS \|30\.50 \|64\.64 \|125\.70 \|115\.70127\.08132\.67 \|104\.99113\.30134\.48 \|104\.23113\.05128\.31 \|103\.69114\.83124\.46 \|103\.28122\.91136\.77 \|106\.32123\.38132\.994k \|Thpt \|OOM\|14\.33 \|29\.80 \|27\.7130\.3531\.62 \|25\.6528\.5932\.81 \|24\.6926\.8427\.48 \|24\.5027\.1929\.35 \|26\.9730\.0632\.89 \|27\.1430\.5931\.80\|TOPS \|OOM\|61\.31 \|127\.48 \|118\.53129\.85135\.26 \|109\.76122\.32140\.39 \|105\.63114\.83117\.56 \|104\.82116\.35125\.56 \|115\.37128\.62140\.73 \|116\.09130\.85136\.038k \|Thpt \|OOM\|OOM\|27\.41 \|23\.1725\.3926\.44 \|23\.2325\.0328\.19 \|23\.0225\.1725\.84 \|22\.8425\.3927\.47 \|22\.7825\.3327\.74 \|22\.8125\.6926\.61\|TOPS \|OOM\|OOM\|136\.76 \|115\.64126\.72131\.96 \|115\.89124\.89140\.73 \|114\.92125\.58128\.90 \|113\.98126\.74137\.09 \|113\.72126\.35138\.42 \|113\.83128\.21132\.84Table 11:Throughput \(Thpt, K tokens/s\) and TOPS onLLaMA 3\.2 3B\-Instructacross sequence lengths\. All methods are evaluated on the same A100 40GB hardware with batch size 8, 3 warmup iterations, and 5 measurement iterations under the same model wrapper\.Onedenotes all legal score\-tile groups routed to INT8\. OOM entries are shown in gray and excluded from relative color comparisons\.

Len\|Metric\|Torch\|Flash\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751k \|Thpt \|6\.98 \|7\.98 \|19\.27 \|15\.1016\.1218\.06 \|14\.2816\.9817\.59 \|15\.0316\.4718\.18 \|14\.7416\.5317\.93 \|14\.9015\.8717\.96 \|15\.0916\.0517\.76\|TOPS \|55\.11 \|63\.01 \|152\.10 \|119\.20127\.24142\.63 \|112\.75134\.02138\.79 \|118\.71130\.01143\.58 \|116\.37130\.48141\.49 \|117\.63125\.32141\.87 \|119\.14126\.73140\.172k \|Thpt \|5\.21 \|7\.72 \|19\.13 \|14\.9515\.9317\.84 \|14\.1816\.8017\.41 \|14\.8816\.2317\.94 \|14\.6316\.3517\.76 \|14\.7115\.7117\.76 \|14\.8915\.8417\.52\|TOPS \|42\.23 \|62\.54 \|155\.00 \|121\.07129\.05144\.61 \|114\.88136\.17140\.99 \|120\.58131\.59145\.36 \|118\.51132\.50143\.85 \|119\.23127\.32143\.91 \|120\.61128\.32141\.984k \|Thpt \|OOM\|7\.09 \|18\.03 \|14\.0914\.9516\.74 \|13\.5015\.9816\.43 \|14\.1215\.3716\.84 \|13\.8915\.4316\.65 \|13\.8914\.7516\.65 \|14\.0314\.9716\.49\|TOPS \|OOM\|60\.41 \|153\.54 \|120\.09127\.37142\.70 \|115\.00136\.10139\.94 \|120\.31130\.89143\.35 \|118\.25131\.40141\.80 \|118\.37125\.64141\.89 \|119\.51127\.56140\.44

Table 12:Throughput \(Thpt, K tokens/s\) and TOPS onQwen 2\.5 7Bacross sequence lengths\.

Dataset\|Len\|FP16\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751\|16k \|33\.88 \|28\.9 \|34\.734\.434\.5 \|33\.932\.032\.1 \|33\.231\.131\.3 \|33\.732\.031\.7 \|34\.833\.733\.2 \|34\.933\.132\.2\|32k \|17\.61 \|13\.6 \|18\.315\.614\.6 \|14\.415\.315\.7 \|17\.816\.415\.9 \|18\.216\.616\.1 \|17\.815\.514\.9 \|18\.317\.416\.9\|64k \|9\.22 \|6\.5 \|9\.38\.17\.0 \|9\.37\.06\.9 \|9\.57\.66\.8 \|9\.48\.37\.4 \|9\.57\.47\.1 \|9\.48\.57\.92\|16k \|17\.37 \|14\.8 \|17\.816\.215\.9 \|18\.115\.816\.4 \|17\.415\.915\.6 \|17\.616\.715\.9 \|17\.415\.815\.7 \|17\.617\.216\.0\|32k \|15\.89 \|12\.6 \|15\.814\.813\.9 \|16\.014\.113\.6 \|15\.914\.213\.8 \|16\.015\.414\.2 \|16\.214\.713\.9 \|16\.015\.314\.8\|64k \|12\.12 \|8\.1 \|12\.410\.29\.5 \|12\.510\.410\.2 \|12\.310\.610\.1 \|12\.411\.210\.5 \|12\.610\.810\.4 \|12\.311\.611\.23\|16k \|10\.22 \|6\.7 \|9\.88\.67\.6 \|9\.93\.76\.8 \|9\.610\.46\.9 \|9\.68\.27\.0 \|10\.17\.26\.9 \|31\.430\.732\.2\|32k \|6\.03 \|3\.1 \|6\.04\.84\.6 \|5\.72\.12\.3 \|3\.72\.91\.9 \|5\.53\.92\.8 \|9\.43\.34\.1 \|5\.44\.44\.2\|64k \|3\.17 \|1\.3 \|3\.22\.31\.5 \|3\.21\.41\.4 \|3\.11\.61\.3 \|3\.12\.11\.4 \|3\.31\.51\.4 \|3\.22\.62\.44\|16k \|4\.64 \|1\.7 \|5\.84\.64\.3 \|4\.33\.63\.3 \|4\.63\.53\.5 \|4\.63\.93\.3 \|4\.73\.84\.0 \|4\.64\.13\.8\|32k \|3\.58 \|2\.8 \|3\.73\.42\.8 \|3\.73\.33\.4 \|3\.73\.33\.2 \|3\.63\.43\.3 \|3\.73\.33\.2 \|3\.63\.53\.6\|64k \|1\.74 \|1\.2 \|1\.81\.71\.5 \|1\.71\.51\.4 \|1\.81\.61\.5 \|1\.81\.61\.5 \|1\.91\.61\.5 \|1\.81\.71\.65\|16k \|19\.13 \|14\.2 \|18\.918\.817\.9 \|19\.217\.116\.7 \|19\.216\.716\.4 \|19\.217\.317\.1 \|19\.117\.018\.0 \|19\.318\.718\.2\|32k \|8\.72 \|6\.8 \|8\.98\.17\.3 \|8\.77\.16\.9 \|8\.67\.47\.3 \|8\.98\.07\.1 \|8\.87\.67\.5 \|8\.98\.38\.0\|64k \|8\.09 \|6\.3 \|9\.07\.56\.7 \|8\.87\.76\.9 \|9\.17\.87\.2 \|9\.17\.97\.1 \|9\.27\.26\.0 \|9\.07\.47\.16\|16k \|23\.19 \|16\.0 \|23\.222\.018\.5 \|21\.818\.113\.1 \|23\.119\.619\.3 \|22\.921\.119\.7 \|22\.519\.219\.8 \|23\.122\.021\.2\|32k \|16\.05 \|7\.7 \|15\.314\.49\.7 \|16\.911\.611\.8 \|15\.712\.510\.7 \|17\.114\.811\.4 \|16\.112\.511\.0 \|17\.115\.814\.4\|64k \|5\.58 \|2\.3 \|6\.05\.73\.1 \|2\.62\.72\.7 \|5\.84\.13\.4 \|5\.74\.93\.2 \|6\.05\.73\.8 \|5\.94\.23\.77\|16k \|21\.60 \|15\.5 \|21\.822\.120\.3 \|26\.819\.118\.5 \|21\.720\.419\.8 \|21\.920\.920\.1 \|22\.020\.220\.0 \|21\.821\.020\.9\|32k \|18\.85 \|11\.6 \|18\.316\.913\.8 \|18\.716\.716\.0 \|19\.417\.817\.0 \|19\.218\.417\.3 \|19\.117\.416\.9 \|19\.318\.818\.1\|64k \|12\.67 \|6\.3 \|12\.612\.19\.4 \|12\.511\.010\.0 \|12\.611\.110\.7 \|12\.711\.911\.1 \|12\.611\.010\.5 \|12\.712\.111\.88\|16k \|21\.27 \|14\.8 \|21\.118\.715\.2 \|21\.619\.518\.2 \|20\.519\.218\.6 \|20\.319\.718\.4 \|21\.119\.119\.2 \|20\.619\.820\.3\|32k \|16\.56 \|10\.7 \|16\.214\.113\.1 \|16\.914\.813\.4 \|16\.114\.914\.4 \|15\.915\.214\.3 \|16\.714\.614\.4 \|16\.115\.615\.4\|64k \|12\.90 \|8\.8 \|12\.910\.99\.8 \|13\.211\.19\.8 \|12\.811\.010\.4 \|13\.111\.811\.2 \|12\.911\.210\.7 \|13\.012\.211\.99\|16k \|41\.68 \|31\.8 \|41\.939\.336\.1 \|41\.839\.939\.0 \|41\.339\.940\.1 \|41\.740\.939\.7 \|41\.640\.140\.3 \|41\.841\.041\.2\|32k \|26\.25 \|17\.7 \|25\.423\.021\.3 \|25\.724\.023\.2 \|26\.124\.623\.5 \|25\.825\.224\.3 \|26\.624\.824\.2 \|25\.925\.625\.0\|64k \|14\.08 \|7\.9 \|14\.113\.010\.6 \|14\.012\.211\.0 \|13\.812\.512\.4 \|14\.113\.112\.6 \|14\.212\.712\.2 \|14\.013\.613\.110\|16k \|26\.81 \|20\.8 \|26\.324\.624\.2 \|26\.925\.123\.9 \|27\.125\.724\.7 \|27\.226\.325\.5 \|26\.624\.925\.1 \|27\.026\.425\.8\|32k \|18\.63 \|13\.8 \|18\.216\.216\.0 \|18\.216\.615\.6 \|18\.517\.116\.8 \|18\.717\.816\.9 \|18\.616\.916\.7 \|18\.718\.017\.6\|64k \|16\.42 \|13\.4 \|16\.916\.415\.5 \|16\.814\.713\.9 \|16\.615\.014\.7 \|16\.715\.714\.9 \|16\.515\.415\.1 \|16\.716\.116\.211\|16k \|33\.47 \|25\.1 \|35\.133\.731\.0 \|33\.331\.329\.6 \|33\.530\.130\.8 \|33\.732\.231\.0 \|33\.430\.831\.9 \|34\.133\.332\.1\|32k \|22\.36 \|18\.1 \|23\.122\.621\.4 \|22\.719\.821\.2 \|23\.621\.319\.6 \|22\.921\.419\.9 \|23\.320\.120\.0 \|22\.822\.122\.0\|64k \|14\.06 \|5\.2 \|12\.911\.58\.8 \|13\.09\.98\.4 \|13\.913\.012\.3 \|13\.912\.411\.2 \|13\.811\.09\.9 \|14\.013\.111\.6Table 10:LV\-Eval long\-context question answering forQwen 2\.5 7Bunder different precision layouts and INT8 ratios \(25/50/75%\)\. Results are reported for FP16 attention \(Baseline\),One\(all legal score\-tile groups routed to INT8\), and mixed\-precision configurations\. Cells are color\-coded by comparison toOneandBaseline:redindicates performance belowOne,blueindicates performance betweenOneand theBaseline, andgreenindicates performance above theBaseline\.

Len\|Metric\|Torch\|Flash\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751k \|Thpt \|7\.16 \|8\.21 \|19\.78 \|16\.2019\.8218\.04 \|16\.1217\.9317\.95 \|16\.2318\.0918\.07 \|16\.2018\.0420\.56 \|16\.2518\.0217\.98 \|16\.2520\.2620\.14\|TOPS \|56\.51 \|64\.81 \|156\.18 \|127\.94156\.43142\.38 \|127\.26141\.56141\.74 \|128\.19142\.87142\.69 \|127\.95142\.38162\.36 \|128\.23142\.28141\.99 \|128\.34159\.91159\.072k \|Thpt \|5\.35 \|7\.89 \|19\.56 \|15\.9919\.5617\.77 \|15\.8918\.5219\.29 \|16\.0417\.8617\.78 \|16\.0217\.7820\.29 \|16\.0117\.7717\.77 \|16\.0119\.8619\.76\|TOPS \|43\.34 \|63\.93 \|158\.52 \|129\.50158\.47143\.92 \|128\.76150\.08156\.33 \|129\.92144\.62144\.07 \|129\.79144\.14164\.38 \|129\.65144\.00144\.00 \|129\.76160\.98160\.154k \|Thpt \|OOM\|7\.22 \|18\.49 \|15\.1716\.8016\.80 \|15\.1717\.6218\.31 \|15\.2616\.9116\.80 \|15\.2516\.9119\.23 \|15\.2216\.8516\.82 \|15\.0718\.7618\.66\|TOPS \|OOM\|61\.48 \|157\.51 \|129\.19143\.03143\.12 \|129\.21150\.04156\.04 \|130\.00144\.09143\.19 \|129\.94144\.05163\.77 \|129\.58143\.53143\.21 \|128\.32159\.69158\.85Table 13:Throughput \(Thpt, K tokens/s\) and TOPS onQwen 2 7Bacross sequence lengths\.

Len\|Metric\|Torch\|Flash\|One\|AlignSparseBandBigBirdGlobalRowRandSpTrans\|\|\|\|\|255075\|255075\|255075\|255075\|255075\|2550751k \|Thpt \|6\.52 \|8\.40 \|18\.13 \|15\.5416\.7118\.02 \|15\.3816\.7118\.02 \|15\.3216\.4817\.88 \|15\.2316\.3818\.36 \|15\.0816\.5417\.83 \|15\.0116\.1517\.52\|TOPS \|44\.86 \|66\.30 \|124\.65 \|106\.85114\.97123\.98 \|105\.80114\.97123\.98 \|105\.39113\.33122\.93 \|104\.72112\.67126\.36 \|103\.68113\.78122\.71 \|103\.28111\.07120\.472k \|Thpt \|4\.60 \|8\.07 \|17\.35 \|14\.9116\.0517\.30 \|14\.7616\.0517\.30 \|14\.6615\.8817\.11 \|14\.6115\.7317\.64 \|14\.4615\.8917\.13 \|14\.3715\.5616\.77\|TOPS \|32\.88 \|65\.40 \|124\.02 \|106\.64114\.82123\.67 \|105\.58114\.82123\.67 \|104\.81113\.53122\.37 \|104\.51112\.53126\.05 \|103\.47113\.64122\.40 \|102\.72111\.26119\.934k \|Thpt \|OOM\|7\.39 \|15\.80 \|13\.7814\.8015\.92 \|13\.6414\.8015\.92 \|13\.5114\.6715\.66 \|13\.5114\.5116\.22 \|13\.3714\.6515\.75 \|13\.2414\.3815\.35\|TOPS \|OOM\|62\.89 \|121\.58 \|106\.03113\.93122\.50 \|104\.98113\.93122\.50 \|103\.93112\.87120\.48 \|103\.91111\.65124\.85 \|102\.88112\.76121\.24 \|101\.85110\.62118\.07

Table 14:Throughput \(Thpt, K tokens/s\) and TOPS onVicuna 7Bacross sequence lengths\.

Across models and lengths, Torch provides the lowest throughput and exhibits earlier OOM behavior, while FlashAttention serves as a strong baseline when it fits in memory\.Oneprovides the uniform INT8 reference, while mixed\-routing layouts form nearby empirical operating points whose ordering depends on complete\-pipeline execution\. Across models, higher INT8 coverage generally increases throughput and TOPS, and several mixed\-routing configurations match or exceedOne\. Differences among layouts reflect complete\-pipeline dispatch, rescaling, memory\-access, and scheduling behavior under different spatial routing arrangements; all layouts preserve the same dense legal connectivity\. Overall, the results suggest thatTileMixprovides a practical accuracy\-efficiency knob: higher INT8 ratios increase low\-precision execution, while more conservative ratios preserve more FP16\-routed tile groups for layouts or tasks that are more accuracy\-sensitive\.

Seq Len\|Torch Value\|Flash\|100% INT8\|75% INT8\|50% INT8\|25% INT81024 \|×1002\.85\\\!\\times\\\!10^\{0\}\|×10−21\.57\\\!\\times\\\!10^\{\-2\}\|×10−25\.40\\\!\\times\\\!10^\{\-2\}\|×10−23\.54\\\!\\times\\\!10^\{\-2\}\|×10−21\.68\\\!\\times\\\!10^\{\-2\}\|×10−31\.95\\\!\\times\\\!10^\{\-3\}2048 \|×1002\.82\\\!\\times\\\!10^\{0\}\|×10−21\.42\\\!\\times\\\!10^\{\-2\}\|×10−26\.49\\\!\\times\\\!10^\{\-2\}\|×10−24\.25\\\!\\times\\\!10^\{\-2\}\|×10−22\.20\\\!\\times\\\!10^\{\-2\}\|×10−31\.95\\\!\\times\\\!10^\{\-3\}4096 \|×1002\.98\\\!\\times\\\!10^\{0\}\|×10−21\.12\\\!\\times\\\!10^\{\-2\}\|×10−25\.86\\\!\\times\\\!10^\{\-2\}\|×10−24\.00\\\!\\times\\\!10^\{\-2\}\|×10−22\.00\\\!\\times\\\!10^\{\-2\}\|×10−31\.95\\\!\\times\\\!10^\{\-3\}8192 \|×1003\.46\\\!\\times\\\!10^\{0\}\|×10−21\.22\\\!\\times\\\!10^\{\-2\}\|×10−26\.25\\\!\\times\\\!10^\{\-2\}\|×10−24\.10\\\!\\times\\\!10^\{\-2\}\|×10−22\.10\\\!\\times\\\!10^\{\-2\}\|×10−31\.95\\\!\\times\\\!10^\{\-3\}Table 15:Single\-layer model on random inputs\. Torch Value reports the maximum absolute logit magnitude under the fixed Torch FP16 reference; all other columns report maximum absolute deviation from this reference\.

Seq Len\|Torch Value\|Flash\|100% INT8\|75% INT8\|50% INT8\|25% INT81024 \|×1002\.91\\\!\\times\\\!10^\{0\}\|×10−11\.48\\\!\\times\\\!10^\{\-1\}\|×10−12\.03\\\!\\times\\\!10^\{\-1\}\|×10−11\.34\\\!\\times\\\!10^\{\-1\}\|×10−26\.93\\\!\\times\\\!10^\{\-2\}\|×10−33\.91\\\!\\times\\\!10^\{\-3\}2048 \|×1002\.79\\\!\\times\\\!10^\{0\}\|×10−11\.40\\\!\\times\\\!10^\{\-1\}\|×10−11\.75\\\!\\times\\\!10^\{\-1\}\|×10−11\.23\\\!\\times\\\!10^\{\-1\}\|×10−26\.40\\\!\\times\\\!10^\{\-2\}\|×10−33\.17\\\!\\times\\\!10^\{\-3\}4096 \|×1003\.05\\\!\\times\\\!10^\{0\}\|×10−11\.47\\\!\\times\\\!10^\{\-1\}\|×10−12\.19\\\!\\times\\\!10^\{\-1\}\|×10−11\.46\\\!\\times\\\!10^\{\-1\}\|×10−27\.42\\\!\\times\\\!10^\{\-2\}\|×10−33\.42\\\!\\times\\\!10^\{\-3\}8192 \|×1003\.22\\\!\\times\\\!10^\{0\}\|×10−11\.52\\\!\\times\\\!10^\{\-1\}\|×10−12\.22\\\!\\times\\\!10^\{\-1\}\|×10−11\.48\\\!\\times\\\!10^\{\-1\}\|×10−27\.25\\\!\\times\\\!10^\{\-2\}\|×10−33\.91\\\!\\times\\\!10^\{\-3\}Table 16:Numerical behavior of a 12\-layer attention model on random inputs\. Torch Value reports the maximum absolute logit magnitude under the selected Torch FP16 reference; all other columns report maximum absolute deviation from this reference\.

Seq Len\|Torch Value\|FlashAttn\|100% INT8\|75% INT8\|50% INT8\|25% INT81024 \|×1003\.02\\\!\\times\\\!10^\{0\}\|×10−13\.55\\\!\\times\\\!10^\{\-1\}\|×10−13\.56\\\!\\times\\\!10^\{\-1\}\|×10−12\.22\\\!\\times\\\!10^\{\-1\}\|×10−11\.14\\\!\\times\\\!10^\{\-1\}\|×10−35\.86\\\!\\times\\\!10^\{\-3\}2048 \|×1002\.90\\\!\\times\\\!10^\{0\}\|×10−13\.18\\\!\\times\\\!10^\{\-1\}\|×10−14\.48\\\!\\times\\\!10^\{\-1\}\|×10−12\.96\\\!\\times\\\!10^\{\-1\}\|×10−11\.37\\\!\\times\\\!10^\{\-1\}\|×10−35\.00\\\!\\times\\\!10^\{\-3\}4096 \|×1003\.13\\\!\\times\\\!10^\{0\}\|×10−13\.79\\\!\\times\\\!10^\{\-1\}\|×10−13\.69\\\!\\times\\\!10^\{\-1\}\|×10−12\.51\\\!\\times\\\!10^\{\-1\}\|×10−11\.28\\\!\\times\\\!10^\{\-1\}\|×10−35\.86\\\!\\times\\\!10^\{\-3\}8192 \|×1003\.21\\\!\\times\\\!10^\{0\}\|×10−13\.31\\\!\\times\\\!10^\{\-1\}\|×10−13\.98\\\!\\times\\\!10^\{\-1\}\|×10−12\.60\\\!\\times\\\!10^\{\-1\}\|×10−11\.34\\\!\\times\\\!10^\{\-1\}\|×10−35\.86\\\!\\times\\\!10^\{\-3\}

Table 17:Numerical behavior of a 32\-layer attention model on random inputs\. Torch Value reports the maximum absolute logit magnitude under the selected Torch FP16 reference; all other columns report maximum absolute deviation from this reference\.ConfigMaxDiffMeanDiff\|ConfigMaxDiffMeanDiffZero \(0% INT8\)1\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}\|BigBird751\.96×10−41\.96\{\\times\}10^\{\-4\}8\.45×10−68\.45\{\\times\}10^\{\-6\}One \(100% INT8\)1\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}\|Global01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}AlignSparse01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}\|Global251\.96×10−41\.96\{\\times\}10^\{\-4\}6\.75×10−66\.75\{\\times\}10^\{\-6\}AlignSparse251\.96×10−41\.96\{\\times\}10^\{\-4\}7\.01×10−67\.01\{\\times\}10^\{\-6\}\|Global501\.96×10−41\.96\{\\times\}10^\{\-4\}8\.01×10−68\.01\{\\times\}10^\{\-6\}AlignSparse501\.96×10−41\.96\{\\times\}10^\{\-4\}7\.23×10−67\.23\{\\times\}10^\{\-6\}\|Global751\.96×10−41\.96\{\\times\}10^\{\-4\}8\.41×10−68\.41\{\\times\}10^\{\-6\}AlignSparse751\.96×10−41\.96\{\\times\}10^\{\-4\}7\.93×10−67\.93\{\\times\}10^\{\-6\}\|RowRand01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}Band01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}\|RowRand251\.96×10−41\.96\{\\times\}10^\{\-4\}6\.62×10−66\.62\{\\times\}10^\{\-6\}Band251\.96×10−41\.96\{\\times\}10^\{\-4\}6\.52×10−66\.52\{\\times\}10^\{\-6\}\|RowRand501\.96×10−41\.96\{\\times\}10^\{\-4\}8\.07×10−68\.07\{\\times\}10^\{\-6\}Band501\.96×10−41\.96\{\\times\}10^\{\-4\}7\.67×10−67\.67\{\\times\}10^\{\-6\}\|RowRand751\.96×10−41\.96\{\\times\}10^\{\-4\}8\.45×10−68\.45\{\\times\}10^\{\-6\}Band754\.84×10−44\.84\{\\times\}10^\{\-4\}8\.88×10−68\.88\{\\times\}10^\{\-6\}\|SpTrans01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}BigBird01\.96×10−41\.96\{\\times\}10^\{\-4\}5\.25×10−65\.25\{\\times\}10^\{\-6\}\|SpTrans251\.96×10−41\.96\{\\times\}10^\{\-4\}6\.33×10−66\.33\{\\times\}10^\{\-6\}BigBird251\.96×10−41\.96\{\\times\}10^\{\-4\}6\.62×10−66\.62\{\\times\}10^\{\-6\}\|SpTrans501\.96×10−41\.96\{\\times\}10^\{\-4\}6\.84×10−66\.84\{\\times\}10^\{\-6\}BigBird501\.96×10−41\.96\{\\times\}10^\{\-4\}8\.07×10−68\.07\{\\times\}10^\{\-6\}\|SpTrans754\.84×10−44\.84\{\\times\}10^\{\-4\}8\.68×10−68\.68\{\\times\}10^\{\-6\}
Table 18:Direct numerical difference betweenTileMixand FlashAttention under different precision layouts\. A nonzero gap exists even at 0% INT8 due to fused\-kernel implementation differences\.
ConfigMaxAbsDiffMeanAbsDiff\|ConfigMaxAbsDiffMeanAbsDiffOne \(fp16acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}6\.99×10−36\.99\{\\times\}10^\{\-3\}\|Global25 \(fp16acc\)1\.24×10−11\.24\{\\times\}10^\{\-1\}2\.44×10−32\.44\{\\times\}10^\{\-3\}One \(fp32acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}6\.99×10−36\.99\{\\times\}10^\{\-3\}\|Global25 \(fp32acc\)1\.24×10−11\.24\{\\times\}10^\{\-1\}2\.44×10−32\.44\{\\times\}10^\{\-3\}FP16 \(fp16acc\)2\.93×10−42\.93\{\\times\}10^\{\-4\}1\.02×10−51\.02\{\\times\}10^\{\-5\}\|Global50 \(fp16acc\)1\.80×10−11\.80\{\\times\}10^\{\-1\}4\.29×10−34\.29\{\\times\}10^\{\-3\}FP16 \(fp32acc\)0000\|Global50 \(fp32acc\)1\.80×10−11\.80\{\\times\}10^\{\-1\}4\.28×10−34\.28\{\\times\}10^\{\-3\}AlignSparse25 \(fp16acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}2\.71×10−32\.71\{\\times\}10^\{\-3\}\|Global75 \(fp16acc\)1\.80×10−11\.80\{\\times\}10^\{\-1\}4\.64×10−34\.64\{\\times\}10^\{\-3\}AlignSparse25 \(fp32acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}2\.71×10−32\.71\{\\times\}10^\{\-3\}\|Global75 \(fp32acc\)1\.80×10−11\.80\{\\times\}10^\{\-1\}4\.64×10−34\.64\{\\times\}10^\{\-3\}AlignSparse50 \(fp16acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}2\.92×10−32\.92\{\\times\}10^\{\-3\}\|RowRand25 \(fp16acc\)1\.56×10−11\.56\{\\times\}10^\{\-1\}2\.39×10−32\.39\{\\times\}10^\{\-3\}AlignSparse50 \(fp32acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}2\.91×10−32\.91\{\\times\}10^\{\-3\}\|RowRand25 \(fp32acc\)1\.56×10−11\.56\{\\times\}10^\{\-1\}2\.38×10−32\.38\{\\times\}10^\{\-3\}AlignSparse75 \(fp16acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}3\.82×10−33\.82\{\\times\}10^\{\-3\}\|RowRand50 \(fp16acc\)2\.27×10−12\.27\{\\times\}10^\{\-1\}4\.34×10−34\.34\{\\times\}10^\{\-3\}AlignSparse75 \(fp32acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}3\.81×10−33\.81\{\\times\}10^\{\-3\}\|RowRand50 \(fp32acc\)2\.27×10−12\.27\{\\times\}10^\{\-1\}4\.34×10−34\.34\{\\times\}10^\{\-3\}Band25 \(fp16acc\)1\.10×10−11\.10\{\\times\}10^\{\-1\}1\.91×10−31\.91\{\\times\}10^\{\-3\}\|RowRand75 \(fp16acc\)2\.12×10−12\.12\{\\times\}10^\{\-1\}4\.60×10−34\.60\{\\times\}10^\{\-3\}Band25 \(fp32acc\)1\.10×10−11\.10\{\\times\}10^\{\-1\}1\.90×10−31\.90\{\\times\}10^\{\-3\}\|RowRand75 \(fp32acc\)2\.12×10−12\.12\{\\times\}10^\{\-1\}4\.60×10−34\.60\{\\times\}10^\{\-3\}Band50 \(fp16acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}3\.62×10−33\.62\{\\times\}10^\{\-3\}\|SpTrans25 \(fp16acc\)1\.40×10−11\.40\{\\times\}10^\{\-1\}1\.73×10−31\.73\{\\times\}10^\{\-3\}Band50 \(fp32acc\)2\.38×10−12\.38\{\\times\}10^\{\-1\}3\.61×10−33\.61\{\\times\}10^\{\-3\}\|SpTrans25 \(fp32acc\)1\.41×10−11\.41\{\\times\}10^\{\-1\}1\.72×10−31\.72\{\\times\}10^\{\-3\}Band75 \(fp16acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}5\.21×10−35\.21\{\\times\}10^\{\-3\}\|SpTrans50 \(fp16acc\)1\.40×10−11\.40\{\\times\}10^\{\-1\}2\.50×10−32\.50\{\\times\}10^\{\-3\}Band75 \(fp32acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}5\.21×10−35\.21\{\\times\}10^\{\-3\}\|SpTrans50 \(fp32acc\)1\.41×10−11\.41\{\\times\}10^\{\-1\}2\.49×10−32\.49\{\\times\}10^\{\-3\}BigBird25 \(fp16acc\)1\.56×10−11\.56\{\\times\}10^\{\-1\}2\.39×10−32\.39\{\\times\}10^\{\-3\}\|SpTrans75 \(fp16acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}5\.09×10−35\.09\{\\times\}10^\{\-3\}BigBird25 \(fp32acc\)1\.56×10−11\.56\{\\times\}10^\{\-1\}2\.38×10−32\.38\{\\times\}10^\{\-3\}\|SpTrans75 \(fp32acc\)6\.59×10−16\.59\{\\times\}10^\{\-1\}5\.09×10−35\.09\{\\times\}10^\{\-3\}BigBird50 \(fp16acc\)2\.27×10−12\.27\{\\times\}10^\{\-1\}4\.34×10−34\.34\{\\times\}10^\{\-3\}\|BigBird75 \(fp16acc\)2\.12×10−12\.12\{\\times\}10^\{\-1\}4\.60×10−34\.60\{\\times\}10^\{\-3\}BigBird50 \(fp32acc\)2\.27×10−12\.27\{\\times\}10^\{\-1\}4\.34×10−34\.34\{\\times\}10^\{\-3\}\|BigBird75 \(fp32acc\)2\.12×10−12\.12\{\\times\}10^\{\-1\}4\.60×10−34\.60\{\\times\}10^\{\-3\}
Table 19:Numerical differences compared to full FP16 with FP32 accumulation under different precision layouts and mixing ratios\. Each configuration is evaluated with FP16 accumulation and FP32 accumulation\.

LMaxMean\|LMaxMean\|LMaxMean\|LMaxMean00\.000\.000\.000\.00\|81\.09×10−21\.09\{\\times\}10^\{\-2\}8\.17×10−48\.17\{\\times\}10^\{\-4\}\|161\.10×10−21\.10\{\\times\}10^\{\-2\}1\.47×10−31\.47\{\\times\}10^\{\-3\}\|241\.11×10−21\.11\{\\times\}10^\{\-2\}3\.52×10−33\.52\{\\times\}10^\{\-3\}11\.08×10−51\.08\{\\times\}10^\{\-5\}1\.75×10−61\.75\{\\times\}10^\{\-6\}\|91\.09×10−21\.09\{\\times\}10^\{\-2\}9\.81×10−49\.81\{\\times\}10^\{\-4\}\|171\.10×10−21\.10\{\\times\}10^\{\-2\}1\.62×10−31\.62\{\\times\}10^\{\-3\}\|251\.11×10−21\.11\{\\times\}10^\{\-2\}3\.96×10−33\.96\{\\times\}10^\{\-3\}21\.08×10−51\.08\{\\times\}10^\{\-5\}3\.91×10−63\.91\{\\times\}10^\{\-6\}\|101\.10×10−21\.10\{\\times\}10^\{\-2\}1\.06×10−31\.06\{\\times\}10^\{\-3\}\|181\.10×10−21\.10\{\\times\}10^\{\-2\}1\.81×10−31\.81\{\\times\}10^\{\-3\}\|261\.11×10−21\.11\{\\times\}10^\{\-2\}4\.39×10−44\.39\{\\times\}10^\{\-4\}31\.09×10−51\.09\{\\times\}10^\{\-5\}5\.58×10−65\.58\{\\times\}10^\{\-6\}\|111\.10×10−21\.10\{\\times\}10^\{\-2\}1\.06×10−31\.06\{\\times\}10^\{\-3\}\|191\.11×10−21\.11\{\\times\}10^\{\-2\}2\.11×10−32\.11\{\\times\}10^\{\-3\}\|271\.11×10−21\.11\{\\times\}10^\{\-2\}4\.98×10−44\.98\{\\times\}10^\{\-4\}41\.09×10−51\.09\{\\times\}10^\{\-5\}6\.42×10−66\.42\{\\times\}10^\{\-6\}\|121\.10×10−21\.10\{\\times\}10^\{\-2\}1\.10×10−31\.10\{\\times\}10^\{\-3\}\|201\.11×10−21\.11\{\\times\}10^\{\-2\}2\.31×10−32\.31\{\\times\}10^\{\-3\}\|281\.11×10−21\.11\{\\times\}10^\{\-2\}5\.54×10−45\.54\{\\times\}10^\{\-4\}51\.09×10−51\.09\{\\times\}10^\{\-5\}6\.95×10−66\.95\{\\times\}10^\{\-6\}\|131\.10×10−21\.10\{\\times\}10^\{\-2\}1\.11×10−31\.11\{\\times\}10^\{\-3\}\|211\.11×10−21\.11\{\\times\}10^\{\-2\}2\.60×10−32\.60\{\\times\}10^\{\-3\}\|291\.11×10−21\.11\{\\times\}10^\{\-2\}6\.65×10−46\.65\{\\times\}10^\{\-4\}61\.09×10−51\.09\{\\times\}10^\{\-5\}7\.20×10−67\.20\{\\times\}10^\{\-6\}\|141\.10×10−21\.10\{\\times\}10^\{\-2\}1\.20×10−31\.20\{\\times\}10^\{\-3\}\|221\.11×10−21\.11\{\\times\}10^\{\-2\}2\.82×10−32\.82\{\\times\}10^\{\-3\}\|301\.16×10−21\.16\{\\times\}10^\{\-2\}8\.42×10−48\.42\{\\times\}10^\{\-4\}71\.09×10−51\.09\{\\times\}10^\{\-5\}7\.51×10−67\.51\{\\times\}10^\{\-6\}\|151\.11×10−21\.11\{\\times\}10^\{\-2\}1\.35×10−31\.35\{\\times\}10^\{\-3\}\|231\.11×10−21\.11\{\\times\}10^\{\-2\}3\.14×10−33\.14\{\\times\}10^\{\-3\}\|311\.94×10−21\.94\{\\times\}10^\{\-2\}1\.27×10−51\.27\{\\times\}10^\{\-5\}

Table 21:Layer\-wise numerical differences between FP16 and FP32 accumulation onLLaMA 3\.1 8B\. Max and Mean report the maximum and mean absolute differences at each layer\.
LMaxMean\|LMaxMean\|LMaxMean\|LMaxMean00\.000\.000\.000\.00\|125\.63×10−25\.63\{\\times\}10^\{\-2\}1\.07×10−41\.07\{\\times\}10^\{\-4\}\|245\.82×10−25\.82\{\\times\}10^\{\-2\}1\.18×10−41\.18\{\\times\}10^\{\-4\}\|365\.97×10−25\.97\{\\times\}10^\{\-2\}1\.29×10−41\.29\{\\times\}10^\{\-4\}13\.23×10−33\.23\{\\times\}10^\{\-3\}2\.30×10−52\.30\{\\times\}10^\{\-5\}\|135\.63×10−25\.63\{\\times\}10^\{\-2\}1\.08×10−41\.08\{\\times\}10^\{\-4\}\|255\.82×10−25\.82\{\\times\}10^\{\-2\}1\.19×10−41\.19\{\\times\}10^\{\-4\}\|375\.95×10−25\.95\{\\times\}10^\{\-2\}1\.31×10−41\.31\{\\times\}10^\{\-4\}26\.38×10−36\.38\{\\times\}10^\{\-3\}3\.50×10−53\.50\{\\times\}10^\{\-5\}\|145\.62×10−25\.62\{\\times\}10^\{\-2\}1\.08×10−41\.08\{\\times\}10^\{\-4\}\|265\.88×10−25\.88\{\\times\}10^\{\-2\}1\.19×10−41\.19\{\\times\}10^\{\-4\}\|385\.92×10−25\.92\{\\times\}10^\{\-2\}1\.34×10−41\.34\{\\times\}10^\{\-4\}39\.78×10−39\.78\{\\times\}10^\{\-3\}5\.82×10−55\.82\{\\times\}10^\{\-5\}\|155\.62×10−25\.62\{\\times\}10^\{\-2\}1\.09×10−41\.09\{\\times\}10^\{\-4\}\|275\.90×10−25\.90\{\\times\}10^\{\-2\}1\.20×10−41\.20\{\\times\}10^\{\-4\}\|395\.88×10−25\.88\{\\times\}10^\{\-2\}1\.35×10−41\.35\{\\times\}10^\{\-4\}45\.10×10−25\.10\{\\times\}10^\{\-2\}7\.92×10−57\.92\{\\times\}10^\{\-5\}\|165\.62×10−25\.62\{\\times\}10^\{\-2\}1\.09×10−41\.09\{\\times\}10^\{\-4\}\|285\.94×10−25\.94\{\\times\}10^\{\-2\}1\.21×10−41\.21\{\\times\}10^\{\-4\}\|405\.82×10−25\.82\{\\times\}10^\{\-2\}1\.37×10−41\.37\{\\times\}10^\{\-4\}55\.43×10−25\.43\{\\times\}10^\{\-2\}8\.40×10−58\.40\{\\times\}10^\{\-5\}\|175\.63×10−25\.63\{\\times\}10^\{\-2\}1\.10×10−41\.10\{\\times\}10^\{\-4\}\|295\.96×10−25\.96\{\\times\}10^\{\-2\}1\.21×10−41\.21\{\\times\}10^\{\-4\}\|415\.76×10−25\.76\{\\times\}10^\{\-2\}1\.39×10−41\.39\{\\times\}10^\{\-4\}65\.64×10−25\.64\{\\times\}10^\{\-2\}8\.95×10−58\.95\{\\times\}10^\{\-5\}\|185\.62×10−25\.62\{\\times\}10^\{\-2\}1\.10×10−41\.10\{\\times\}10^\{\-4\}\|305\.96×10−25\.96\{\\times\}10^\{\-2\}1\.22×10−41\.22\{\\times\}10^\{\-4\}\|425\.74×10−25\.74\{\\times\}10^\{\-2\}1\.41×10−41\.41\{\\times\}10^\{\-4\}75\.65×10−25\.65\{\\times\}10^\{\-2\}9\.94×10−59\.94\{\\times\}10^\{\-5\}\|195\.62×10−25\.62\{\\times\}10^\{\-2\}1\.11×10−41\.11\{\\times\}10^\{\-4\}\|315\.98×10−25\.98\{\\times\}10^\{\-2\}1\.23×10−41\.23\{\\times\}10^\{\-4\}\|435\.71×10−25\.71\{\\times\}10^\{\-2\}1\.44×10−41\.44\{\\times\}10^\{\-4\}85\.64×10−25\.64\{\\times\}10^\{\-2\}1\.02×10−41\.02\{\\times\}10^\{\-4\}\|205\.62×10−25\.62\{\\times\}10^\{\-2\}1\.12×10−41\.12\{\\times\}10^\{\-4\}\|325\.98×10−25\.98\{\\times\}10^\{\-2\}1\.24×10−41\.24\{\\times\}10^\{\-4\}\|443\.65×10−23\.65\{\\times\}10^\{\-2\}1\.37×10−41\.37\{\\times\}10^\{\-4\}95\.64×10−25\.64\{\\times\}10^\{\-2\}1\.05×10−41\.05\{\\times\}10^\{\-4\}\|215\.62×10−25\.62\{\\times\}10^\{\-2\}1\.12×10−41\.12\{\\times\}10^\{\-4\}\|335\.98×10−25\.98\{\\times\}10^\{\-2\}1\.25×10−41\.25\{\\times\}10^\{\-4\}\|452\.96×10−22\.96\{\\times\}10^\{\-2\}1\.39×10−41\.39\{\\times\}10^\{\-4\}105\.64×10−25\.64\{\\times\}10^\{\-2\}1\.06×10−41\.06\{\\times\}10^\{\-4\}\|225\.78×10−25\.78\{\\times\}10^\{\-2\}1\.14×10−41\.14\{\\times\}10^\{\-4\}\|345\.98×10−25\.98\{\\times\}10^\{\-2\}1\.26×10−41\.26\{\\times\}10^\{\-4\}\|462\.98×10−22\.98\{\\times\}10^\{\-2\}1\.76×10−41\.76\{\\times\}10^\{\-4\}115\.64×10−25\.64\{\\times\}10^\{\-2\}1\.07×10−41\.07\{\\times\}10^\{\-4\}\|235\.79×10−25\.79\{\\times\}10^\{\-2\}1\.17×10−41\.17\{\\times\}10^\{\-4\}\|355\.98×10−25\.98\{\\times\}10^\{\-2\}1\.27×10−41\.27\{\\times\}10^\{\-4\}\|478\.93×10−48\.93\{\\times\}10^\{\-4\}1\.83×10−51\.83\{\\times\}10^\{\-5\}

Table 22:Layer\-wise numerical differences between FP16 and FP32 accumulation onQwen 2\.5 14B\. Max and Mean report the maximum and mean absolute differences at each layer\.
ConfigTop5Top10Top20Top30SpTrans258\.57%8\.63%8\.63%8\.48%SpTrans5019\.19%19\.15%18\.63%17\.76%SpTrans7521\.55%21\.66%21\.60%21\.15%Table 23:Weighted INT8 exposure of heavy\-hitter importance under SpTrans precision layouts\. Lower values indicate that less high\-importance attention mass is routed to INT8\.

ConfigMaxAbsDiffMeanAbsDiff\|ConfigMaxAbsDiffMeanAbsDiffOne3\.91×10−43\.91\{\\times\}10^\{\-4\}1\.51×10−51\.51\{\\times\}10^\{\-5\}\|Global252\.93×10−42\.93\{\\times\}10^\{\-4\}1\.21×10−51\.21\{\\times\}10^\{\-5\}FP162\.93×10−42\.93\{\\times\}10^\{\-4\}1\.02×10−51\.02\{\\times\}10^\{\-5\}\|Global503\.91×10−43\.91\{\\times\}10^\{\-4\}1\.36×10−51\.36\{\\times\}10^\{\-5\}AlignSparse253\.91×10−43\.91\{\\times\}10^\{\-4\}1\.24×10−51\.24\{\\times\}10^\{\-5\}\|Global753\.91×10−43\.91\{\\times\}10^\{\-4\}1\.44×10−51\.44\{\\times\}10^\{\-5\}AlignSparse503\.91×10−43\.91\{\\times\}10^\{\-4\}1\.27×10−51\.27\{\\times\}10^\{\-5\}\|RowRand253\.91×10−43\.91\{\\times\}10^\{\-4\}1\.17×10−51\.17\{\\times\}10^\{\-5\}AlignSparse752\.93×10−42\.93\{\\times\}10^\{\-4\}1\.37×10−51\.37\{\\times\}10^\{\-5\}\|RowRand503\.91×10−43\.91\{\\times\}10^\{\-4\}1\.37×10−51\.37\{\\times\}10^\{\-5\}Band253\.91×10−43\.91\{\\times\}10^\{\-4\}1\.19×10−51\.19\{\\times\}10^\{\-5\}\|RowRand753\.91×10−43\.91\{\\times\}10^\{\-4\}1\.46×10−51\.46\{\\times\}10^\{\-5\}Band503\.91×10−43\.91\{\\times\}10^\{\-4\}1\.32×10−51\.32\{\\times\}10^\{\-5\}\|SpTrans253\.91×10−43\.91\{\\times\}10^\{\-4\}1\.15×10−51\.15\{\\times\}10^\{\-5\}Band753\.42×10−43\.42\{\\times\}10^\{\-4\}1\.41×10−51\.41\{\\times\}10^\{\-5\}\|SpTrans504\.39×10−44\.39\{\\times\}10^\{\-4\}1\.21×10−51\.21\{\\times\}10^\{\-5\}BigBird253\.91×10−43\.91\{\\times\}10^\{\-4\}1\.17×10−51\.17\{\\times\}10^\{\-5\}\|SpTrans754\.39×10−44\.39\{\\times\}10^\{\-4\}1\.37×10−51\.37\{\\times\}10^\{\-5\}BigBird503\.91×10−43\.91\{\\times\}10^\{\-4\}1\.37×10−51\.37\{\\times\}10^\{\-5\}\|BigBird753\.91×10−43\.91\{\\times\}10^\{\-4\}1\.46×10−51\.46\{\\times\}10^\{\-5\}

Table 20:Direct comparison between FP16 and FP32 accumulation under different precision layouts and mixing ratios\.
## Appendix GNumerical Analysis

The fixed Torch FP16 implementation provides a common reference for comparing kernel schedules and routed arithmetic paths\. Reported deviations measure implementation\-level output agreement with this reference across differences in quantization, accumulation order, rounding behavior, and reduction schedule\.

This appendix complements the main numerical analysis in Section[5\.4](https://arxiv.org/html/2608.17336#S5.SS4)\. We examine numerical behavior from six perspectives: \(i\) kernel\-level output deviation on random attention inputs, \(ii\) model\-depth and sequence\-length effects, \(iii\) INT8 coverage ratio, \(iv\) direct comparison with FlashAttention, \(v\) accumulation precision and larger\-model accumulation checks, and \(vi\) static\-routing exposure of high\-mass attention interactions\.

### G\.1Depth and Sequence\-Length Effects

Tables[15](https://arxiv.org/html/2608.17336#A6.T15),[16](https://arxiv.org/html/2608.17336#A6.T16), and[17](https://arxiv.org/html/2608.17336#A6.T17)report results for a single\-layer model, a 12\-layer model, and a 32\-layer model, respectively\. All experiments use identical random inputs and shared weights across attention implementations\. For each sequence length, the*Torch Value*column reports the maximum absolute logit magnitude produced by the selected Torch FP16 reference, serving as a scale anchor\. All other entries report maximum absolute deviation of model logits relative to this reference\.

Across model depths, deviations generally increase as depth increases, reflecting accumulation of implementation\-level differences across layers\. Uniform INT8 attention produces larger deviations than mixed\-routing configurations\. Reducing the fraction of INT8\-routed tile groups generally reduces deviation from the fixed Torch FP16 reference\. Within this controlled setup, model depth and INT8 coverage produce larger changes in logit deviation than sequence length over the evaluated range\.

### G\.2Direct Comparison with FlashAttention

Table[18](https://arxiv.org/html/2608.17336#A6.T18)reports direct output differences betweenTileMixand FlashAttention under different precision layouts and INT8 ratios\.

A nonzero difference exists at 0% INT8 becauseTileMixand FlashAttention use different fused\-kernel schedules\. Within most structured layouts, MeanDiff increases with INT8 coverage; repeated MaxDiff or MeanDiff entries indicate equality at the reported numerical precision, not bitwise\-identical outputs\.

### G\.3Pattern and Accumulation Effects

Table[19](https://arxiv.org/html/2608.17336#A6.T19)separates the effects of precision layout, INT8 coverage, and accumulation precision\.

The results show that numerical behavior depends mainly on the precision layout and INT8 coverage ratio, while FP16 and FP32 accumulation produce similar deviations in this setting\.

Table[20](https://arxiv.org/html/2608.17336#A6.T20)directly compares FP16 and FP32 accumulation under matched precision layouts\.

The small differences across configurations indicate that routing layout and INT8 coverage produce much larger numerical effects than accumulation precision in the evaluated setting\.

### G\.4Larger\-Model Accumulation Stability

We further compareTileMixwith FP16 accumulation againstTileMixwith FP32 accumulation on LLaMA3\.1\-8B and Qwen2\.5\-14B\. Tables[21](https://arxiv.org/html/2608.17336#A6.T21)and[22](https://arxiv.org/html/2608.17336#A6.T22)show that FP16 accumulation remains close to FP32 accumulation across larger models\.

On LLaMA 3\.1 8B, the maximum layer\-wise difference remains on the order of10−210^\{\-2\}, while most mean differences stay around10−310^\{\-3\}or lower\. On Qwen 2\.5 14B, maximum differences are larger but remain stable across layers, and mean differences stay around10−410^\{\-4\}\. Together, these larger\-model checks show that FP16\-to\-FP32 accumulation differences remain smaller than the routed\-precision effects characterized above\.

### G\.5Positional Routing and Heavy\-Hitter Exposure

We quantify how static SpTrans layouts distribute INT8 execution over high\-importance interactions while retaining constant\-time routing lookup inside the kernel\.

For each layer\-head attention map, we define first\-order query\-key importance as

Iq​k=\|Pq​k​∂ℒ∂Pq​k\|I\_\{qk\}=\\left\|P\_\{qk\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial P\_\{qk\}\}\\right\|For each queryqq, letℋqτ\\mathcal\{H\}\_\{q\}^\{\\tau\}be the smallest key set whose cumulative importance reachesτ%\\tau\\%, whereτ∈\{5,10,20,30\}\\tau\\in\\\{5,10,20,30\\\}\. LetDq​k=1D\_\{qk\}=1indicate that position\(q,k\)\(q,k\)is routed to INT8\. We define the weighted INT8 exposure of heavy hitters as

Eτ=∑q∑k∈ℋqτIq​k​Dq​k∑q∑k∈ℋqτIq​k\.E^\{\\tau\}=\\frac\{\\sum\_\{q\}\\sum\_\{k\\in\\mathcal\{H\}\_\{q\}^\{\\tau\}\}I\_\{qk\}D\_\{qk\}\}\{\\sum\_\{q\}\\sum\_\{k\\in\\mathcal\{H\}\_\{q\}^\{\\tau\}\}I\_\{qk\}\}\.A lowerEτE^\{\\tau\}indicates stronger protection of high\-importance attention mass from INT8 routing\.

Table[23](https://arxiv.org/html/2608.17336#A6.T23)shows that SpTrans25 routes only about 8\.5% of the selected high\-importance mass to INT8, substantially below its nominal 25% tile\-group coverage\.

Across all three coverage levels, structured spatial routing retains a larger share of high\-importance interactions in FP16 without online detection\.