Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

# Functional Reconstruction for MLA Draft Models in Speculative Decoding
Source: [https://arxiv.org/html/2607.27269](https://arxiv.org/html/2607.27269)
###### Abstract

Multi\-head latent attention \(MLA\) is an increasingly important architecture for long\-context LLM inference, as it replaces the growing key\-value \(KV\) cache with compact latent states and thereby reduces memory traffic during decoding\. Yet most capable open checkpoints were pretrained with multi\-head or grouped\-query attention \(MHA/GQA\), making conversion to MLA essential for realizing these cache\-efficiency benefits without costly retraining from scratch\. Meanwhile, speculative decoding has emerged as a complementary approach to accelerate autoregressive generation\. Its practical speedup, however, depends critically on agreement between draft proposals and target verification: stochastic sampling is governed by distributional overlap, whereas greedy decoding requires top\-ranked\-token agreement\. We find that direct MHA/GQA\-to\-MLA conversion can severely degrade this agreement\. Low\-rank factorization and RoPE handling introduce attention\-function errors that may be tolerable in standalone generation, but cause sharply lower token acceptance in speculative decoding\. We therefore formulate MLA draft construction as a functional reconstruction problem rather than a cache\-compression problem\. Our end\-to\-end \(E2E\) reconstruction directly optimizes each converted MLA attention module to reproduce the post\-output\-projection response of its original MHA/GQA counterpart on calibration hidden states\. It is a converter\-agnostic post\-conversion procedure that preserves the converted cache structure and uses no verifier\-model logits or supervision\. We evaluate 192 model–converter–backend–method–task configurations spanning four Llama/Qwen draft–target pairs, TransMLA and MHA2MLA, HF and vLLM backends, and four 200\-prompt tasks\. Using a 0\.5\-percentage\-point reporting tolerance, Functional Reconstruction materially improves acceptance in 37 of 64 matched task cells, leaves 26 practically unchanged, and materially decreases one\. We release our implementation and full evaluation artifact athttps://github\.com/swyhahaha/FunctionalMLA\.

## Introduction

Large language model inference is increasingly shaped by memory movement rather than arithmetic\. During autoregressive decoding, every generated token reads a growing key\-value \(KV\) cache, so serving latency and throughput depend heavily on how efficiently this cache is represented and accessed\. Grouped\-query attention \(GQA\) reduces the number of KV heads relative to multi\-head attention, but it still stores explicit keys and values\. Multi\-head latent attention \(MLA\), used in DeepSeek\-style models, goes further: it caches a low\-dimensional latent vector and a small positional component, then computes attention through absorbed projections\(DeepSeek\-AI[2024a](https://arxiv.org/html/2607.27269#bib.bib5)\)\. Recent conversion methods make this design practical without pretraining an MLA model from scratch: TransMLA converts existing GQA/MHA checkpoints, while MHA2MLA provides a related MHA\-to\-MLA migration path\(Menget al\.[2025](https://arxiv.org/html/2607.27269#bib.bib6); Ji and others[2025](https://arxiv.org/html/2607.27269#bib.bib7)\)\.

This paper studies a failure mode that is easy to miss if one only measures standalone decoding\. A converted MLA model may be an efficient generator but still a poor*draft model*\. In speculative decoding, a draft proposes several tokens and a target model verifies them in parallel\(Leviathanet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib8); Chenet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib9)\)\. When draft proposals agree with target verification, many tokens are accepted and the target advances by multiple positions per verification step\. When conversion changes proposal probabilities or rankings, verification rejects more tokens and the draft becomes overhead\.

In this paper, we introduce a new functional reconstruction method to improve draft acceptance and output\-token speed during speculative decoding\. It is a converter\-agnostic, training\-time\-only post\-conversion stage, requiring no verifier logits, verifier activations, benchmark labels, or generated targets, and making no change to the converted cache structure or inference graph\. Starting from a TransMLA\- or MHA2MLA\-converted checkpoint, it trains the converter\-introduced query and KV projections so that the complete MLA attention modules reproduce the post\-WOW\_\{O\}responses of the frozen original MHA/GQA blocks on calibration hidden states\. Across the 64 matched task cells in Table[1](https://arxiv.org/html/2607.27269#Sx6.T1), our method yields 37 material acceptance improvements, 26 practically unchanged outcomes, and one material decrease; 12 of the 37 improved cells also show a material throughput increase\. These results span TransMLA and MHA2MLA, HF and vLLM, and the Llama and Qwen model families\.

Our contributions are threefold:

\(1\) Conversion and drafting are distinct objectives\.Structural conversion can succeed while the proposal agreement needed for draft acceptance is lost\.

\(2\) Draft quality can be optimized entirely at training time\.Our functional objective uses calibration states and the frozen original attention block, without verifier supervision or inference\-time changes\.

\(3\) Training\-time functional reconstruction is converter\-agnostic\.The same objective improves or remains within the reporting tolerance in 63 of 64 matched task cells spanning TransMLA/MHA2MLA, Llama/Qwen, and HF/vLLM\.

## Related Work

KV\-cache efficient attention\.Autoregressive Transformers retain a key and value state for each prior token and layer during decoding\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.27269#bib.bib1)\)\. MQA and GQA reduce this state by sharing KV heads across query heads\(Shazeer[2019](https://arxiv.org/html/2607.27269#bib.bib3); Ainslieet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib2)\)\. Other approaches preserve the original attention representation but reduce its runtime footprint: PagedAttention virtualizes explicit KV blocks\(Kwonet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib10)\), H2O and StreamingLLM retain selected cache entries for long sequences\(Zhanget al\.[2023b](https://arxiv.org/html/2607.27269#bib.bib19); Xiaoet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib20)\), and KIVI quantizes the cache\(Liuet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib21)\)\. FlashAttention and FlashAttention\-2 instead optimize exact attention kernels without changing the cached representation\(Daoet al\.[2022](https://arxiv.org/html/2607.27269#bib.bib11); Dao[2023](https://arxiv.org/html/2607.27269#bib.bib12)\)\. MLA is architecturally distinct: it replaces the explicit KV cache with a learned latent cache, so it can reduce the representation itself rather than only its storage or execution cost\(DeepSeek\-AI[2024a](https://arxiv.org/html/2607.27269#bib.bib5),[b](https://arxiv.org/html/2607.27269#bib.bib22)\)\.

MLA and conversion\.MLA was introduced in DeepSeek\-V2 and retained in DeepSeek\-V3 as an inference\-oriented attention architecture\(DeepSeek\-AI[2024a](https://arxiv.org/html/2607.27269#bib.bib5),[b](https://arxiv.org/html/2607.27269#bib.bib22)\)\. Its migration from pretrained MHA/GQA is nontrivial because both low\-rank KV compression and rotary position handling can perturb the original attention map\(Suet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib4)\)\. TransMLA uses RoPE decomposition with low\-rank QKV conversion\(Menget al\.[2025](https://arxiv.org/html/2607.27269#bib.bib6)\), whereas MHA2MLA combines partial\-RoPE removal with joint low\-rank approximation\(Ji and others[2025](https://arxiv.org/html/2607.27269#bib.bib7)\)\. These methods solve the initialization problem with converter\-local reconstruction objectives\. Our E2E procedure is complementary: it begins from either converted checkpoint and uses original MHA/GQA attention outputs on calibration states as the functional target that matters for a speculative draft\.

Speculative decoding\.Speculative decoding uses a cheap draft to propose multiple tokens and an exact target\-side correction to preserve the target distribution\(Leviathanet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib8); Chenet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib9)\)\. Tree\-based methods enlarge the parallel candidate set through fixed or hardware\-aware draft trees\(Miaoet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib13); Chenet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib25)\); Medusa uses auxiliary decoding heads\(Caiet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib14)\); and EAGLE/EAGLE\-2 draft in feature space with static or context\-adaptive trees\(Liet al\.[2024b](https://arxiv.org/html/2607.27269#bib.bib15),[a](https://arxiv.org/html/2607.27269#bib.bib26)\)\. Self\-speculative approaches reuse the target model through layer skipping or early exit\(Zhanget al\.[2023a](https://arxiv.org/html/2607.27269#bib.bib23); Elhoushiet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib24)\), while Lookahead decoding uses Jacobi\-style parallel updates\(Fuet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib16)\)\. These works alter proposal generation or verification scheduling\. We instead improve the proposal fidelity of an MLA\-converted draft, which is complementary to those designs and directly raises the acceptance on which their speedups depend\.

## Preliminary: Why Converted MLA Is Not Draft\-Preserving

### MHA/GQA Attention

Letht∈ℝDh\_\{t\}\\in\\mathbb\{R\}^\{D\}be the hidden state at positiontt, whereDDdenotes the model hidden dimension, withHHquery heads andGGKV groups\. We use GQA notation below; standard MHA is the special caseG=HG=Hwith one KV group per query head\. For query headiiin groupg​\(i\)g\(i\), GQA computes

qt,i\\displaystyle q\_\{t,i\}=WiQ​ht,\\displaystyle=W^\{Q\}\_\{i\}h\_\{t\},kj,g​\(i\)\\displaystyle k\_\{j,g\(i\)\}=Wg​\(i\)K​hj,\\displaystyle=W^\{K\}\_\{g\(i\)\}h\_\{j\},vj,g​\(i\)\\displaystyle v\_\{j,g\(i\)\}=Wg​\(i\)V​hj\.\\displaystyle=W^\{V\}\_\{g\(i\)\}h\_\{j\}\.\(1\)LetRtR\_\{t\}be the RoPE rotation at positionttand define the relative rotationRt​j=Rt⊤​RjR\_\{tj\}=R\_\{t\}^\{\\top\}R\_\{j\}\(Suet al\.[2024](https://arxiv.org/html/2607.27269#bib.bib4)\)\. With key dimensiondkd\_\{k\}, scaled causal attention is

st​i​jGQA\\displaystyle s^\{\\mathrm\{GQA\}\}\_\{tij\}=qt,i⊤​Rt​j​kj,g​\(i\)dk,\\displaystyle=\\frac\{q\_\{t,i\}^\{\\top\}R\_\{tj\}k\_\{j,g\(i\)\}\}\{\\sqrt\{d\_\{k\}\}\},\(2\)at​i​jGQA\\displaystyle a^\{\\mathrm\{GQA\}\}\_\{tij\}=exp⁡\(st​i​jGQA\)∑u≤texp⁡\(st​i​uGQA\),\\displaystyle=\\frac\{\\exp\(s^\{\\mathrm\{GQA\}\}\_\{tij\}\)\}\{\\sum\_\{u\\leq t\}\\exp\(s^\{\\mathrm\{GQA\}\}\_\{tiu\}\)\},\(3\)ot,iGQA\\displaystyle o^\{\\mathrm\{GQA\}\}\_\{t,i\}=∑j≤tat​i​jGQA​vj,g​\(i\)\.\\displaystyle=\\sum\_\{j\\leq t\}a^\{\\mathrm\{GQA\}\}\_\{tij\}v\_\{j,g\(i\)\}\.\(4\)

### Latent KV Bottleneck

An MLA layer stores a latent cachecj=WD​K​V​hj∈ℝrc\_\{j\}=W^\{DKV\}h\_\{j\}\\in\\mathbb\{R\}^\{r\}and a small positional keykjP​Ek^\{PE\}\_\{j\}\. The non\-positional keys and values of all KV groups are stacked and produced from the latent:

\[k^jn​o​p​ev^j\]=WU​K​V​WD​K​V​hj\.\\begin\{bmatrix\}\\hat\{k\}^\{nope\}\_\{j\}\\\\ \\hat\{v\}\_\{j\}\\end\{bmatrix\}=W^\{UKV\}W^\{DKV\}h\_\{j\}\.\(5\)HereWD​K​V∈ℝr×DW^\{DKV\}\\in\\mathbb\{R\}^\{r\\times D\}andWU​K​V∈ℝm×rW^\{UKV\}\\in\\mathbb\{R\}^\{m\\times r\}, wherem=G​\(dkn​o​p​e\+dv\)m=G\(d\_\{k\}^\{nope\}\+d\_\{v\}\)\. Exact reproduction of this stacked linear map for all hidden states requires

\[WK,n​o​p​eWV\]=WU​K​V​WD​K​V\.\\begin\{bmatrix\}W^\{K,nope\}\\\\ W^\{V\}\\end\{bmatrix\}=W^\{UKV\}W^\{DKV\}\.\(6\)The right\-hand side has rank at mostrr\. Therefore exact equivalence requires

rank​\(\[WK,n​o​p​eWV\]\)≤r\.\\mathrm\{rank\}\\\!\\left\(\\begin\{bmatrix\}W^\{K,nope\}\\\\ W^\{V\}\\end\{bmatrix\}\\right\)\\leq r\.\(7\)Whenever this rank exceedsrr, the chosen compressed factorization cannot reproduce the stacked K/V map exactly and must leave a residual\. This conditional statement concerns direct map reconstruction at the selected rank: it neither rules out every functionally equivalent attention parameterization nor contradicts exact MLA rewrites at a sufficiently large rank\. It shows why an aggressively compressed, cache\-valid conversion is generally an approximation of the original K/V projections\.

![Refer to caption](https://arxiv.org/html/2607.27269v1/x1.png)ℒE2E\(ℓ\)=‖M⊙\(AMLAℓ−sg⁡\[Aorigℓ\]\)‖F2D​∑tMt\\mathcal\{L\}\_\{\\mathrm\{E2E\}\}^\{\(\\ell\)\}=\\dfrac\{\\left\\\|M\\odot\\left\(A^\{\\ell\}\_\{\\mathrm\{MLA\}\}\-\\operatorname\{sg\}\[A^\{\\ell\}\_\{\\mathrm\{orig\}\}\]\\right\)\\right\\\|\_\{F\}^\{2\}\}\{D\\sum\_\{t\}M\_\{t\}\}

Figure 1:Converter\-agnostic E2E functional reconstruction\. A TransMLA or MHA2MLA conversion first supplies a cache\-capable MLA initialization\. For each calibration hidden statehℓh^\{\\ell\}, the frozen original GQA block and the trainable converted MLA block produce post\-WOW\_\{O\}outputs\. E2E minimizes their masked mean\-squared error, stops gradients through the original path, and updates only the converter\-introduced query and KV projections\. The lower schematic illustrates the empirical pathway tested: reducing the residual can preserve proposal rankings and improve acceptance\.
### RoPE Handling and Acceptance

The rank bottleneck is not the only mismatch\. Conversion pipelines must also choose how to handle RoPE\. To make the resulting error explicit, split an original key askj=kjk​e​e​p\+kjd​r​o​pk\_\{j\}=k\_\{j\}^\{keep\}\+k\_\{j\}^\{drop\}, wherekjk​e​e​pk\_\{j\}^\{keep\}remains on the rotary path andkjd​r​o​pk\_\{j\}^\{drop\}is moved to a non\-positional path\. Let the converted query beq^t=qt\+δ​qt\\hat\{q\}\_\{t\}=q\_\{t\}\+\\delta q\_\{t\}and the reconstructed retained key bek^jk​e​e​p=kjk​e​e​p\+δ​kj\\hat\{k\}\_\{j\}^\{keep\}=k\_\{j\}^\{keep\}\+\\delta k\_\{j\}\. Under this representative routing, the original logit usesqt⊤​Rt​j​\(kjk​e​e​p\+kjd​r​o​p\)/dkq\_\{t\}^\{\\top\}R\_\{tj\}\(k\_\{j\}^\{keep\}\+k\_\{j\}^\{drop\}\)/\\sqrt\{d\_\{k\}\}, whereas the converted logit uses\(qt\+δ​qt\)⊤​\[Rt​j​\(kjk​e​e​p\+δ​kj\)\+kjd​r​o​p\]/dk\(q\_\{t\}\+\\delta q\_\{t\}\)^\{\\top\}\[R\_\{tj\}\(k\_\{j\}^\{keep\}\+\\delta k\_\{j\}\)\+k\_\{j\}^\{drop\}\]/\\sqrt\{d\_\{k\}\}\. Their exact difference is

s^t​j−st​j=1dk\[\\displaystyle\\hat\{s\}\_\{tj\}\-s\_\{tj\}=\\frac\{1\}\{\\sqrt\{d\_\{k\}\}\}\\Big\[qt⊤​Rt​j​δ​kj\+qt⊤​\(I−Rt​j\)​kjd​r​o​p\\displaystyle q\_\{t\}^\{\\top\}R\_\{tj\}\\delta k\_\{j\}\+q\_\{t\}^\{\\top\}\(I\-R\_\{tj\}\)k\_\{j\}^\{drop\}\(8\)\+δqt⊤Rt​j\(kjk​e​e​p\+δkj\)\+δqt⊤kjd​r​o​p\]\.\\displaystyle\+\\delta q\_\{t\}^\{\\top\}R\_\{tj\}\(k\_\{j\}^\{keep\}\+\\delta k\_\{j\}\)\+\\delta q\_\{t\}^\{\\top\}k\_\{j\}^\{drop\}\\Big\]\.This decomposition exposes retained\-key reconstruction error, positional\-path error, and first\-order query perturbation; the third term also contains their interaction\. Softmax maps the logit residual into attention\-weight error, and the output residual additionally inherits value reconstruction error\.

Speculative decoding makes this residual operationally visible\. At a fixed contextxx, writepDx​\(y\)=pD​\(y∣x\)p\_\{D\}^\{x\}\(y\)=p\_\{D\}\(y\\mid x\)and similarly forpTxp\_\{T\}^\{x\}\. Stochastic rejection\-sampling verification has one\-token expected acceptance\(Leviathanet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib8); Chenet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib9)\)

αsample​\(x\)\\displaystyle\\alpha\_\{\\mathrm\{sample\}\}\(x\)=∑ymin⁡\(pDx​\(y\),pTx​\(y\)\)\\displaystyle=\\sum\_\{y\}\\min\\\!\\left\(p\_\{D\}^\{x\}\(y\),p\_\{T\}^\{x\}\(y\)\\right\)\(9\)=1−TV​\(pDx,pTx\)\.\\displaystyle=1\-\\mathrm\{TV\}\\\!\\left\(p\_\{D\}^\{x\},p\_\{T\}^\{x\}\\right\)\.Our experiments instead use greedy decoding, whose one\-token acceptance event is

αgreedy​\(x\)=𝟏​\[arg⁡maxy⁡pDx​\(y\)=arg⁡maxy⁡pTx​\(y\)\]\.\\alpha\_\{\\mathrm\{greedy\}\}\(x\)=\\mathbf\{1\}\\\!\\left\[\\arg\\max\_\{y\}p\_\{D\}^\{x\}\(y\)=\\arg\\max\_\{y\}p\_\{T\}^\{x\}\(y\)\\right\]\.\(10\)MLA conversion can therefore hurt stochastic overlap or flip the top\-ranked token under greedy verification\. Neither quantity is guaranteed to vary monotonically with a local attention residual; functional reconstruction is a training\-time proxy intended to preserve the original draft’s conditional behavior, whose acceptance effect is measured empirically below\.

## Method

Figure[1](https://arxiv.org/html/2607.27269#Sx3.F1)separates conversion, reconstruction, and downstream acceptance\. TransMLA or MHA2MLA first supplies a cache\-capable low\-rank MLA initialization with its converter\-defined retained\-RoPE path\. During reconstruction, the same layer input is sent through a frozen original GQA attention block and the corresponding converted MLA block\. Their post\-WOW\_\{O\}responses define a masked mean\-squared functional error\. Gradients are stopped through the original path and applied only to the converter\-introduced query and KV projections;WOW\_\{O\}remains frozen\. Direct conversion can leave a residual that changes proposal rankings, whereas E2E reduces this residual without changing the cache topology\. Its effect on acceptance is empirical rather than guaranteed by the local loss\. The target verifier is not part of this optimization\. Partial RoPE reconstruction remains the matched converter\-local baseline below, not a second branch of the E2E training graph\.

### Functional Reconstruction

For layerℓ\\elland calibration minibatchbb, letHbℓ∈ℝTb×DH\_\{b\}^\{\\ell\}\\in\\mathbb\{R\}^\{T\_\{b\}\\times D\}contain the cached layer inputs and letMb∈\{0,1\}TbM\_\{b\}\\in\\\{0,1\\\}^\{T\_\{b\}\}mark valid tokens, broadcast over the hidden dimension\. The implementation optimizes each layer independently with

ℒE2E\(ℓ\)=𝔼b​\[‖Mb⊙\(AMLAℓ​\(Hbℓ\)−sg⁡\[Aorigℓ​\(Hbℓ\)\]\)‖F2D​∑tMb,t\]\.\\mathcal\{L\}\_\{\\mathrm\{E2E\}\}^\{\(\\ell\)\}=\\mathbb\{E\}\_\{b\}\\left\[\\frac\{\\left\\\|M\_\{b\}\\odot\\left\(A^\{\\ell\}\_\{\\mathrm\{MLA\}\}\(H\_\{b\}^\{\\ell\}\)\-\\operatorname\{sg\}\\\!\\left\[A^\{\\ell\}\_\{\\mathrm\{orig\}\}\(H\_\{b\}^\{\\ell\}\)\\right\]\\right\)\\right\\\|\_\{F\}^\{2\}\}\{D\\sum\_\{t\}M\_\{b,t\}\}\\right\]\.\(11\)Thus, the denominatorD​∑tMb,tD\\sum\_\{t\}M\_\{b,t\}averages the squared error over every hidden channel of every valid token\. Heresg\\operatorname\{sg\}denotes stop\-gradient, andAℓA^\{\\ell\}is the output of attention layerℓ\\ellafter its fixed output projection\. Unlike a converter\-local projection reconstruction objective, this masked MSE asks whether the complete converted attention block behaves like the original MHA/GQA attention on real calibration hidden states\. In all reported Llama and Qwen experiments,Aorig=AGQAA\_\{\\mathrm\{orig\}\}=A\_\{\\mathrm\{GQA\}\}; the MHA case is covered byG=HG=H\. The loss does not require the rank condition in Eq\. \([7](https://arxiv.org/html/2607.27269#Sx3.E7)\) to hold exactly; it searches within the fixed compressed MLA family for a closer functional match\. The term E2E refers to the full attention function from layer input to post\-output\-projection output, so the objective jointly covers RoPE\-handling and low\-rank KV conversion errors\.

### Converter\-Agnostic Initialization

Let𝒞\\mathcal\{C\}denote a base MHA/GQA\-to\-MLA converter\. E2E takes𝒞​\(θorig\)\\mathcal\{C\}\(\\theta\_\{\\mathrm\{orig\}\}\)as an initialization but does not reuse𝒞\\mathcal\{C\}’s local conversion objective\. It requires only the original attention block, the initialized MLA block, and calibration hidden states\. Consequently, the same E2E objective can refine TransMLA and MHA2MLA checkpoints while preserving the cache topology produced by the selected converter\. Our evaluation instantiates both choices of𝒞\\mathcal\{C\}and compares Functional Reconstruction with the matched Partial RoPE reconstruction for every model, backend, and task combination\.

## Experiments

### Experimental Setup

#### Models and conversion methods\.

We study two model families and two draft sizes per family\. Llama\-1B and Llama\-3B draft for Llama\-8B, while Qwen\-1\.5B and Qwen\-3B draft for Qwen\-7B\. For each pair, we construct MLA drafts with both TransMLA and MHA2MLA\. This design tests whether the proposed objective transfers across model families, compression pipelines, and draft capacities rather than depending on one converter\-specific parameterization\.

#### Comparative methods\.

We report three draft forms\.*Original GQA*is the unconverted draft and serves as a reference for the acceptance available before introducing an MLA bottleneck\.*Partial RoPE reconstruction*is the matched converter\-local baseline\.*Functional Reconstruction \(ours\)*begins from the same converted initialization and retains exactly the same attention topology, latent\-cache budget, and inference interface, but optimizes Eq\. \([11](https://arxiv.org/html/2607.27269#Sx4.E11)\)\. Consequently, the Partial RoPE–Functional comparison isolates the reconstruction objective under a fixed model, converter, backend, and task\. Partial and Functional Reconstruction update the same converter\-introduced query and KV projections; output projections and non\-attention parameters remain frozen\.

### Evaluation Protocol

#### Benchmarks and decoding\.

We evaluate HumanEval\(Chen and others[2021](https://arxiv.org/html/2607.27269#bib.bib27)\), Alpaca\(Taoriet al\.[2023](https://arxiv.org/html/2607.27269#bib.bib28); Wanget al\.[2023](https://arxiv.org/html/2607.27269#bib.bib29)\), Natural Questions \(NQ\)\(Kwiatkowski and others[2019](https://arxiv.org/html/2607.27269#bib.bib30)\), and CNN/DailyMail \(CNN/DM\)\(Hermannet al\.[2015](https://arxiv.org/html/2607.27269#bib.bib31); Nallapatiet al\.[2016](https://arxiv.org/html/2607.27269#bib.bib32)\), representing code generation, instruction following, open\-domain question answering, and summarization\. Each benchmark uses a fixed 200\-prompt manifest and a maximum of 128 generated tokens\. HumanEval contains its 164 canonical problems plus 36 deterministic repeats to match the common sample count; the other manifests contain 200 unique prompts\. All runs use BF16, batch size one, greedy decoding, and seed 42\. The proposal length isγ=4\\gamma=4for the 1B/1\.5B drafts andγ=3\\gamma=3for the 3B drafts\.

#### Systems and metrics\.

We evaluate Hugging Face \(HF\) and vLLM implementations using token acceptance and generated\-output tok/s\. Partial and Functional Reconstruction use identical runtime configurations\. vLLM uses the supported converter\-specific tensor parallelism \(TP2 for TransMLA and TP1 for MHA2MLA\), so absolute throughput is not compared across converters\. The benchmark contains 192 method–task configurations and 38,400 generations, with 200 valid outputs per configuration\. For reporting, differences within 0\.5 percentage points \(pp\) in acceptance or 0\.5 tok/s are treated as practically unchanged; these are descriptive tolerances, not confidence intervals\.

## Results and Analysis

Table 1:Complete results over all 192 method–task configurations \(200 prompts per task\)\. Cells report acceptance \(%\) / output tok/s\. Original, Partial, and Ours denote unconverted GQA, Partial RoPE, and Functional Reconstruction, respectively\. Bold marks Ours only when it exceeds matched Partial by more than 0\.5 pp in acceptance or 0\.5 tok/s in throughput; smaller differences are left unbolded\. Comparisons are within converter–backend blocks\.### Matched\-Cell Effectiveness

Table[1](https://arxiv.org/html/2607.27269#Sx6.T1)reports detailed results for every evaluated configuration\. Partial RoPE and Functional Reconstruction are compared within each fixed model–converter–backend–task cell\. Under the reporting tolerance, 37 of 64 cells show a material acceptance improvement, 26 are practically unchanged, and one materially decreases\. Thus, 37 of the 38 changes outside the tolerance favor Functional Reconstruction\.

The sole material decrease is−0\.55\-0\.55pp on CNN/DM for Qwen\-3B with TransMLA under vLLM\. The largest improvements are\+4\.23\+4\.23pp on CNN/DM \(Llama\-1B, TransMLA, vLLM\),\+3\.92\+3\.92pp on HumanEval and\+3\.78\+3\.78pp on NQ \(Llama\-1B, TransMLA, HF\), and\+3\.72\+3\.72pp on Alpaca \(Llama\-1B, TransMLA, vLLM\)\. Every Partial–Ours pair holds latent rank, retained\-RoPE width, cache budget, proposal length, and inference graph fixed\. The material differences therefore isolate the reconstruction objective within the same compressed MLA parameterization rather than an increase in draft capacity\.

### Cross\-Factor Analysis

#### Task breadth reflects a shared functional correction\.

All four benchmarks contain material gains, but the small differences in their counts are better explained by model–converter interactions than by task identity\. HumanEval contains eight material improvements and eight practically unchanged cells; Alpaca and NQ each contain 10 improvements and six unchanged cells; CNN/DM contains nine improvements, six unchanged cells, and one borderline decrease\. Each task receives the same four improvements from Llama–TransMLA and the same four unchanged comparisons from Llama–MHA2MLA\. The remaining variation comes entirely from the Qwen paths\. Thus, no single domain—code, instruction following, open\-domain QA, or summarization—accounts for the overall result\. This task breadth is expected from a reconstruction objective trained on calibration hidden states rather than benchmark labels: it corrects a shared attention\-block mapping, and the benefit transfers when a downstream task visits the corrected state region\. The isolated CNN/DM deviation is only 0\.05 pp beyond the descriptive tolerance and does not constitute a repeated task\-level regression\.

#### Model family and converter exhibit a clear interaction\.

Functional Reconstruction is applicable to both converters, but its measurable headroom is converter\- and family\-dependent\. All 16 Llama–TransMLA cells improve, by\+0\.68\+0\.68to\+4\.23\+4\.23pp, whereas all 16 Llama–MHA2MLA cells are unchanged task by task\. Qwen shows the complementary pattern: Qwen–MHA2MLA improves in 15 of 16 cells, with the remaining cell within tolerance, while Qwen–TransMLA improves in six, leaves nine unchanged, and contains the borderline CNN/DM deviation\. This cross\-over rules out a converter\-specific explanation for the method: the same post\-WOW\_\{O\}objective helps TransMLA most consistently for Llama and MHA2MLA most consistently for Qwen\. It also rules out the stronger claim that conversion method is irrelevant\. Rather, converter initialization determines which functional residual remains inside the trainable query and KV projections\. An exact Llama–MHA2MLA tie means that the refinement does not alter enough proposal rankings to change acceptance under this protocol; the broad Qwen–MHA2MLA and Llama–TransMLA gains indicate residual errors that the same training\-only objective can remove\.

#### Backend dependence is localized rather than universal\.

Under HF, 20 cells materially improve and 12 remain within tolerance; under vLLM, 17 improve, 14 remain within tolerance, and one is borderline\. The aggregate difference is concentrated in Qwen–TransMLA\. For HF, Qwen\-1\.5B–TransMLA is unchanged on all four tasks, whereas Qwen\-3B–TransMLA improves on all four\. Under vLLM, only Alpaca improves for Qwen\-1\.5B–TransMLA, and only NQ improves for Qwen\-3B–TransMLA\. In contrast, the Llama pattern is invariant across backends—TransMLA improves and MHA2MLA ties—and Qwen–MHA2MLA improves in all eight HF cells and seven of eight vLLM cells\. The objective therefore transfers across both runtime implementations, but the realized operator can limit how much of the learned correction survives deployment\. This is especially relevant because vLLM uses TP2 for TransMLA and TP1 for MHA2MLA; the absolute numbers are not a controlled cross\-converter comparison\.

#### Draft size alone does not explain the gains\.

The 1B/1\.5B partition contains 16 material improvements and 16 unchanged cells, while the 3B partition contains 21 improvements, 10 unchanged cells, and the borderline deviation\. This apparent size difference is family\-specific: Llama\-1B and Llama\-3B each contribute exactly eight improvements and eight ties, whereas Qwen\-1\.5B contributes eight improvements and eight unchanged cells and Qwen\-3B contributes 13 improvements, two unchanged cells, and the borderline point\. The additional 3B gains therefore come from Qwen’s interaction with the two converter initializations, not from a general scaling law\. Moreover, the smaller and 3B drafts useγ=4\\gamma=4andγ=3\\gamma=3, respectively, so their absolute acceptances are not directly comparable\. Within\-cell Partial–Ours differences remain the controlled evidence\.

#### Recovery relative to Original identifies the method boundary\.

The Original GQA rows diagnose conversion damage but are not formal upper bounds\. Qwen–MHA2MLA exhibits a moderate, recoverable gap: on CNN/DM with HF, Qwen\-3B changes from 58\.82% \(Original\) to 54\.00% \(Partial\) and 56\.29% \(Ours\), and on Alpaca with vLLM it changes from 78\.63% to 75\.59% and 77\.59%\. Llama–TransMLA also shows partial recovery from larger losses: Llama\-1B on HumanEval with HF changes from 77\.75% to 67\.61% and 71\.53%, while its CNN/DM vLLM cell changes from 74\.53% to 43\.54% and 47\.77%\. However, initial damage and recoverability are not monotonic\. For Qwen\-3B–TransMLA with vLLM, NQ changes from 71\.03% to 10\.95% and 12\.73%; the gain is real but small relative to the conversion loss\. Functional matching can correct approximation error that remains representable by a valid converted block, but it cannot create missing latent rank or repair an incompatible tensor\-parallel or kernel realization\. This distinction explains both the broad matched gains and the severe gaps that remain\.

### Throughput and the Quality–Efficiency Trade\-off

Functional Reconstruction is training\-only and introduces no additional inference operation, parameter branch, or cache state\. Under the throughput reporting tolerance, 12 cells are materially faster, 50 are practically unchanged, and two are materially slower; changes outside the tolerance range from−1\.01\-1\.01to\+2\.77\+2\.77tok/s\. Among the 37 cells with a material acceptance gain, 12 are also materially faster, 24 are unchanged in throughput, and one is slower\. Without repeated timing runs, the individual tok/s differences should be treated as a combination of acceptance effects and run\-level variation\. The controlled conclusion is that improved acceptance does not require an added inference operation or a changed cache budget\.

The present measurements do not establish an end\-to\-end speedup over the unconverted draft\. Absolute Original GQA–MLA throughput differences combine model fidelity with GQA/MLA kernel maturity, converter\-specific tensor parallelism, and backend integration\. Optimized MLA kernels and a common runtime configuration remain necessary to translate the model\-side gain into a clean system\-level speedup over native GQA\.

## Discussion

#### \(1\) Conversion and drafting define different success criteria\.

The results expose three nested notions of success\.*Structural validity*asks whether an MHA/GQA checkpoint has been mapped to an executable MLA parameterization with the intended latent rank, RoPE path, and cache budget\.*Functional fidelity*asks whether that parameterization realizes the same attention\-block mapping on states encountered by the model\.*Draft utility*asks whether the resulting token distribution agrees with a separate verifier closely enough to produce long accepted runs\. A converter can satisfy the first criterion while failing the latter two: the severe absolute acceptance losses in several executable TransMLA paths make this separation explicit\. Conversely, the matched Partial–Ours gains change neither structure nor inference cost, demonstrating that structural conversion does not exhaust the optimization problem\.

This hierarchy also clarifies what should be preserved\. Factor\-level error inQQ,KK, orVVis only an indirect proxy because attention composes low\-rank projections, RoPE, softmax, value aggregation, andWOW\_\{O\}before affecting the residual stream\. Small errors can cancel, while a localized logit perturbation can cross a token\-ranking margin and sharply reduce acceptance\. Conversion should therefore be evaluated not only by cache validity or standalone generation, but also by an interface\-level functional metric and matched draft acceptance\. The Original GQA draft is a useful semantic anchor, not a formal acceptance upper bound: speculative utility is agreement with the target model, not self\-fidelity to the pre\-conversion draft\. This distinction is the central reason that MLA conversion and MLA draft construction should be treated as related but separate tasks\.

#### \(2\) Functional reconstruction provides a training\-time control knob for draft quality\.

The broad gains across code, instruction following, QA, and summarization show that draft usefulness can be improved offline without labels from any of those tasks and without placing the verifier in the optimization graph\. Post\-WOW\_\{O\}matching is important here because it trains against the observable output of the complete attention transformation rather than against one converter\-specific factorization\. Calibration hidden states supply the operating distribution, and the frozen original block supplies a stable functional reference\. In this sense, Functional Reconstruction is not another cache design; it searches for a better function inside the fixed function class and memory budget established by conversion\.

Acceptance nevertheless remains a discrete, end\-to\-end consequence of many layers, whereas Eq\. \([11](https://arxiv.org/html/2607.27269#Sx4.E11)\) is a continuous, layer\-local surrogate\. The borderline negative cell and the unrecovered gaps are therefore informative: reducing local block error is generally useful, but need not move the final token ranking monotonically\. Later layers may attenuate a residual, amplify it, or project it onto a direction with little decision relevance\. This suggests a principled path beyond uniform MSE, including layer\-sensitivity weighting, token\-margin\-aware reconstruction, or short sequence\-level functional matching, while retaining a verifier\-free training stage\. Crucially, such refinement adds no inference branch or cache state\. It can improve the acceptance term in speculative speedup while leaving draft cost fixed; whether that gain becomes wall\-clock speedup remains a separate kernel and runtime question\.

#### \(3\) Generality comes from the functional interface\.

The Llama/Qwen cross\-over in Table[1](https://arxiv.org/html/2607.27269#Sx6.T1)is particularly revealing\. The same objective is most consistently effective for Llama–TransMLA and Qwen–MHA2MLA, while Llama–MHA2MLA has little measurable headroom and Qwen–TransMLA is more backend\-sensitive\. Thus, generality comes from placing the objective at a shared functional interface, not from assuming that every converter leaves the same error\. A converter fixes both the representable MLA function class and its initialization within that class; model architecture, rank allocation, and RoPE treatment determine the remaining approximation residual\. Functional Reconstruction can optimize the portion of that residual reachable through the converted query and KV projections, but it cannot recover information excluded by the latent rank\. Likewise, training cannot repair a backend that realizes different tensor semantics from the calibrated module\. The Qwen–TransMLA vLLM gap is therefore best read as a precondition failure for reconstruction, not evidence that a shared functional objective is tied to one converter\.

#### Implications for MLA conversion pipelines\.

These observations motivate a three\-level workflow\. First, conversion should establish a cache\-efficient parameterization and verify RoPE, tensor\-parallel, and kernel correctness\. Second, functional reconstruction should optimize the converted attention map under that fixed deployment budget\. Third, deployment validation should measure matched acceptance and backend\-specific throughput\. Keeping these levels separate makes failures diagnosable: structural tests locate representation or runtime errors, functional tests expose recoverable approximation error, and speculative evaluation measures decision\-level value\. More broadly, latent rank should be viewed as a resource constraint rather than a quality certificate\. Future converters can use functional residuals to allocate rank across layers, choose between initializations, and decide where additional reconstruction capacity yields the largest draft\-level return\.

## Limitations

Although the matrix contains 192 configurations, it remains modest in model and data scale: two model families, four draft–target pairs no larger than 8B, four 200\-prompt benchmarks, a 128\-token generation limit, and one fixed seed\. Experiments with larger targets, more architectures and converters, repeated seeds, and long\-context workloads are needed to establish how the conclusions scale beyond the present setting\. Moreover, Functional Reconstruction consistently refines Partial RoPE conversion, but it usually recovers only part of the acceptance lost relative to Original GQA and does not surpass the corresponding Original acceptance in the evaluated matrix\. The residual gap is especially large when the initial converter–backend path is severely mismatched\. Although Original GQA is not a theoretical upper bound on target agreement, this empirical gap shows that the current layer\-local objective cannot fully overcome fixed\-rank information loss or runtime realization errors\. Stronger sequence\-level reconstruction and conversion–runtime co\-design are required for full recovery\.

## Conclusion

MLA conversion and draft usefulness are different objectives\. A converted MLA checkpoint may be a valid generator but a poor speculative draft because the latent cache imposes rank constraints and RoPE handling perturbs attention logits\. E2E functional reconstruction targets this residual without changing the MLA cache or using verifier supervision\. Across 192 evaluated method–task configurations, Functional Reconstruction materially improves acceptance in 37 of 64 matched task cells, leaves 26 practically unchanged, and materially decreases one relative to Partial RoPE reconstruction\. The result spans both TransMLA and MHA2MLA, Llama and Qwen, and HF and vLLM, but the evaluation also shows that reconstruction cannot rescue a severely mismatched converter/backend path\. Functional reconstruction is therefore a general post\-conversion refinement, while converter quality and optimized MLA kernels remain prerequisites for strong speculative\-decoding speedups\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024\)Medusa: simple llm inference acceleration framework with multiple decoding heads\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[Introduction](https://arxiv.org/html/2607.27269#Sx1.p2.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1),[RoPE Handling and Acceptance](https://arxiv.org/html/2607.27269#Sx3.SSx3.p2.3)\.
- M\. Chenet al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- Z\. Chen, A\. May, R\. Svirschevski, Y\. Huang, M\. Ryabinin, Z\. Jia, and B\. Chen \(2024\)Sequoia: scalable, robust, and hardware\-aware speculative decoding\.arXiv preprint arXiv:2402\.12374\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.InAdvances in Neural Information Processing Systems,Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- T\. Dao \(2023\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- DeepSeek\-AI \(2024a\)DeepSeek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[Introduction](https://arxiv.org/html/2607.27269#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p2.1)\.
- DeepSeek\-AI \(2024b\)DeepSeek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p2.1)\.
- M\. Elhoushi, A\. Shrivastava, D\. Liskovich, B\. Hosmer, B\. Wasti, L\. Lai, A\. Mahmoud, B\. Acun, S\. Agarwal, A\. Roman, A\. A\. Aly, B\. Chen, and C\. Wu \(2024\)LayerSkip: enabling early exit inference and self\-speculative decoding\.arXiv preprint arXiv:2404\.16710\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- Y\. Fu, P\. Bailis, I\. Stoica, and H\. Zhang \(2024\)Break the sequential dependency of llm inference using lookahead decoding\.arXiv preprint arXiv:2402\.02057\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- K\. M\. Hermann, T\. Kocisky, E\. Grefenstette, L\. Espeholt, W\. Kay, M\. Suleyman, and P\. Blunsom \(2015\)Teaching machines to read and comprehend\.InAdvances in Neural Information Processing Systems,Vol\.28\.Cited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- T\. Jiet al\.\(2025\)Towards economical inference: enabling deepseek’s multi\-head latent attention in any transformer\-based llms\.arXiv preprint arXiv:2502\.14837\.Cited by:[Introduction](https://arxiv.org/html/2607.27269#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p2.1)\.
- T\. Kwiatkowskiet al\.\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.Cited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,pp\. 611–626\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InProceedings of the 40th International Conference on Machine Learning,pp\. 19274–19286\.Cited by:[Introduction](https://arxiv.org/html/2607.27269#Sx1.p2.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1),[RoPE Handling and Acceptance](https://arxiv.org/html/2607.27269#Sx3.SSx3.p2.3)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024a\)EAGLE\-2: faster inference of language models with dynamic draft trees\.arXiv preprint arXiv:2406\.16858\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2024b\)EAGLE: speculative sampling requires rethinking feature uncertainty\.arXiv preprint arXiv:2401\.15077\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024\)KIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.arXiv preprint arXiv:2402\.02750\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- F\. Meng, P\. Tang, X\. Tang, Z\. Yao, X\. Sun, and M\. Zhang \(2025\)TransMLA: mla is all you need\.arXiv preprint arXiv:2502\.07864\.Cited by:[Introduction](https://arxiv.org/html/2607.27269#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.27269#Sx2.p2.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi, C\. Shi, Z\. Chen, D\. Arfeen, R\. Abhyankar, and Z\. Jia \(2023\)SpecInfer: accelerating generative large language model serving with tree\-based speculative inference and verification\.arXiv preprint arXiv:2305\.09781\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- R\. Nallapati, B\. Zhou, C\. dos Santos, C\. Gulcehre, and B\. Xiang \(2016\)Abstractive text summarization using sequence\-to\-sequence RNNs and beyond\.InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning,pp\. 280–290\.Cited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- N\. Shazeer \(2019\)Fast transformer decoding: one write\-head is all you need\.arXiv preprint arXiv:1911\.02150\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p2.1),[MHA/GQA Attention](https://arxiv.org/html/2607.27269#Sx3.SSx1.p1.12)\.
- R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto \(2023\)Stanford alpaca: an instruction\-following LLaMA model\.Note:Stanford Center for Research on Foundation ModelsCited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in Neural Information Processing Systems\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,pp\. 13484–13508\.Cited by:[Benchmarks and decoding\.](https://arxiv.org/html/2607.27269#Sx5.SSx2.SSS0.Px1.p1.2)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2023\)Efficient streaming language models with attention sinks\.arXiv preprint arXiv:2309\.17453\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.
- J\. Zhang, J\. Wang, H\. Li, L\. Shou, K\. Chen, G\. Chen, and S\. Mehrotra \(2023a\)Draft & verify: lossless large language model acceleration via self\-speculative decoding\.arXiv preprint arXiv:2309\.08168\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p3.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023b\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.arXiv preprint arXiv:2306\.14048\.Cited by:[Related Work](https://arxiv.org/html/2607.27269#Sx2.p1.1)\.