ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

# Activation-Metric Residual Correction for Large Language Model Output Heads
Source: [https://arxiv.org/html/2608.02703](https://arxiv.org/html/2608.02703)
Şuayp Talha Kocabay Independent Researcher kocabaysuayptalha08@gmail\.com&Talha Rüzgar Akkuş Independent Researcher talharuzgarakkus@gmail\.com

###### Abstract

Weight\-only quantization substantially reduces the storage of large language model \(LLM\) transformer blocks, but practical backends often retain the final language\-modeling head \(LM\-head\) in BF16 or FP16\. Quantizing this projection naively can strongly perturb the vocabulary\-logit distribution\. We present ARCHead, a packed LM\-head compressor that combines a quantized low\-rank core, group\-wise INT4 residuals, and a low\-rank correction fitted in an activation\-derived metric\. ARCHead stores no dense BF16 head and reduces persistent LM\-head storage by 3\.7–3\.9×\\times\. On Qwen3\-8B\-Base, it uses 25\.6% of BF16 head storage while attaining 1\.007 relative perplexity; storage\-matched naive INT4 yields 1\.14–1\.16\. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0\.006–0\.007 cross\-entropy, with less than 2% throughput change in our measurements\. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched\. Code is available at[https://github\.com/suayptalha/archead](https://github.com/suayptalha/archead)\.

ARCHead: Activation\-Metric Residual Correction for Large Language Model Output Heads

Şuayp Talha KocabayIndependent Researcherkocabaysuayptalha08@gmail\.comTalha Rüzgar AkkuşIndependent Researchertalharuzgarakkus@gmail\.com

Kamer Ali YükselaiXplain, Inc\.kamer@aixplain\.com

## 1Introduction

Post\-training quantization \(PTQ\) methods such as GPTQ\(Frantaret al\.,[2023](https://arxiv.org/html/2608.02703#bib.bib1)\), AWQ\(Linet al\.,[2025](https://arxiv.org/html/2608.02703#bib.bib2)\), and bitsandbytes NF4\(Dettmerset al\.,[2023a](https://arxiv.org/html/2608.02703#bib.bib5)\)make large language models \(LLMs\) substantially easier to deploy\. Their practical implementations, however, often focus on transformer\-block weights and retain the final language\-modeling head \(LM\-head\) in BF16 or FP16\. This head maps the final hidden state directly to a logit for every vocabulary item\. Unlike errors inside a transformer block, an LM\-head error is not followed by normalization, a residual path, or another learned transformation; it reaches the softmax directly\.

This matters increasingly for large\-vocabulary models\. The BF16 output projections of Qwen3\-8B\-Base\(Yanget al\.,[2025](https://arxiv.org/html/2608.02703#bib.bib11)\)and Gemma\-4\-E4B\(Team,[2026](https://arxiv.org/html/2608.02703#bib.bib12)\)occupy approximately 1\.18 and 1\.28 GB, respectively\. Once transformer blocks have been reduced to four bits, such a projection can become one of the largest remaining dense tensors\. In our inspection of quantized Qwen3\-8B\-Base checkpoints, both AWQ and bitsandbytes NF4 retainedlm\_headas a BF16Linearlayer of shape151,936×4,096151\{,\}936\\times 4\{,\}096\.

Naive low\-bit head quantization is not an adequate remedy\. It minimizes a weight\-space reconstruction error that treats every hidden direction equally, even though the final hidden states occupy an anisotropic distribution\. A small error along a frequently activated direction can alter many vocabulary logits, whereas a larger error along a nearly inactive direction may have little effect\. The relevant objective is therefore the expected output error induced by the hidden\-state distribution\.

We introduceARCHead, a specialized, packed LM\-head compressor\. ARCHead first represents the head with quantized low\-rank factors and a group\-wise INT4 residual\. It then approximates the remaining error with a low\-rank branch fitted after an activation\-derived metric transform\. The resulting module is a drop\-in output head and stores no dense BF16 copy of the original matrix\.

Our contributions are:

- •We formulate output\-head compression in an activation\-derived metric and show that, for a fixed quantized core, the ARCHead correction is the best rank\-rrapproximation in that metric before factor quantization\.
- •We develop a packed representation whose measured state\-dictionary footprint is 25–27% of the corresponding BF16 head across five model families\.
- •We show that ARCHead avoids the large quality loss of storage\-matched naive INT4, generalizes across three output heads, and can compress the BF16 head left by AWQ and bitsandbytes at a small additional loss\.
- •We evaluate logit fidelity, downstream accuracy, calibration sensitivity, construction time, and generation throughput, and separate persistent parameter storage from backend\-dependent runtime peak memory\.

## 2Related Work

#### LLM quantization\.

GPTQ applies second\-order information during sequential weight quantization\(Frantaret al\.,[2023](https://arxiv.org/html/2608.02703#bib.bib1)\); AWQ protects activation\-salient channels\(Linet al\.,[2025](https://arxiv.org/html/2608.02703#bib.bib2)\); and SmoothQuant moves quantization difficulty between activations and weights\(Xiaoet al\.,[2023](https://arxiv.org/html/2608.02703#bib.bib3)\)\. Other approaches optimize quantization parameters\(Shaoet al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib4)\), use layer\-wise distillation\(Yaoet al\.,[2022](https://arxiv.org/html/2608.02703#bib.bib21)\), isolate outliers\(Dettmerset al\.,[2023b](https://arxiv.org/html/2608.02703#bib.bib8); Kimet al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib19)\), or introduce incoherence transforms and rotations\(Cheeet al\.,[2023](https://arxiv.org/html/2608.02703#bib.bib6); Tsenget al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib7); Ashkbooset al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib9); Liuet al\.,[2025](https://arxiv.org/html/2608.02703#bib.bib10)\)\. AQLM uses additive codes for extreme compression\(Egiazarianet al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib18)\), while LLM\.int8\(\) and NF4 provide widely used mixed\-precision representations\(Dettmerset al\.,[2022](https://arxiv.org/html/2608.02703#bib.bib16),[2023a](https://arxiv.org/html/2608.02703#bib.bib5)\)\. These methods primarily target transformer\-block linear layers; ARCHead instead addresses the large output projection that practical pipelines may leave dense\.

#### Low\-rank and residual correction\.

Low\-rank language\-model compression\(Hsuet al\.,[2022](https://arxiv.org/html/2608.02703#bib.bib24)\)and parameter\-efficient low\-rank updates\(Huet al\.,[2022](https://arxiv.org/html/2608.02703#bib.bib20)\)demonstrate that structured matrix components can be represented compactly\. ARCHead differs in both target and objective: it computes a low\-rank approximation of the*quantization residual*after transforming that residual by a metric estimated from LM\-head inputs\. This directly allocates correction capacity to directions that affect the observed logits\.

#### Realized storage\.

An intended bit width does not itself establish deployable compression: an implementation can retain a dequantized tensor or materialize one in its serialized state\. We therefore measure bytes from the actual tensors registered by the packed head and report persistent storage separately from temporary forward\-pass memory\.

## 3Method

### 3\.1Activation\-Metric Objective

Let the dense output\-head weight be𝐖∈ℝV×D\\mathbf\{W\}\\in\\mathbb\{R\}^\{V\\times D\}and let𝐇∈ℝN×D\\mathbf\{H\}\\in\\mathbb\{R\}^\{N\\times D\}contain final hidden states fromNNcalibration tokens\. For an approximation𝐖^\\widehat\{\\mathbf\{W\}\}, write𝚫=𝐖−𝐖^\\boldsymbol\{\\Delta\}=\\mathbf\{W\}\-\\widehat\{\\mathbf\{W\}\}\. The empirical squared logit error is

ℰ\\displaystyle\\mathcal\{E\}=1N​∥𝐇​𝚫⊤∥F2=Tr⁡\(𝚫​𝐂​𝚫⊤\),\\displaystyle=\\frac\{1\}\{N\}\\lVert\\mathbf\{H\}\\boldsymbol\{\\Delta\}^\{\\top\}\\rVert\_\{F\}^\{2\}=\\operatorname\{Tr\}\\\!\\left\(\\boldsymbol\{\\Delta\}\\mathbf\{C\}\\boldsymbol\{\\Delta\}^\{\\top\}\\right\),\(1\)where𝐂=𝐇⊤​𝐇/N\\mathbf\{C\}=\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}/N\. A Frobenius weight objective is the special case𝐂=𝐈\\mathbf\{C\}=\\mathbf\{I\}and ignores the activation geometry\.

ARCHead uses a damped metric transform\. With𝐂λ=𝐂\+λ​c¯​𝐈=𝐐​𝚲​𝐐⊤\\mathbf\{C\}\_\{\\lambda\}=\\mathbf\{C\}\+\\lambda\\bar\{c\}\\mathbf\{I\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}\\mathbf\{Q\}^\{\\top\}, wherec¯\\bar\{c\}is the mean diagonal of𝐂\\mathbf\{C\}, define

𝐓p=𝐐​𝚲p​𝐐⊤,𝐓p−1=𝐐​𝚲−p​𝐐⊤\.\\mathbf\{T\}\_\{p\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{p\}\\mathbf\{Q\}^\{\\top\},\\qquad\\mathbf\{T\}\_\{p\}^\{\-1\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{\-p\}\\mathbf\{Q\}^\{\\top\}\.\(2\)Atp=12p=\\tfrac\{1\}\{2\}, the induced objective is the damped form of Eq\.[1](https://arxiv.org/html/2608.02703#S3.E1); other values adjust how strongly dominant activation directions are emphasized\. We selectppon calibration data and usep=0\.75p=0\.75for Qwen3\-8B\-Base\.

### 3\.2Core and Residual Correction

![Refer to caption](https://arxiv.org/html/2608.02703v1/x1.png)Figure 1:ARCHead decomposes the dense output head into a packed quantized core𝐖d\\mathbf\{W\}\_\{d\}and an activation\-metric low\-rank correction\. The original BF16 matrix is discarded after construction\.The quantized core is

𝐖d=Q5​\(𝐀c\)​Q8​\(𝐁c\)\+QSC4​\(𝐖−𝐀c​𝐁c\),\\mathbf\{W\}\_\{d\}=Q\_\{5\}\(\\mathbf\{A\}\_\{c\}\)Q\_\{8\}\(\\mathbf\{B\}\_\{c\}\)\+Q\_\{\\mathrm\{SC4\}\}\\\!\\left\(\\mathbf\{W\}\-\\mathbf\{A\}\_\{c\}\\mathbf\{B\}\_\{c\}\\right\),\(3\)where𝐀c​𝐁c\\mathbf\{A\}\_\{c\}\\mathbf\{B\}\_\{c\}is a rank\-rcr\_\{c\}approximation,QbQ\_\{b\}denotesbb\-bit group quantization, and SC4 denotes the packed signed\-INT4 residual with quantized group scales\. The residual left by this core is𝐄=𝐖−𝐖d\\mathbf\{E\}=\\mathbf\{W\}\-\\mathbf\{W\}\_\{d\}\.

ARCHead computes a rank\-rrr\_\{r\}randomized truncated SVD

𝐄𝐓p≈𝐔rr​𝚺rr​𝐕rr⊤\\mathbf\{E\}\\mathbf\{T\}\_\{p\}\\approx\\mathbf\{U\}\_\{r\_\{r\}\}\\boldsymbol\{\\Sigma\}\_\{r\_\{r\}\}\\mathbf\{V\}\_\{r\_\{r\}\}^\{\\top\}\(4\)and maps the right factor back:

𝐀w=𝐔rr​𝚺rr,𝐁w=𝐕rr⊤​𝐓p−1\.\\mathbf\{A\}\_\{w\}=\\mathbf\{U\}\_\{r\_\{r\}\}\\boldsymbol\{\\Sigma\}\_\{r\_\{r\}\},\\qquad\\mathbf\{B\}\_\{w\}=\\mathbf\{V\}\_\{r\_\{r\}\}^\{\\top\}\\mathbf\{T\}\_\{p\}^\{\-1\}\.\(5\)The final approximation and logits are

𝐖^\\displaystyle\\widehat\{\\mathbf\{W\}\}=𝐖d\+𝐀w​𝐁w,\\displaystyle=\\mathbf\{W\}\_\{d\}\+\\mathbf\{A\}\_\{w\}\\mathbf\{B\}\_\{w\},\(6\)𝐘^\\displaystyle\\widehat\{\\mathbf\{Y\}\}=𝐇𝐖d⊤\+\(𝐇𝐁w⊤\)​𝐀w⊤\.\\displaystyle=\\mathbf\{H\}\\mathbf\{W\}\_\{d\}^\{\\top\}\+\(\\mathbf\{H\}\\mathbf\{B\}\_\{w\}^\{\\top\}\)\\mathbf\{A\}\_\{w\}^\{\\top\}\.\(7\)For the packed ARCHead variant, both correction factors are stored with row/group\-wise INT8 quantization\.

#### Conditional optimality\.

For fixed𝐖d\\mathbf\{W\}\_\{d\}and invertible𝐓p\\mathbf\{T\}\_\{p\}, the unquantized factors above minimize

‖\(𝐄−𝐀𝐁\)​𝐓p‖F2\\left\\lVert\(\\mathbf\{E\}\-\\mathbf\{A\}\\mathbf\{B\}\)\\mathbf\{T\}\_\{p\}\\right\\rVert\_\{F\}^\{2\}\(8\)over all rank\-rrr\_\{r\}products𝐀𝐁\\mathbf\{A\}\\mathbf\{B\}\. This follows directly from the Eckart–Young–Mirsky theorem after changing variables to𝐄~=𝐄𝐓p\\widetilde\{\\mathbf\{E\}\}=\\mathbf\{E\}\\mathbf\{T\}\_\{p\}\. The guarantee is conditional on the fixed core and applies before correction\-factor quantization\.

#### Proof\.

Set𝐁~=𝐁𝐓p\\widetilde\{\\mathbf\{B\}\}=\\mathbf\{B\}\\mathbf\{T\}\_\{p\}\. Because𝐓p\\mathbf\{T\}\_\{p\}is invertible,rank⁡\(𝐀​𝐁~\)=rank⁡\(𝐀𝐁\)\\operatorname\{rank\}\(\\mathbf\{A\}\\widetilde\{\\mathbf\{B\}\}\)=\\operatorname\{rank\}\(\\mathbf\{A\}\\mathbf\{B\}\), so the problem is equivalent to

minrank⁡\(𝐀​𝐁~\)≤rr⁡‖𝐄~−𝐀​𝐁~‖F2\.\\min\_\{\\operatorname\{rank\}\(\\mathbf\{A\}\\widetilde\{\\mathbf\{B\}\}\)\\leq r\_\{r\}\}\\left\\lVert\\widetilde\{\\mathbf\{E\}\}\-\\mathbf\{A\}\\widetilde\{\\mathbf\{B\}\}\\right\\rVert\_\{F\}^\{2\}\.\(9\)By the Eckart–Young–Mirsky theorem, the minimizer is the rank\-rrr\_\{r\}truncated SVD𝐔rr​𝚺rr​𝐕rr⊤\\mathbf\{U\}\_\{r\_\{r\}\}\\boldsymbol\{\\Sigma\}\_\{r\_\{r\}\}\\mathbf\{V\}\_\{r\_\{r\}\}^\{\\top\}of𝐄~\\widetilde\{\\mathbf\{E\}\}\. Choosing𝐀=𝐔rr​𝚺rr\\mathbf\{A\}=\\mathbf\{U\}\_\{r\_\{r\}\}\\boldsymbol\{\\Sigma\}\_\{r\_\{r\}\}and𝐁=𝐕rr⊤​𝐓p−1\\mathbf\{B\}=\\mathbf\{V\}\_\{r\_\{r\}\}^\{\\top\}\\mathbf\{T\}\_\{p\}^\{\-1\}yields the stated factors\. Forp=12p=\\tfrac\{1\}\{2\}, the objective is the damped empirical logit MSE\. Our randomized truncated SVD and subsequent factor quantization approximate this ideal correction, so the result does not assert global optimality of the quantized end\-to\-end module\.

### 3\.3Packed Representation and Integration

The packedARCHeadregisters only the SC4 residual and scales, quantized core factors, quantized correction factors, and shape metadata\. Its persistent size is measured directly as

bytes⁡\(ARCHead\)\\displaystyle\\operatorname\{bytes\}\(\\mathrm\{ARCHead\}\)=∑inumel⁡\(θi\)\\displaystyle=\\sum\_\{i\}\\operatorname\{numel\}\(\\theta\_\{i\}\)\(10\)×element​\_​size⁡\(θi\),\\displaystyle\\quad\{\}\\times\\operatorname\{element\\\_size\}\(\\theta\_\{i\}\),over all parameters and buffersθi\\theta\_\{i\}in the serialized module\. The originalV×DV\\times DBF16 matrix is not registered\. During model conversion, the dense head is used to construct ARCHead and is then discarded\. The module can replace the output embedding after block quantization, so ARCHead is complementary to AWQ, bitsandbytes, or another block backend\. Algorithm[1](https://arxiv.org/html/2608.02703#alg1)gives the complete construction procedure\.

#### State\-dictionary accounting\.

The packed implementation is not merely a theoretical bit\-counting scheme: unlike prototypes that retain dequantized dense tensors at load time, it stores no dense BF16V×DV\\times Dhead\. Dense storage isV​D×2VD\\times 2bytes, whereas packed storage is the measured sum above over the registeredstate\_dict\. The stored components are the packed INT4 residual and its FP16/BF16 scales; the 5\-bit left and 8\-bit right core factors and their scales; row/group\-wise INT8 correction factors𝐀w\\mathbf\{A\}\_\{w\}and𝐁w\\mathbf\{B\}\_\{w\}; quantization scales and zero\-point metadata; and group\-size and shape metadata\. Table[1](https://arxiv.org/html/2608.02703#S3.T1)reports values measured from instantiated PyTorch buffers rather than theoretical bit widths\.

Table 1:Detailed packedARCHeadstorage validation, calculated from the instantiated PyTorchstate\_dictbuffers\.
#### Load\-time versus peak memory\.

Persistent LM\-head memory measures the parameter allocation when loading the model onto the GPU and is the appropriate quantity for evaluating the stored output\-head footprint\. Table[2](https://arxiv.org/html/2608.02703#S3.T2)reports the corresponding savings\. Forward\-pass peak memory is separate: it includes activations, output logits, temporary buffers, CUDA allocator behavior, backend workspaces, and the KV cache\. ARCHead reduces persistent output\-head parameters; we do not conflate this with backend\- and workload\-dependent runtime peak memory\.

Table 2:Load\-time persistent LM\-head memory savings\.
#### Peak\-memory measurement protocol\.

Before measurement, we executetorch\.cuda\.empty\_cache\(\), reset PyTorch peak\-memory statistics, and synchronize the CUDA device before and after execution\. We measure and report load\-time persistent parameter memory separately from the forward\-pass peak, which inherently includes dynamic logits, temporary buffers, allocator state, and backend workspaces; consequently, we avoid unsupported claims about backend\-dependent peak\-memory reductions\.

### 3\.4Construction Algorithm

The inputs are the dense LM\-head weight𝐖∈ℝV×D\\mathbf\{W\}\\in\\mathbb\{R\}^\{V\\times D\}, hidden calibration activations𝐇∈ℝN×D\\mathbf\{H\}\\in\\mathbb\{R\}^\{N\\times D\}, core and correction ranksrcr\_\{c\}andrrr\_\{r\}, group sizegg, metric powerpp, and dampingλ\\lambda\. The procedure first estimates the damped activation covariance𝐂=𝐇⊤​𝐇/N\+λ​𝐈\\mathbf\{C\}=\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}/N\+\\lambda\\mathbf\{I\}\. It then constructs the quantized core𝐖d\\mathbf\{W\}\_\{d\}by quantizing a low\-rank approximation of𝐖\\mathbf\{W\}and applying group\-wise low\-bit quantization to the remaining residual\.

Next, ARCHead computes the core error𝐄=𝐖−𝐖d\\mathbf\{E\}=\\mathbf\{W\}\-\\mathbf\{W\}\_\{d\}, eigendecomposes𝐂=𝐐​𝚲​𝐐⊤\\mathbf\{C\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}\\mathbf\{Q\}^\{\\top\}, and forms𝐓p=𝐐​𝚲p​𝐐⊤\\mathbf\{T\}\_\{p\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{p\}\\mathbf\{Q\}^\{\\top\}and its inverse\. The transformed residual𝐄~=𝐄𝐓p\\widetilde\{\\mathbf\{E\}\}=\\mathbf\{E\}\\mathbf\{T\}\_\{p\}is truncated to rankrrr\_\{r\}; mapping its right factor through𝐓p−1\\mathbf\{T\}\_\{p\}^\{\-1\}produces𝐀w\\mathbf\{A\}\_\{w\}and𝐁w\\mathbf\{B\}\_\{w\}\. We store𝐀w\\mathbf\{A\}\_\{w\}with row\-wise INT8 quantization and𝐁w\\mathbf\{B\}\_\{w\}with group\-wise INT8 quantization, pack every component, and discard the dense BF16 tensor\. Inference then evaluates𝐘=𝐇𝐖d⊤\+\(𝐇𝐁w⊤\)​𝐀w⊤\\mathbf\{Y\}=\\mathbf\{H\}\\mathbf\{W\}\_\{d\}^\{\\top\}\+\(\\mathbf\{H\}\\mathbf\{B\}\_\{w\}^\{\\top\}\)\\mathbf\{A\}\_\{w\}^\{\\top\}\.

Algorithm 1ARCHead compression0:Dense head

𝐖\\mathbf\{W\}, activations

𝐇\\mathbf\{H\}, ranks

rc,rrr\_\{c\},r\_\{r\}, group size

gg, metric power

pp, damping

λ\\lambda
0:PackedARCHeadmodule

1:

𝐂←𝐇⊤​𝐇/N\+λ​𝐈\\mathbf\{C\}\\leftarrow\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}/N\+\\lambda\\mathbf\{I\}
2:

𝐖d←QuantizeCore​\(𝐖,rc,g\)\\mathbf\{W\}\_\{d\}\\leftarrow\\text\{QuantizeCore\}\(\\mathbf\{W\},r\_\{c\},g\)
3:

𝐄←𝐖−𝐖d\\mathbf\{E\}\\leftarrow\\mathbf\{W\}\-\\mathbf\{W\}\_\{d\}
4:

𝐐,𝚲←Eigendecompose​\(𝐂\)\\mathbf\{Q\},\\boldsymbol\{\\Lambda\}\\leftarrow\\text\{Eigendecompose\}\(\\mathbf\{C\}\)
5:

𝐓p←𝐐​𝚲p​𝐐⊤\\mathbf\{T\}\_\{p\}\\leftarrow\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{p\}\\mathbf\{Q\}^\{\\top\}
6:

𝐄~←𝐄𝐓p\\widetilde\{\\mathbf\{E\}\}\\leftarrow\\mathbf\{E\}\\mathbf\{T\}\_\{p\}
7:

𝐔,𝚺,𝐕←TruncatedSVD​\(𝐄~,rr\)\\mathbf\{U\},\\boldsymbol\{\\Sigma\},\\mathbf\{V\}\\leftarrow\\text\{TruncatedSVD\}\(\\widetilde\{\\mathbf\{E\}\},r\_\{r\}\)
8:

𝐀w←𝐔​𝚺\\mathbf\{A\}\_\{w\}\\leftarrow\\mathbf\{U\}\\boldsymbol\{\\Sigma\};

𝐁w←𝐕⊤​𝐓p−1\\mathbf\{B\}\_\{w\}\\leftarrow\\mathbf\{V\}^\{\\top\}\\mathbf\{T\}\_\{p\}^\{\-1\}
9:

\(𝐀w,𝐁w\)←QuantizeINT8​\(𝐀w,𝐁w\)\(\\mathbf\{A\}\_\{w\},\\mathbf\{B\}\_\{w\}\)\\leftarrow\\text\{QuantizeINT8\}\(\\mathbf\{A\}\_\{w\},\\mathbf\{B\}\_\{w\}\)
10:return

PackBuffers​\(𝐖d,𝐀w,𝐁w\)\\text\{PackBuffers\}\(\\mathbf\{W\}\_\{d\},\\mathbf\{A\}\_\{w\},\\mathbf\{B\}\_\{w\}\)

## 4Experimental Setup

#### Models\.

We evaluate head quality on Qwen3\-8B\-Base\(Yanget al\.,[2025](https://arxiv.org/html/2608.02703#bib.bib11)\), Gemma\-4\-E4B\(Team,[2026](https://arxiv.org/html/2608.02703#bib.bib12)\), and VibeThinker\-3B\(Xuet al\.,[2026](https://arxiv.org/html/2608.02703#bib.bib13)\)\. Their\(V,D\)\(V,D\)dimensions are\(151,936,4096\)\(151\{,\}936,4096\),\(262,144,2560\)\(262\{,\}144,2560\), and\(151,936,2048\)\(151\{,\}936,2048\)\. Packed\-storage validation additionally includes Mistral\-7B\-v0\.3\(Jianget al\.,[2023](https://arxiv.org/html/2608.02703#bib.bib15)\)and LFM2\.5\-8B\-A1B\(AI,[2026](https://arxiv.org/html/2608.02703#bib.bib14)\)\.

#### Data and metrics\.

We collect calibration activations from the training split of WikiText\-103\(Merityet al\.,[2016](https://arxiv.org/html/2608.02703#bib.bib23)\)and evaluate cross\-entropy \(CE\) on 16,384 held\-out test tokens\. We report perplexity \(PPL\), relative PPL \(compressed PPL divided by dense PPL\),Δ\\DeltaCE, logit MSE, KL divergence, dense\-prediction agreement, and persistent head bytes\. For the GPTQ\-head comparison, evaluation activations are fixed and disjoint from calibration; each 2K–16K budget uses three matched calibration seeds\.

#### Baselines and implementation\.

Head\-only baselines are BF16, row\-wise INT8, group INT4, SVD8\+INT4, and a GPTQ\-style INT4 output head\. Transformer blocks remain BF16 unless a row explicitly names AWQ or bitsandbytes NF4\. On Qwen, ARCHead usesrc=10r\_\{c\}=10,rr=6r\_\{r\}=6, groups of 64,p=0\.75p=0\.75, and ridge10−310^\{\-3\}\. Primary deployment measurements use one NVIDIA RTX Pro 6000\.

#### Reproducibility protocol\.

We use the checkpointsQwen/Qwen3\-8B\-Base,google/gemma\-4\-E4B,WeiboAI/VibeThinker\-3B,mistralai/Mistral\-7B\-v0\.3, andLiquidAI/LFM2\.5\-8B\-A1B\. Calibration activations come from the training split ofSalesforce/wikitext\-103\-raw\-v1; quality is evaluated on a fixed 16,384\-token test set\. The matched GPTQ\-head comparison uses 2,048, 4,096, 8,192, and 16,384 calibration tokens with seeds 0, 1, and 2; the broader ARCHead sensitivity sweep uses 4,096–65,536 tokens\. AutoAWQ and bitsandbytes are used for the full\-model hybrid experiments\. GPTQ is evaluated as a standalone head\-only INT4 baseline with the same activation cache and evaluation protocol as ARCHead; it is omitted from the hybrid experiments because of backend compatibility constraints\. Construction times are measured on the same NVIDIA RTX Pro 6000 and reported as paired relative speedups\. Code is available at[https://github\.com/suayptalha/archead](https://github.com/suayptalha/archead)\.

## 5Results

### 5\.1Head\-Only Quality

Table 3:Qwen3\-8B\-Base head\-only results on WikiText\-103\. Ratio is persistent head storage relative to BF16\.At nearly the same storage, Group INT4 raises relative PPL to 1\.151, and adding an ordinary low\-rank component does not repair the loss \(Table[3](https://arxiv.org/html/2608.02703#S5.T3)\)\. ARCHead instead remains near the dense head at 1\.007 relative PPL\. The ablation in Section[5\.6](https://arxiv.org/html/2608.02703#S5.SS6)shows that the improvement does not come from the quantized core alone\.

Across architectures, ARCHead obtains relative PPL values of 1\.007, 1\.010, and 1\.027 on Qwen, Gemma, and VibeThinker, respectively \(Table[4](https://arxiv.org/html/2608.02703#S5.T4)\)\. The gain over naive INT4 is largest on Qwen and narrower on Gemma, whose group\-INT4 head is already strong\. This variation argues against assuming that every output head has the same sensitivity\.

Table 4:Head\-only results across model families\. The bold entry marks the lower relative PPL when the displayed rounded values differ materially\.
### 5\.2Compressing Heads Left by Block Quantizers

AWQ and bitsandbytes NF4 both retained a 1\.18 GB BF16 head in our Qwen checkpoint\. Replacing only this head with ARCHead reduced its stored size to 25\.6% of BF16\. The extra CE over each already\-quantized model was 0\.006 for AWQ and 0\.007 for NF4 \(Table[5](https://arxiv.org/html/2608.02703#S5.T5)\)\. ARCHead is not claimed to improve the block quantizer; it recovers most of the persistent storage occupied by the dense output projection at a small additional loss\.

#### Backend inspection\.

We verified the retained head directly\. Under bitsandbytes NF4,lm\_headremains atorch\.nn\.Linearwith BF16 weights of shape\[151936, 4096\]oncuda:0\. Under AWQ it likewise remains a BF16torch\.nn\.Linearof the same shape; it is on CPU immediately after loading and can be moved to CUDA for evaluation\. Thus, deployments described as fully quantized can still contain a dense BF16 output projection, and ARCHead operates orthogonally to their transformer\-block quantization\.

Table 5:ARCHead replaces the dense head left by two block\-quantization backends on Qwen3\-8B\-Base\.

### 5\.3Packed Storage Is Realized

Summing the tensors actually registered inARCHeadyields 3\.71–3\.91×\\timescompression across five heads \(Table[6](https://arxiv.org/html/2608.02703#S5.T6)\)\. This is a persistent state\-dictionary and load\-time parameter reduction, not a claim about total forward\-pass peak VRAM\. The latter also includes logits, activations, allocator state, backend workspaces, and the KV cache\.

Table 6:Measured persistent LM\-head storage\. Values are sums over actual packed buffers\.
### 5\.4Comparison with GPTQ\-Style Head Quantization

We compare ARCHead with a GPTQ\-style INT4 quantizer applied only to the Qwen LM\-head\. Both methods use the same calibration activations and fixed test activations, and their effective sizes are comparable \(25\.61% for ARCHead and 25\.78% for GPTQ\)\. Table[7](https://arxiv.org/html/2608.02703#S5.T7)reports means and standard deviations over three matched calibration seeds\.

Table 7:Three\-seed, head\-only comparison on Qwen3\-8B\-Base\. Build speedup is GPTQ time divided by ARCHead time\.ARCHead has lower mean relative PPL and logit MSE at every tested budget\. Measured as the excess above dense relative PPL, it reduces GPTQ’s degradation by 44\.0%, 49\.3%, 33\.6%, and 40\.6% from 2K through 16K tokens\. It also constructs the head 2\.36–2\.58×\\timesfaster in this implementation\. At 8K, GPTQ is better on one of three individual seeds, but ARCHead retains the lower mean and markedly lower variance; at the other budgets ARCHead is better on all three seeds\.

The methods use activation information differently\. GPTQ commits sequential discrete weight updates and propagates their error through an inverse\-Hessian factor\. ARCHead first fixes a quantized core, then globally allocates rank\-rrr\_\{r\}correction capacity to the dominant modes of the entire vocabulary\-wide residual\. The conditional optimum in Section 3\.2 explains the specific theoretical advantage of this stage: among rank\-rrr\_\{r\}corrections to that core, no other unquantized correction has lower error in ARCHead’s chosen activation metric\. The empirical result shows that this global residual repair is more effective than our storage\-matched GPTQ\-head implementation in the tested regime; it is not a claim that ARCHead dominates GPTQ for arbitrary layers or settings\.

### 5\.5Logit Fidelity

Table 8:Fidelity to dense Qwen logits\. Top\-1 is agreement in percent\. ARCHead is compared with storage\-matched low\-bit baselines; INT8 uses roughly twice its storage\.ARCHead preserves the dense top\-1 token on 93\.05% of positions, compared with approximately 76% for the storage\-matched baselines \(Table[8](https://arxiv.org/html/2608.02703#S5.T8)\)\. It also reduces their KL divergence by more than an order of magnitude\. Top\-5 agreement reaches 99\.95% and top\-10 agreement 100\.00%\. Row INT8 remains more accurate, but uses approximately twice the head storage\.

### 5\.6Ablation: The Correction Is Decisive

Table 9:Qwen head\-only ablation\. Removing the activation\-metric correction accounts for most of the quality loss\.The quantized core alone yields 1\.134 relative PPL \(Table[9](https://arxiv.org/html/2608.02703#S5.T9)\)\. Adding the rank\-6 activation\-metric correction lowers this to 1\.007 for a 2\.6\-point increase in storage ratio\. Thus, the low\-rank core is not sufficient: the decisive component is the correction fitted to the core’s remaining, activation\-weighted error\.

### 5\.7Downstream Sanity Check

We evaluate the Qwen3\-8B\-Base head variants withlm\-evaluation\-harness\(Gaoet al\.,[2024](https://arxiv.org/html/2608.02703#bib.bib32)\)on HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2608.02703#bib.bib29)\), TruthfulQA MC2\(Linet al\.,[2022](https://arxiv.org/html/2608.02703#bib.bib30)\), and WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2608.02703#bib.bib31)\)\. Transformer blocks and evaluation settings are fixed across heads\. The differences are small, and the suite is too limited to support a claim of downstream improvement; we use it only to check for an obvious regression after replacing the output head\.

Table 10:Downstream accuracy sanity check\. Small differences should not be interpreted as statistically established task\-level gains\.ARCHead matches the dense head on HellaSwag and remains within the small spread among head variants on TruthfulQA MC2 and WinoGrande \(Table[10](https://arxiv.org/html/2608.02703#S5.T10)\)\. These results reveal no obvious regression, but they do not establish task\-level gains or preservation of every model capability\.

### 5\.8Efficiency and Calibration

Generation throughput changes by less than 2% in all measured configurations\. Qwen throughput is 4468 versus 4467 tokens/s for BF16 and ARCHeads, 3452 versus 3451 with AWQ blocks, and 1789 versus 1813 with NF4 blocks; VibeThinker measures 7091 versus 6947\. Small positive differences are treated as measurement noise, not speedups\. A preliminary fused\-kernel evaluation gives the same conclusion \(Appendix[L](https://arxiv.org/html/2608.02703#A12)\)\.

ARCHead’s Qwen relative PPL remains between 1\.0070 and 1\.0076 as calibration size varies from 4K to 64K tokens \(Table[11](https://arxiv.org/html/2608.02703#S5.T11)\)\. This suggests that the dominant hidden\-space directions can be estimated with a modest sample for this model, but we do not claim calibration\-size invariance in general\. A few thousand tokens suffice to estimate the covariance accurately in this experiment; this is not a claim of universal calibration independence\.

Table 11:Calibration sensitivity sweep on Qwen3\-8B\-Base\.

## 6Conclusion

ARCHead addresses a concrete gap in quantized LLM deployment: the large BF16 output projection that can remain after transformer\-block quantization\. Its activation\-metric residual branch repairs the structured error left by a compact quantized core, while its packed module realizes a measured 3\.7–3\.9×\\timespersistent head\-storage reduction\. Across the tested heads, ARCHead provides a practical quality–storage trade\-off and can be composed with AWQ or bitsandbytes without materially changing generation throughput\. The method is intentionally complementary to full\-model quantizers rather than a replacement for them\.

## Limitations

ARCHead is specialized for output heads and does not compress transformer MLP or attention weights\. Its benefit is largest when the vocabulary projection is both dense and sensitive to ordinary low\-bit quantization; on Gemma\-4\-E4B, naive INT4 is already slightly better in the displayed rounded relative\-PPL result\. The cross\-model study covers three quality evaluations, while packed size is checked on five heads; broader architectures, languages, context lengths, and calibration domains remain to be tested\.

Our principal memory claim concerns serialized and load\-time parameter tensors\. Total peak GPU memory is workload\- and backend\-dependent because logits, activations, workspaces, allocator behavior, and KV caches can dominate\. The three downstream tasks are only a sanity check and do not establish preservation of every model capability or safety property\. Finally, the conditional optimality result applies to the unquantized rank\-rrcorrection for a fixed core; quantizing its factors introduces additional approximation error, and the end\-to\-end ARCHead construction is not claimed to be globally optimal\.

## Ethical Considerations

This work uses publicly released model checkpoints and WikiText\-103; it does not introduce human\-subject data collection or annotation\. Compression can lower deployment costs and thereby broaden access, but it can also make models with existing biases or unsafe behaviors easier to deploy\. ARCHead does not remove such behavior: a compressed model inherits the risks, licenses, and intended\-use constraints of its source checkpoint\. Practitioners should therefore repeat application\-specific quality and safety evaluations after compression rather than relying only on perplexity or logit fidelity\.

#### Acknowledgments\.

We thank TextCortex AI for providing GPU support for this study\.

## References

- L\. AI \(2026\)LFM2\.5\-8b\-a1b: personal assistant on your laptop\.Liquid AI Blog\.Note:www\.liquid\.ai/blog/lfm2\-5\-8b\-a1bCited by:[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px1.p1.4)\.
- S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman \(2024\)Quarot: outlier\-free 4\-bit inference in rotated llms\.Advances in Neural Information Processing Systems37,pp\. 100213–100240\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Chee, Y\. Cai, V\. Kuleshov, and C\. M\. De Sa \(2023\)Quip: 2\-bit quantization of large language models with guarantees\.Advances in neural information processing systems36,pp\. 4396–4429\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023a\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§1](https://arxiv.org/html/2608.02703#S1.p1.1),[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dettmers, R\. Svirschevski, V\. Egiazarian, D\. Kuznedelev, E\. Frantar, S\. Ashkboos, A\. Borzunov, T\. Hoefler, and D\. Alistarh \(2023b\)SpQR: a sparse\-quantized representation for near\-lossless llm weight compression\.External Links:2306\.03078,[Link](https://arxiv.org/abs/2306.03078)Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- V\. Egiazarian, A\. Panferov, D\. Kuznedelev, E\. Frantar, A\. Babenko, and D\. Alistarh \(2024\)Extreme compression of large language models via additive quantization\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. Alistarh \(2023\)GPTQ: accurate post\-training quantization for generative pre\-trained transformers\.External Links:2210\.17323,[Link](https://arxiv.org/abs/2210.17323)Cited by:[§1](https://arxiv.org/html/2608.02703#S1.p1.1),[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5\.7](https://arxiv.org/html/2608.02703#S5.SS7.p1.1)\.
- Y\. Hsu, T\. Hua, S\. Chang, Q\. Lou, Y\. Shen, and H\. Jin \(2022\)Language model compression with weighted low\-rank factorization\.External Links:2207\.00112,[Link](https://arxiv.org/abs/2207.00112)Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px1.p1.4)\.
- S\. Kim, C\. Hooper, A\. Gholami, Z\. Dong, X\. Li, S\. Shen, M\. W\. Mahoney, and K\. Keutzer \(2024\)SqueezeLLM: dense\-and\-sparse quantization\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Lin, J\. Tang, H\. Tang, S\. Yang, G\. Xiao, and S\. Han \(2025\)AWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.GetMobile: Mobile Computing and Communications28\(4\),pp\. 12–17\.Cited by:[§1](https://arxiv.org/html/2608.02703#S1.p1.1),[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Truthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 3214–3252\.Cited by:[§5\.7](https://arxiv.org/html/2608.02703#S5.SS7.p1.1)\.
- Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort \(2025\)Spinquant: llm quantization with learned rotations\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 92009–92032\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.External Links:1609\.07843,[Link](https://arxiv.org/abs/1609.07843)Cited by:[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px2.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)Winogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5\.7](https://arxiv.org/html/2608.02703#S5.SS7.p1.1)\.
- W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, G\. Peng, Y\. Qiao, and P\. Luo \(2024\)Omniquant: omnidirectionally calibrated quantization for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 45472–45496\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Team \(2026\)Gemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§1](https://arxiv.org/html/2608.02703#S1.p2.1),[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px1.p1.4)\.
- A\. Tseng, J\. Chee, Q\. Sun, V\. Kuleshov, and C\. De Sa \(2024\)Quip\#: even better llm quantization with hadamard incoherence and lattice codebooks\.Proceedings of machine learning research235,pp\. 48630\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2023\)Smoothquant: accurate and efficient post\-training quantization for large language models\.InInternational conference on machine learning,pp\. 38087–38099\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Xu, S\. Liu, W\. Wang, J\. Min, Y\. Dai, Z\. Yin, Y\. Chen, X\. Zhou, and J\. Zhang \(2026\)VibeThinker\-3b: exploring the frontier of verifiable reasoning in small language models\.External Links:2606\.16140,[Link](https://arxiv.org/abs/2606.16140)Cited by:[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px1.p1.4)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2608.02703#S1.p2.1),[§4](https://arxiv.org/html/2608.02703#S4.SS0.SSS0.Px1.p1.4)\.
- Z\. Yao, R\. Yazdani Aminabadi, M\. Zhang, X\. Wu, C\. Li, and Y\. He \(2022\)Zeroquant: efficient and affordable post\-training quantization for large\-scale transformers\.Advances in neural information processing systems35,pp\. 27168–27183\.Cited by:[§2](https://arxiv.org/html/2608.02703#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)Hellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[§5\.7](https://arxiv.org/html/2608.02703#S5.SS7.p1.1)\.

## Appendix AExtended Notation and Objective

This section collects the notation used throughout the construction and makes explicit the relationship between head\-weight error and logit error\. LetVVdenote vocabulary size,DDhidden width, andNNthe number of calibration tokens\. The dense LM\-head is𝐖∈ℝV×D\\mathbf\{W\}\\in\\mathbb\{R\}^\{V\\times D\}, the calibration activations are𝐇∈ℝN×D\\mathbf\{H\}\\in\\mathbb\{R\}^\{N\\times D\}, and the compressed head is𝐖^\\widehat\{\\mathbf\{W\}\}\. Table[12](https://arxiv.org/html/2608.02703#A1.T12)summarizes the remaining symbols\.

Table 12:Notation used in the ARCHead construction\.For an arbitrary weight error𝚫=𝐖−𝐖^\\boldsymbol\{\\Delta\}=\\mathbf\{W\}\-\\widehat\{\\mathbf\{W\}\}, the average squared logit error over the calibration activations is

1N​∥𝐇​𝚫⊤∥F2\\displaystyle\\frac\{1\}\{N\}\\lVert\\mathbf\{H\}\\boldsymbol\{\\Delta\}^\{\\top\}\\rVert\_\{F\}^\{2\}=1N​Tr⁡\(𝐇​𝚫⊤​𝚫​𝐇⊤\)\\displaystyle=\\frac\{1\}\{N\}\\operatorname\{Tr\}\\\!\\left\(\\mathbf\{H\}\\boldsymbol\{\\Delta\}^\{\\top\}\\boldsymbol\{\\Delta\}\\mathbf\{H\}^\{\\top\}\\right\)\(11\)=Tr⁡\(𝚫​𝐇⊤​𝐇N​𝚫⊤\)\.\\displaystyle=\\operatorname\{Tr\}\\\!\\left\(\\boldsymbol\{\\Delta\}\\frac\{\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}\}\{N\}\\boldsymbol\{\\Delta\}^\{\\top\}\\right\)\.\(12\)Thus, directions with greater activation energy receive greater weight\. Ordinary Frobenius reconstruction treats the covariance as the identity and cannot distinguish frequently activated directions from nearly inactive ones\.

ARCHead uses the damped covariance

𝐂λ=𝐇⊤​𝐇N\+λ​c¯​𝐈=𝐐​𝚲​𝐐⊤,\\mathbf\{C\}\_\{\\lambda\}=\\frac\{\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}\}\{N\}\+\\lambda\\bar\{c\}\\mathbf\{I\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}\\mathbf\{Q\}^\{\\top\},\(13\)wherec¯\\bar\{c\}is the mean diagonal covariance\. The transform𝐓p=𝐐​𝚲p​𝐐⊤\\mathbf\{T\}\_\{p\}=\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{p\}\\mathbf\{Q\}^\{\\top\}interpolates between an unweighted residual approximation atp=0p=0and the damped empirical logit\-MSE geometry atp=12p=\\tfrac\{1\}\{2\}\. Values above one half emphasize dominant activation directions more strongly; the Qwen configuration selectsp=0\.75p=0\.75on calibration data\. Damping keeps the transform numerically stable in weakly observed directions and makes its inverse well defined\.

#### Scope of the optimality statement\.

For a fixed core𝐖d\\mathbf\{W\}\_\{d\}, invertible𝐓p\\mathbf\{T\}\_\{p\}, and unquantized rank\-rrr\_\{r\}correction, the truncated SVD of𝐄𝐓p\\mathbf\{E\}\\mathbf\{T\}\_\{p\}is optimal in the induced metric\. This statement does not optimize the core, metric power, or damping jointly\. It also precedes INT8 factor quantization and therefore characterizes the ideal correction stage rather than the complete packed module\. These qualifications are important when interpreting the theorem as an explanation of the correction design rather than as a claim of global end\-to\-end optimality\.

## Appendix BExpanded Construction Procedure

The construction consumes a dense head only while fitting the packed replacement\. It proceeds as follows\.

1. 1\.Collect head inputs\.Run calibration text through the model and retain final hidden states immediately before the LM\-head\. The evaluation activations used for reported quality metrics remain disjoint from calibration\.
2. 2\.Estimate activation geometry\.Accumulate𝐇⊤​𝐇/N\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}/N, apply damping, and eigendecompose the resultingD×DD\\times Dmatrix\.
3. 3\.Construct the low\-rank core\.Compute the rank\-rcr\_\{c\}core factors and quantize the left and right factors with their designated precisions\.
4. 4\.Quantize the core residual\.Subtract the unquantized low\-rank core from𝐖\\mathbf\{W\}and encode the remaining matrix with signed group\-wise INT4 values and quantized scales\.
5. 5\.Measure the realized core error\.Dequantize the packed core representation for construction only and form𝐄=𝐖−𝐖d\\mathbf\{E\}=\\mathbf\{W\}\-\\mathbf\{W\}\_\{d\}\. Using the realized core error ensures that the correction targets errors introduced by both approximation and factor quantization\.
6. 6\.Fit in the activation metric\.Form𝐄𝐓p\\mathbf\{E\}\\mathbf\{T\}\_\{p\}, compute its randomized rank\-rrr\_\{r\}truncated SVD, and map the right factor back through𝐓p−1\\mathbf\{T\}\_\{p\}^\{\-1\}\.
7. 7\.Quantize and pack the correction\.Store𝐀w\\mathbf\{A\}\_\{w\}row\-wise and𝐁w\\mathbf\{B\}\_\{w\}group\-wise in INT8 together with their scales and shape metadata\.
8. 8\.Discard the dense source\.Register only the packed tensors inARCHead; the original BF16V×DV\\times Dtensor and construction temporaries are not part of the serialized module\.

Algorithm 2Expanded packed\-head construction0:

𝐖\\mathbf\{W\},

𝐇\\mathbf\{H\},

rcr\_\{c\},

rrr\_\{r\}, group size

gg,

pp,

λ\\lambda
0:Packed module with no dense BF16 head

1:

𝐂λ←𝐇⊤​𝐇/N\+λ​c¯​𝐈\\mathbf\{C\}\_\{\\lambda\}\\leftarrow\\mathbf\{H\}^\{\\top\}\\mathbf\{H\}/N\+\\lambda\\bar\{c\}\\mathbf\{I\}
2:

\(𝐐,𝚲\)←Eigh⁡\(𝐂λ\)\(\\mathbf\{Q\},\\boldsymbol\{\\Lambda\}\)\\leftarrow\\operatorname\{Eigh\}\(\\mathbf\{C\}\_\{\\lambda\}\)
3:

𝐓p←𝐐​𝚲p​𝐐⊤\\mathbf\{T\}\_\{p\}\\leftarrow\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{p\}\\mathbf\{Q\}^\{\\top\}
4:

𝐓p−1←𝐐​𝚲−p​𝐐⊤\\mathbf\{T\}\_\{p\}^\{\-1\}\\leftarrow\\mathbf\{Q\}\\boldsymbol\{\\Lambda\}^\{\-p\}\\mathbf\{Q\}^\{\\top\}
5:

𝐖d←QuantizeCore⁡\(𝐖,rc,g\)\\mathbf\{W\}\_\{d\}\\leftarrow\\operatorname\{QuantizeCore\}\(\\mathbf\{W\},r\_\{c\},g\)
6:

𝐄←𝐖−𝐖d\\mathbf\{E\}\\leftarrow\\mathbf\{W\}\-\\mathbf\{W\}\_\{d\}
7:

\(𝐔,𝚺,𝐕\)←RandomizedSVD⁡\(𝐄𝐓p,rr\)\(\\mathbf\{U\},\\boldsymbol\{\\Sigma\},\\mathbf\{V\}\)\\leftarrow\\operatorname\{RandomizedSVD\}\(\\mathbf\{E\}\\mathbf\{T\}\_\{p\},r\_\{r\}\)
8:

𝐀w←𝐔​𝚺\\mathbf\{A\}\_\{w\}\\leftarrow\\mathbf\{U\}\\boldsymbol\{\\Sigma\}
9:

𝐁w←𝐕⊤​𝐓p−1\\mathbf\{B\}\_\{w\}\\leftarrow\\mathbf\{V\}^\{\\top\}\\mathbf\{T\}\_\{p\}^\{\-1\}
10:

\(𝐀wq,𝐁wq\)←QuantizeINT8⁡\(𝐀w,𝐁w\)\(\\mathbf\{A\}\_\{w\}^\{q\},\\mathbf\{B\}\_\{w\}^\{q\}\)\\leftarrow\\operatorname\{QuantizeINT8\}\(\\mathbf\{A\}\_\{w\},\\mathbf\{B\}\_\{w\}\)
11:return

Pack⁡\(𝐖d,𝐀wq,𝐁wq,g\)\\operatorname\{Pack\}\(\\mathbf\{W\}\_\{d\},\\mathbf\{A\}\_\{w\}^\{q\},\\mathbf\{B\}\_\{w\}^\{q\},g\)

At inference, ARCHead evaluates the core path and correction path separately,

𝐘^=m​a​t​h​b​f​H​𝐖d⊤\+\(𝐇𝐁w⊤\)​𝐀w⊤\.\\widehat\{\\mathbf\{Y\}\}=mathbf\{H\}\\mathbf\{W\}\_\{d\}^\{\\top\}\+\(\\mathbf\{H\}\\mathbf\{B\}\_\{w\}^\{\\top\}\)\\mathbf\{A\}\_\{w\}^\{\\top\}\.\(14\)This factorized evaluation avoids registering a reconstructed dense head\. A backend may dequantize tiles or factors internally, but such temporary execution state is distinct from the persistent module representation measured in the storage tables\.

## Appendix CPacked Buffers and Model\-Conversion Lifecycle

Table[13](https://arxiv.org/html/2608.02703#A3.T13)lists the logical components of the serialized module\. Exact tensor layouts can vary with the packing implementation, but every reported byte count is obtained from the actual registered parameters and buffers rather than inferred solely from nominal bit widths\.

Table 13:Logical contents of the packedARCHeadstate dictionary\.#### Conversion lifecycle\.

The source checkpoint is first loaded with its dense LM\-head available to the constructor\. Calibration activations and the metric are computed, the quantized core and correction are fitted, and a packedARCHeadinstance is created\. The model’s output embedding is then replaced by this module\. Before serialization, the dense source tensor and temporary decompositions are released\. Reloading the converted checkpoint instantiates only the packed buffers listed above\.

#### Persistent storage versus working memory\.

Persistent storage is the sum ofnumel×\\timeselement\_sizeover registered tensors\. Construction working memory can be larger because it includes𝐖\\mathbf\{W\}, covariance factors, randomized\-SVD workspaces, and temporary dequantized values\. Forward working memory is different again and can include logits, activations, the KV cache, allocator state, and backend workspaces\. The 3\.7–3\.9×\\timesclaim concerns the first quantity only; separating these categories prevents a packed\-parameter result from being misread as an identical reduction in end\-to\-end peak GPU memory\.

#### Integration order\.

ARCHead is applied after the chosen transformer\-block quantizer\. This order preserves the block backend and targets only the remaining dense output projection\. In the inspected Qwen AWQ and bitsandbytes checkpoints, the head remained a BF16Linearof shape151,936×4,096151\{,\}936\\times 4\{,\}096, so replacement does not require changing the transformer\-block representation\.

## Appendix DEvaluation and Reproducibility Checklist

The following checklist clarifies how the reported comparisons isolate the output head\.

- •Head\-only experiments:transformer blocks remain BF16 and only the LM\-head representation changes\.
- •Hybrid experiments:AWQ or bitsandbytes quantizes the blocks first; BF16\-head and ARCHead rows share the same block backend\.
- •Calibration split:final hidden states are sampled from the WikiText\-103 training split\.
- •Evaluation split:quality is measured on 16,384 held\-out test tokens that are not reused for fitting\.
- •Matched seeds:the GPTQ\-style head and ARCHead use the same activation cache at each 2K, 4K, 8K, and 16K calibration budget with seeds 0, 1, and 2\.
- •Storage measurement:bytes are summed from instantiated packed tensors, including scales and metadata\-bearing buffers\.
- •Timing comparison:paired construction times use the same NVIDIA RTX Pro 6000; speedup is GPTQ time divided by ARCHead time\.
- •Throughput interpretation:changes within 2% are treated as negligible measurement variation, and small positive values are not claimed as speedups\.

#### Metric interpretation\.

Cross\-entropy and relative perplexity measure language\-model quality under the fixed evaluation activations\. Logit MSE and KL divergence measure fidelity to the dense head, while top\-kkagreement records whether dense predictions remain among the compressed head’s leading candidates\. Persistent head ratio measures serialized module size relative to BF16\. These metrics answer different questions and should not be collapsed into a single claim: storage does not determine quality, logit fidelity does not guarantee every downstream capability, and persistent bytes do not equal runtime peak memory\.

#### Downstream sanity check\.

HellaSwag, TruthfulQA MC2, and WinoGrande are evaluated with fixed transformer blocks and evaluation settings across head variants\. The suite is intentionally described as a sanity check because its small differences do not establish task\-level improvement or comprehensive capability preservation\. Its role is to identify an obvious regression that might be hidden by aggregate perplexity alone\.

## Appendix EFull Head\-Only Results

Table[14](https://arxiv.org/html/2608.02703#A5.T14)provides the comprehensive head\-only quantization results for Qwen3\-8B\-Base\. In these experiments, the transformer blocks are deliberately kept in BF16 to isolate the performance impact of the LM\-head\. As the results indicate, ARCHead achieves a storage footprint similar to naive Group INT4, but with significantly lower cross\-entropy and perplexity degradation\.

Table 14:Full head\-only quantization results on Qwen3\-8B\-Base\. Transformer blocks remain in BF16\.
## Appendix FCross\-Model Head\-Only Results

Table[15](https://arxiv.org/html/2608.02703#A6.T15)shows the performance of ARCHead across models with differing vocabularies and hidden dimensions\.

Qwen3\-8B exhibits the strongest relative improvement because its naive INT4 baseline fails drastically\. For Gemma\-4\-E4B, the naive INT4 baseline is already strong, meaning ARCHead’s advantage is narrower, though it remains highly competitive\. VibeThinker\-3B shows a larger overall loss due to its smaller hidden dimension, but ARCHead still provides a substantial improvement over the naive INT4 baseline\. This highlights the model\-dependent behavior of LM\-head quantization sensitivity\.

Table 15:Cross\-model head\-only results demonstrating model\-dependent sensitivity\.
## Appendix GHybrid Quantizer \+ ARCHead Results

ARCHead is intended as a drop\-in replacement for backends like AWQ and bitsandbytes\. Table[16](https://arxiv.org/html/2608.02703#A7.T16)presents the hybrid setup\. ARCHead does not improve the base perplexity of AWQ or BNB; rather, it compresses the dense BF16 head they leave behind\. The additional cross\-entropy cost introduced by ARCHead is extremely small \(\+0\.006\+0\.006to\+0\.007\+0\.007\), validating the drop\-in replacement claim\.

Table 16:Hybrid model results on Qwen3\-8B\-Base\. ARCHead compresses the remaining dense head with minimal extra CE\.
## Appendix HAblation Details

The extended ablation study in Table[17](https://arxiv.org/html/2608.02703#A8.T17)underscores the necessity of the activation\-metric residual correction branch\. The “ARCHead core only” variant exhibits a relative PPL of 1\.134\. Adding the residual correction branch reduces this sharply to 1\.007\. This confirms that the core alone is insufficient and supports the central novelty of optimizing in the covariance\-weighted logit space\.

Table 17:Expanded ablation details highlighting the importance of the correction branch\.
## Appendix IThroughput Measurement Details

Table[18](https://arxiv.org/html/2608.02703#A9.T18)outlines generation throughput measurements\. ARCHead does not bottleneck generation\. The small positive difference observed in the BNB NF4 configuration represents measurement noise rather than a real speedup\. Throughput is measured independently from persistent memory to provide an accurate picture of inference viability\.

Table 18:ARCHead maintains generation throughput\.
## Appendix JAdditional Result Visualizations

![Refer to caption](https://arxiv.org/html/2608.02703v1/x2.png)Figure 2:Qwen3\-8B\-Base head\-storage and perplexity trade\-off\. ARCHead occupies a similar storage range to group INT4 while remaining close to the dense head\.![Refer to caption](https://arxiv.org/html/2608.02703v1/x3.png)Figure 3:Measured generation throughput with a BF16 or packed ARCHead head\. Differences below 2% are treated as negligible measurement variation\.![Refer to caption](https://arxiv.org/html/2608.02703v1/x4.png)Figure 4:Qwen3\-8B\-Base calibration sensitivity\. Relative PPL remains in a narrow band from 4K through 64K calibration tokens\.
## Appendix KFailure Modes and Limitations

The scope and limitations of ARCHead include:

- •Density Requirement:ARCHead is most beneficial when the LM\-head constitutes a large dense parameter block\.
- •Model Sensitivity:If a model’s naive INT4 head quantization already performs exceptionally well \(as observed to some degree with Gemma\), ARCHead’s relative advantage diminishes\.
- •Prior Compression:If a backend natively and successfully quantizes the LM\-head, applying ARCHead may not yield additional quality improvements\.
- •Scope:ARCHead is explicitly developed for LM\-head compression\. It is not intended for compressing Feed\-Forward Networks \(FFN/MLP\) in this paper\.
- •Evaluation:WikiText perplexity and logit fidelity are the primary quality measures\. The three downstream tasks are a limited sanity check; exhaustive capability and safety evaluation remains future work\.
- •Kernel Optimization:The included fused\-kernel result is preliminary and limited to one accelerator\. Broader hardware and production\-backend optimization remains future work\.

## Appendix LPreliminary Future Work: Fused Triton Kernel Integration

We present preliminary results from a custom Triton kernel designed for the ARCHead compressed head\. The kernel performs INT8 dequantization and evaluates the low\-rank residual branch within a fused pass, loading the quantized core and row\-wise scales directly while computing the low\-rank residual\.

This prevents the prohibitive memory bandwidth overhead of sequentially instantiating the full FP16 matrix before multiplication\. Table[19](https://arxiv.org/html/2608.02703#A12.T19)presents the throughput results of integrating this fused kernel on a single NVIDIA RTX Pro 6000\.

These preliminary findings confirm that ARCHead does not bottleneck inference generation in practical deployment scenarios, even when paired with aggressively quantized transformer backends like AWQ or bitsandbytes NF4\.

Table 19:Preliminary throughput validation of the ARCHead fused Triton kernel on one NVIDIA RTX Pro 6000\. The measured differences are within 2%; small positive values are treated as noise rather than speedups\.