SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

# SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
Source: [https://arxiv.org/html/2608.28911](https://arxiv.org/html/2608.28911)
Daeha LeeEmail:[bigsum@etri\.re\.kr](mailto:[email protected])Corresponding author:Corresponding author\.Affiliation:Electronics and Telecommunications Research Institute \(ETRI\), Daejeon, Republic of KoreaDo\-Hyung KimAffiliation:Electronics and Telecommunications Research Institute \(ETRI\), Daejeon, Republic of KoreaJae\-Hong KimAffiliation:Electronics and Telecommunications Research Institute \(ETRI\), Daejeon, Republic of Korea

###### Abstract

The key–value \(KV\) cache is a primary memory bottleneck in long\-context large language model inference, since its size grows linearly with context length\. Existing KV\-cache compression spans uniform quantization, token deletion, and importance\-aware mixed precision, but it remains unclear how to choose precision levels without crossing a sharp quality boundary — and deletion additionally risks removing evidence that long\-context reasoning depends on\. We introduce SemKV, an all\-token\-preserving mixed\-precision KV\-cache quantization framework guided by an empirically measured*quality cliff*\.

Sweeping uniform quantization over a fractional\-level grid under a prespecified multi\-seed statistical protocol, we find that Llama\-3\.1\-8B\-Instruct with an affine scalar quantizer is statistically indistinguishable from FP16 KV down to 2\.322 code bits/value and degrades sharply at 2\.0 bits\. SemKV measures this boundary offline, then assigns two adjacent above\-cliff precisions \(2\.585/2\.322 bits\) to tokens by a model\-internal importance ranking, realizing an average of 2\.39 code bits/value \(2\.65 effective bits/value after packing and metadata\) — a measured6\.0×6\.0\\timesstorage reduction with no statistically detectable quality difference from FP16 KV on our sampled LongBench evaluation \(n=900n\{=\}900, three seeds\)\. Interior probes show that the tested partial\-protection mixtures do not recover full\-KV\-level quality when they cross the cliff, and a block\-wise deferred allocation extends the mechanism to generated tokens and multi\-turn dialogue\. Preserving all tokens at low precision also substantially outperforms FP16 token pruning even though the pruning baseline uses a1\.5×1\.5\\timeslarger memory footprint, and the cliff, the interpolation property, and the operating point transfer to Mistral\-7B\-Instruct\-v0\.3, where informative selection remains necessary for safe mixing\.

Finally, the measured cliff depends on the model–quantizer pair under a specified evaluation protocol: replacing the affine base with a distortion\-optimized quantizer \(TurboQuant\-MSE\) lowers the collapse boundary in every protocol we test, and SemKV converts the extra headroom directly into compression — interpolating at 2\.0/1\.585 bits inside the newly opened grid gap, it is statistically indistinguishable from full KV at an effective 2\.025 bits/value, raising the no\-detectable\-loss operating point from6\.0×6\.0\\timesto7\.9×7\.9\\times\. A small above\-cliff floor persists in full\-cache multi\-turn quantization under both bases \(its magnitude2\.5×2\.5\\timessmaller under TurboQuant\)\. In the tested multi\-turn comparisons, SemKV remains statistically indistinguishable from the upper grid point while using fewer bits\. These results motivate a general recipe: measure the cliff for the target deployment setting, then interpolate above it\.

###### Keywords:

KV cache compression , Long\-context inference , Large language models , Mixed precision quantization , Semantic importance

## 1Introduction

Large language models \(LLMs\) increasingly operate on contexts of tens of thousands of tokens for document QA, code understanding, retrieval\-augmented generation, and multi\-turn agents \[31–36, 44–46\]\. In Transformer decoding, the key–value \(KV\) cache grows linearly with context length, and its memory footprint governs the feasible batch size and context budget of practical serving systems \[37–38, 47\]\. Uniform KV quantization \[1–2, 9\], eviction and pruning \[13–24, 55\], and efficient attention \[25–29\] have all been proposed, but uniform quantization is blind to token\-level differences, and deletion\-based methods risk removing evidence tokens\.

This paper starts from an empirical observation rather than an architectural one\. When uniform KV quantization is swept over a*fractional\-bit*grid \(e\.g\.,log2⁡5≈2\.322\\log\_\{2\}5\\approx 2\.322,log2⁡6≈2\.585\\log\_\{2\}6\\approx 2\.585bits realized by integer quantization levels\), quality does not degrade gradually: on our LongBench protocol with Llama\-3\.1\-8B\-Instruct, all uniform settings from 3\.0 down to 2\.322 bits are statistically indistinguishable from full\-precision KV, and quality collapses only in the interval\(2\.0,2\.322\]\(2\.0,2\.322\]— a*quality cliff*that co\-occurs with a sharp increase in runaway generation\. Under the same affine base, the same collapse location reappears in single\-turn QA, in generation\-token quantization, and in multi\-turn dialogue \(where a small above\-cliff residual floor additionally exists\), and even on a different backbone \(Mistral\-7B\-Instruct\-v0\.3\)\.

The cliff reframes what mixed precision is for\. In the tested setting, mixtures crossing the cliff fail to recover full\-KV performance, whereas allocation has little measurable effect when both precision levels remain above it\. The useful regime is the sparse fractional\-bit grid just above the cliff, where uniform quantization can only sit*on*grid points, while mixed precision can*interpolate between*them\. SemKV therefore preserves every token, scores each token with a model\-internal semantic importance indicator, and assigns the higher of two above\-cliff grid precisions to the top\-ranked fraction, reaching average precisions \(e\.g\., 2\.39 bits between 2\.322 and 2\.585\) that uniform quantization cannot realize, at quality statistically indistinguishable from full KV\.

The main contributions of this paper are as follows\.

- 1\.Quality\-cliff map on a fractional\-bit grid\.We chart uniform KV quantization on a fractional\-bit grid under a prespecified multi\-seed statistical protocol and locate a quality cliff in\(2\.0,2\.322\]\(2\.0,2\.322\]bits — whose catastrophic transition is reproduced across prefill, generation, and multi\-turn protocols under this base \(multi\-turn additionally shows a small above\-cliff floor; Sec\. 4\.6\) — for Llama\-3\.1\-8B, co\-occurring with runaway generation; on the primary single\-turn LongBench protocol the region\[2\.322,3\.0\]\[2\.322,3\.0\]is statistically indistinguishable from full KV\.
- 2\.Grid interpolation by all\-token\-preserving mixed precision\.SemKV keeps all tokens and interpolates between above\-cliff grid points via importance\-ranked two\-level allocation, reaching otherwise unreachable average precisions with no statistically detectable deficit from full KV — a measured6\.0×6\.0\\timesKV\-storage reduction versus FP16 at the 2\.39\-bit operating point, metadata included\. Ablations over eight model\-internal indicators \(hidden\-state, KV\-norm, logit, and attention families\) show the effect is structural in the flat regime: indicator choice does not change the outcome there\.
- 3\.Generation\-token and multi\-turn extension\.A block\-exact deferred\-quantization mechanism extends token\-wise mixed precision to generation\-time tokens; under the affine base, generation quality degrades sharply at 2\.0 bits whereas 2\.322\-bit generation remains statistically indistinguishable from FP16, and the extension holds across multi\-turn dialogue\.
- 4\.Deletion versus low\-precision preservation\.Under matched \(in fact adverse\) memory budgets, FP16 pruning collapses \(0\.065–0\.149\) while SemKV at 2\.39 bits shows no statistically detectable difference from full KV \(0\.438\); random pruning is statistically indistinguishable from indicator\-based pruning and exhibits the same qualitative collapse, implicating token deletion as the dominant source of failure in this controlled comparison\.
- 5\.Stress and transfer\.A widened\-gap stress test whose low\-bit side lies below the cliff produces a clear ranking among the tested model\-internal indicators, showing that indicator quality becomes important when most tokens receive a below\-cliff precision, and the cliff and interpolation property transfer to Mistral\-7B\-Instruct\-v0\.3, where SemKV remains statistically indistinguishable from full KV when informative indicators are used; replacing the base quantizer with TurboQuant\-MSE shows the cliff is quantizer\-dependent, and that the downward shift is protocol\-wide: the collapse boundary moves one grid step down to\(1\.585,2\.0\]\(1\.585,2\.0\]in prefill and multi\-turn, and to\(1\.0,1\.585\]\(1\.0,1\.585\]on the generation side, while SemKV’s token\-axis allocation composes unchanged, matches TurboQuant’s channel\-axis 2\.5\-bit recipe at a smaller effective footprint, and occupies the affine\-inaccessible 1\.9\-code\-bit gap at performance statistically indistinguishable from full KV \(7\.9×7\.9\\times\)\.

## 2Related Work

### 2\.1KV Cache Quantization

In long\-context LLM inference, the KV cache grows in proportion to the input length and becomes a major GPU memory bottleneck when long contexts and large batch sizes are used\. To address this issue, recent KV\-cache quantization methods have been actively developed\. KVQuant \[1\] targets sub\-4\-bit KV cache quantization and combines per\-channel quantization for the key cache, pre\-RoPE key quantization, non\-uniform datatypes, and outlier\-aware dense\-and\-sparse quantization to achieve low performance degradation even at the 3\-bit level\. KIVI \[2\] analyzes the distributional characteristics of the KV cache and proposes an asymmetric 2\-bit KV cache quantization method that quantizes the key cache per channel and the value cache per token\. TurboQuant \[9\] takes a quantizer\-design view: it applies a random rotation so that coordinates approach a known distribution, quantizes each coordinate with a distribution\-matched optimal \(Lloyd–Max\) scalar quantizer, and optionally corrects inner\-product bias with a 1\-bit residual sketch, achieving near\-optimal distortion without calibration data; its fractional operating points \(e\.g\., 2\.5 bits\) are realized by*channel\-wise*mixed precision that quantizes outlier channels at a higher bit\-width\. TurboQuant is therefore orthogonal to SemKV along two axes: it improves the per\-vector quantizer while allocating precision across*channels*, whereas SemKV allocates precision across*tokens*on a fractional\-level grid; the two compose, and Sec\. 4\.9 evaluates SemKV with TurboQuant as its base quantizer\.

General LLM quantization studies are also related to SemKV\. SmoothQuant \[3\], GPTQ \[4\], AWQ \[5\], LLM\.int8\(\) \[6\], ZeroQuant \[7\], QServe \[8\], QuaRot \[10\], Atom \[11\], block reconstruction\-based post\-training quantization \[56\], and post\-training quantization \[57\-59\] aim to improve LLM inference efficiency through weight or activation quantization\. However, these studies mainly focus on tensor\-, channel\-, or weight\-level quantization\. In contrast, SemKV differs in that it assigns KV cache precision differently based on token\-level semantic importance\.

A line of work closer to SemKV applies*mixed\-precision*KV quantization\. MiKV \[71\] retains would\-be\-evicted KV pairs at low precision and important pairs at high precision, with importance imported from external eviction policies such as H2O; QAQ \[72\] adapts bit allocation to token importance and sensitivity; SKVQ \[73\] combines clipped dynamic quantization with a high\-precision recency window; ZipCache \[74\] identifies salient tokens via normalized attention scores; GEAR \[75\] compresses most entries to ultra\-low precision with low\-rank residual correction; WKVQuant \[76\] and RotateKV \[77\] preserve recent or sink tokens at high precision; IntactKV \[78\] keeps outlier\-token caches intact\. Recent analysis also studies the token\-count versus precision trade\-off directly \[79\]\. SemKV differs from these methods in three respects\. First, SemKV treats token importance as a pluggable model\-internal signal and systematically evaluates eight indicators across four families \(hidden\-state, KV, logit, and attention\), rather than committing to a single eviction policy or fixed positional heuristic\. Second, SemKV explicitly maps the uniform\-quantization quality cliff on a fractional\-bit grid and positions mixed precision as*grid interpolation strictly above the cliff*, which explains*when*allocation matters \(near or below the cliff\) and when it does not \(the flat regime\)\. Third, the same ranking\-based mechanism is extended block\-exactly to generation\-time tokens and multi\-turn inference under a prespecified multi\-seed statistical protocol\.

### 2\.2KV Cache Compression and Eviction

Another direction for reducing KV cache memory is eviction or compression, which keeps only a subset of KV cache tokens and removes the rest\. Based on the observation that heavy\-hitter tokens are repeatedly and importantly referenced in attention, H2O \[13\] proposes a KV cache eviction policy that retains both recent tokens and heavy\-hitter tokens\. SnapKV \[14\] observes attention patterns within an observation window before generation and selects important KV positions on which each attention head focuses\. PyramidKV \[15\] exploits the phenomenon of pyramidal information funneling, in which layer\-wise information flow is broadly distributed in lower layers and concentrated on core tokens in upper layers\. KVzip \[81\] scores KV pairs by their necessity for reconstructing the original context, enabling query\-agnostic eviction whose compressed cache remains reusable across subsequent queries\. FastGen \[17\], ChunkKV \[18\], InfLLM \[19\], RetrievalAttention \[20\], context compression \[66\], and semantic\-aware cache compression \[67\] also address KV cache or attention compression for long\-context inference\.

Although these studies are effective in reducing the size of the KV cache, they commonly rely on selecting or removing a subset of KV tokens\. In contrast, SemKV does not remove tokens\. SemKV preserves all tokens and adjusts only their precision according to importance\.

### 2\.3Token Pruning and Sparse Inference

Token pruning and sparse inference studies aim to reduce computation and memory by decreasing the number of input tokens or intermediate tokens\. LazyLLM \[21\], DynamicViT \[22\], Token Merging \[23\], TokenLearner \[24\], and Adaptive Token Pruning \[55\] perform dynamic token selection or merging based on token importance\. Sparse Transformer \[25\], Longformer \[26\], and Big Bird \[27\] make the attention pattern itself sparse to enable processing of long sequences\.

However, pruning or sparse selection can remove tokens themselves or restrict their accessibility\. Since SemKV preserves all tokens and adjusts only their precision, it is an all\-token\-preserving compression method that is fundamentally different from pruning\.

### 2\.4Efficient Attention and Long\-Context Inference

FlashAttention \[28\] and FlashAttention\-2 \[29\] greatly improve the efficiency of attention computation through IO\-aware exact attention\. RoFormer \[30\] improves the positional representation of Transformers through rotary position embeddings\. StreamingLLM \[16\] uses the attention sink phenomenon to enable stable inference even for long streaming sequences\. ALiBi \[65\] addresses the difficulty of extrapolating Transformers to contexts longer than those seen during training\.

These studies improve attention computation or positional generalization, whereas SemKV is a complementary approach in that it compresses the KV cache memory itself using semantic mixed precision during the decoding stage\.

### 2\.5Long\-Context Evaluation

Benchmarks such as LongBench \[40\], LongBench v2 \[42\], and InfiniteBench \[43\] have been proposed to evaluate long\-context LLMs\. Lost in the Middle \[41\] showed that LLMs do not always use long input contexts uniformly and that performance may degrade when answer information is located in the middle of the context\. These studies show that evidence position, retrieval difficulty, and distractor composition are important in long\-context evaluation\. Based on this line of work, this paper evaluates SemKV on QA/retrieval tasks, a recency stress test, pruning comparison, and a mixed LongBench boundary setting\.

### 2\.6LLM Serving, Memory Optimization, and Mixed Precision

To address memory bottlenecks in LLM inference and training, FlexGen \[12\], vDNN \[49\], ZeRO \[50\], ZeRO\-Infinity \[51\], Megatron\-LM \[52\], studies on large\-scale training systems \[53\-54\], GShard \[61\], Switch Transformer \[62\], PaLM \[63\], and Chinchilla \[64\] have been proposed\. In addition, mixed\-precision training \[48\] and adaptive mixed\-precision inference \[68\] demonstrate the importance of precision allocation\. SemKV extends this mixed\-precision perspective to the token dimension of the KV cache\.

### 2\.7Position of SemKV

Taken together, existing methods for reducing KV cache memory can be broadly divided into three categories\. First, uniform KV quantization quantizes all tokens with the same bit\-width \[1\-2, 9\]\. Second, pruning\- or eviction\-based methods remove tokens considered less important \[13\-14, 21, 55\]\. Third, some methods retain a subset of tokens based on specific positions or attention patterns, such as attention sinks, recent tokens, and heavy\-hitter tokens \[16, 41\]\. SemKV occupies a different position from these approaches\. SemKV does not remove tokens; it preserves all tokens\. At the same time, it does not quantize all tokens uniformly, but assigns higher precision to tokens with higher semantic importance\. Table 1 summarizes this positioning\.

Table 1:Positioning against representative KV compression families\. Examples: uniform quantization — KIVI, KVQuant; eviction — H2O, SnapKV; importance\-aware quantization — MiKV, QAQ, ZipCache\. SemKV is distinguished by the combination of all\-token preservation, token\-wise fractional\-grid precision mapped from a measured cliff, and generation\-time support\. Entries summarize common characteristics of representative methods and do not imply that every method in a family has all listed properties\.

## 3SemKV Method

This section describes the proposed SemKV method\. The central objective of SemKV is to reduce KV cache memory usage in long\-context LLM inference while preserving as much semantic information as possible for long\-context question answering\.

The overall architecture of SemKV is shown in Fig\. 1\. The input prompt passes through the Transformer prefill stage to generate a full\-precision KV cache\. SemKV then computes token\-level semantic importance and separates high\-precision and low\-precision tokens according to the importance ranking\. Finally, it constructs a mixed\-precision KV cache using the token\-wise bit allocation results and uses it during decoding\. Through this structure, SemKV reduces KV cache memory without token pruning while retaining all token positions and prioritizing tokens with high model\-internal importance scores\.

### 3\.1Overall Architecture

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig1.png)Fig\. 1:Overall SemKV architecture\. SemKV first generates the full\-precision KV cache, estimates token\-level semantic importance, ranks tokens, allocates higher precision to important tokens, and performs mixed\-precision KV quantization before decoding\. The two precision levels shown inside the panel \(bhighb\_\{\\mathrm\{high\}\}/blowb\_\{\\mathrm\{low\}\}\) are set in our experiments to fractional\-level codes such as 2\.585/2\.322 bits \(Sec\. 4\.1\)\.
### 3\.2Problem Formulation

Let the input sequence length beTT, the number of Transformer layers beLL, the number of key–value heads beHKVH\_\{\\mathrm\{KV\}\}\(for grouped\-query\-attention models such as Llama\-3\.1 this is smaller than the attention\-head count\), the dimension of each head beDD, and the KV cache precision bebbbits\. Then, for a single sequence and ignoring quantization metadata, the KV cache storage in bits can be approximated as follows\.

MKV=2⋅L⋅HKV⋅T⋅D⋅b\.M\_\{\\mathrm\{KV\}\}=2\\cdot L\\cdot H\_\{\\mathrm\{KV\}\}\\cdot T\\cdot D\\cdot b\.\(1\)
SemKV keeps the sequence lengthTTunchanged while assigning a different token\-wise bit\-widthbib\_\{i\}\.

TSemKV=T,bi∈\{blow,bhigh\}\.T\_\{\\mathrm\{SemKV\}\}=T,\\quad b\_\{i\}\\in\\\{b\_\{\\mathrm\{low\}\},b\_\{\\mathrm\{high\}\}\\\}\.\(2\)
As illustrated in Fig\. 2, SemKV estimates semantic importance from model\-internal signals — by default, hidden\-state dynamics from the final Transformer layers\. Rather than relying on token position or recency heuristics, SemKV analyzes how token representations evolve across late layers\. Tokens with larger representation changes are hypothesized to contribute more strongly to contextual processing, and we use this quantity as an importance proxy in long\-context QA\.

### 3\.3Semantic Importance Estimation

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig2.png)Fig\. 2:Semantic importance scoring pipeline\. SemKV computes a token\-importance score from one selected model\-internal indicator and ranks tokens within each prompt; because allocation depends only on the ranking, score normalization is not required \(the 0–1 score scale in the panel is illustrative\)\.For an input tokenii, let the hidden representation at layerllbehi\(l\)h\_\{i\}^\{\(l\)\}\. When the lastmmlayers are used, the delta\-based semantic importance score of SemKV can be defined as follows\.

si=1m​∑l=L−m\+1L‖hi\(l\)−hi\(l−1\)‖2\.s\_\{i\}=\\frac\{1\}\{m\}\\sum\_\{l=L\-m\+1\}^\{L\}\\left\\\|h\_\{i\}^\{\(l\)\}\-h\_\{i\}^\{\(l\-1\)\}\\right\\\|\_\{2\}\.\(3\)
Equation \(3\) defines the default*delta\-norm*indicator\. SemKV treats the indicator as a pluggable module and, throughout the ablations, draws on four families of model\-internal signals, all obtained from tensors the model already computes\. The*hidden\-state*family reads the last\-mm\-layer representations: besides delta\-norm, it includes the final\-layer magnitude \(*norm*,‖hi\(L\)‖2\\\|h\_\{i\}^\{\(L\)\}\\\|\_\{2\}\), the coordinate\-wise*variance*ofhi\(L\)h\_\{i\}^\{\(L\)\}, and the angular counterpart of Eq\. \(3\) \(*cosine\-delta*,1−cos⁡\(hi\(l\),hi\(l−1\)\)1\-\\cos\(h\_\{i\}^\{\(l\)\},h\_\{i\}^\{\(l\-1\)\}\)averaged over the lastmmlayers\), which captures direction change even when magnitudes are stable\. The*KV\-norm*family scores each token by the magnitude of the very tensors being compressed \(*key\-norm*‖Ki‖2\\\|K\_\{i\}\\\|\_\{2\}and*value\-norm*‖Vi‖2\\\|V\_\{i\}\\\|\_\{2\}\), a proxy for how strongly a token can contribute to attention logits and outputs\. The*logit*family uses*self\-information*,−log⁡pθ​\(xi∣x<i\)\-\\log p\_\{\\theta\}\(x\_\{i\}\\mid x\_\{<i\}\): tokens that the model finds surprising tend to introduce new information\. Finally, the*attention*family uses accumulated attention received by tokenii\(∑j≥iAj,i\\sum\_\{j\\geq i\}A\_\{j,i\}\), the signal closest to eviction heuristics such as H2O; unlike the other families it requires materializing attention weights, which drives the cost analysis of Sec\. 4\.5\. A*random*selector serves as the uninformative baseline in ablations\. Scores enter the method only through their within\-prompt ranking, so no normalization is required, and the allocation procedure of Sec\. 3\.4 remains identical across indicators\. We restrict the average to the*final*mmlayers under the heuristic that, as depth increases, Transformer hidden states integrate progressively broader contextual information about each token than earlier layers; we treat Eq\. \(3\) as a practical proxy for a token’s semantic contribution to the context, the quantity the allocation should protect\.

### 3\.4Token\-Wise Mixed\-Precision Allocation

Operating rule: map the cliff first\.The bit\-precision pair\(bh​i​g​h,bl​o​w\)\(b\_\{high\},b\_\{low\}\)consumed by the allocation of Sec\. 3\.4 is not a free hyperparameter: it is determined by an offline*cliff\-mapping*procedure performed once for each fixed deployment setting \(model, base quantizer, evaluation protocol, and context\-budget composition\), illustrated in Fig\. 3\. The procedure sweeps uniform quantization over the integer\-level grid, compares each grid point against the full\-precision cache with paired statistics on a representative evaluation set, examines adjacent grid\-point differences, and locates the*quality cliff*— a sharp, seed\-consistent increase in degradation between two adjacent grid points, below which output quality collapses catastrophically rather than degrading gracefully \(Sec\. 4\.2\)\.\(bh​i​g​h,bl​o​w\)\(b\_\{high\},b\_\{low\}\)are then chosen as grid points above the measured cliff; because no uniform grid point exists between adjacent levels, the token\-mixed allocation of Fig\. 4 provides a practical mechanism for realizing average bit\-widths inside that gap\. The mapping is input\-independent — it characterizes the target deployment setting, not individual prompts — and need not be repeated for individual prompts, but it should be revalidated when the model, base quantizer, evaluation protocol, or context\-budget composition changes materially; Sec\. 4\.9 shows that replacing the quantizer moves the boundary downward in all three protocols studied in this paper — prefill compression \(Sec\. 4\.2\), generation\-time quantization \(Sec\. 4\.4\), and multi\-turn dialogue \(Sec\. 4\.6\) — with the same operating rule carrying over; under the affine base the bracketed interval coincides across protocols, while under TurboQuant the generation\-side boundary sits one further grid step lower\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig3.png)Fig\. 3:The quality\-cliff concept and the offline mapping procedure\. Uniform quantization on the level grid remains in a relatively flat regime until degradation increases sharply between two adjacent grid points; small residual deficits may exist above this collapse bracket, particularly in full\-cache multi\-turn evaluation \(Sec\. 4\.6\)\. In the single\-turn setting illustrated here, the above\-cliff grid points are statistically indistinguishable from full precision\. The boundary is an empirically measured property of the model, base quantizer, and target evaluation setting; improving the quantizer shifts it downward in every protocol we test \(the generation\-side boundary may sit lower than the prefill and multi\-turn boundaries\)\. The curve is schematic; only the grid points are empirically evaluated\. SemKV sets\(bh​i​g​h,bl​o​w\)\(b\_\{high\},b\_\{low\}\)to grid points above the measured cliff and interpolates between them\.Fig\. 4 illustrates the adaptive mixed\-precision allocation process in SemKV\. After computing semantic importance scores, SemKV ranks tokens and allocates higher precision to the most important subset of tokens\. The remaining tokens are quantized using lower precision\. This strategy allows SemKV to significantly reduce average KV precision while retaining all token positions and prioritizing tokens with high importance scores\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig4.png)Fig\. 4:Adaptive bit allocation based on token importance\. The top\-ranked tokens are assigned high precision, while the remaining tokens are assigned low precision\. The dashed box indicates that the bit\-precision pair\(bh​i​g​h,bl​o​w\)\(b\_\{high\},b\_\{low\}\)is supplied by the offline cliff mapping of Fig\. 3 and revalidated when the target model, quantizer, protocol, or context\-budget composition changes materially\. The allocation is realized purely by the top\-rrranking; no score threshold is estimated\. Token subscripts in the two groups denote membership inℋ\\mathcal\{H\}andℐ∖ℋ\\mathcal\{I\}\\setminus\\mathcal\{H\}respectively; the low\-precision group is generally not contiguous in token position\.Once the semantic importance scoresis\_\{i\}is computed, SemKV selects the toprrfraction of all tokens as high\-precision tokens\.

K=⌊r​T⌋,ℋ=TopK⁡\(\{si\}i=1T,K\)\.K=\\lfloor rT\\rfloor,\\quad\\mathcal\{H\}=\\mathrm\{TopK\}\(\\\{s\_\{i\}\\\}\_\{i=1\}^\{T\},K\)\.\(4\)
The bit\-widthbib\_\{i\}assigned to tokeniiis then defined as follows\.

bi=\{bhigh,if​i∈ℋ,blow,otherwise\.b\_\{i\}=\\begin\{cases\}b\_\{\\mathrm\{high\}\},&\\text\{if \}i\\in\\mathcal\{H\},\\\\ b\_\{\\mathrm\{low\}\},&\\text\{otherwise\}\.\\end\{cases\}\(5\)
The average bit\-width is given as follows\.

b¯=1T​∑i=1Tbi\.\\bar\{b\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}b\_\{i\}\.\(6\)

### 3\.5KV Cache Quantization

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig5.png)Fig\. 5:Mixed\-precision KV quantization process\. The token\-level bit allocation is applied to key and value tensors, resulting in a mixed\-precision quantized KV cache used for decoding\. The token\-wise precision assignment is broadcast across all layers and KV heads \(HKVH\_\{\\mathrm\{KV\}\}\); bits are never allocated along the head dimension itself\. Thebhighb\_\{\\mathrm\{high\}\}/blowb\_\{\\mathrm\{low\}\}labels inside the panel denote the two precision levels; experiments use fractional\-level codes such as 2\.585/2\.322 bits \(Sec\. 4\.1\)\.Let the key and value of tokeniiat layerllbeKi\(l\)K\_\{i\}^\{\(l\)\}andVi\(l\)V\_\{i\}^\{\(l\)\}, respectively\. Then the quantized KV cache of SemKV can be expressed as follows\.

K^i\(l\)=Qbi​\(Ki\(l\)\),V^i\(l\)=Qbi​\(Vi\(l\)\)\.\\hat\{K\}\_\{i\}^\{\(l\)\}=Q\_\{b\_\{i\}\}\(K\_\{i\}^\{\(l\)\}\),\\quad\\hat\{V\}\_\{i\}^\{\(l\)\}=Q\_\{b\_\{i\}\}\(V\_\{i\}^\{\(l\)\}\)\.\(7\)

### 3\.6All\-Token\-Preserving Property

Pruning\-based methods retain only a proper subset𝒫\\mathcal\{P\}of the input\-token setℐ\\mathcal\{I\}\. In contrast, SemKV preserves the complete token set and changes only the precision assigned to each token\.

𝒫⊂ℐ,\|𝒫\|<\|ℐ\|\.\\mathcal\{P\}\\subset\\mathcal\{I\},\\quad\|\\mathcal\{P\}\|<\|\\mathcal\{I\}\|\.\(8\)
SemKV instead satisfies the following condition\.

ℐSemKV=ℐ\.\\mathcal\{I\}\_\{\\mathrm\{SemKV\}\}=\\mathcal\{I\}\.\(9\)
That is, SemKV keeps every token in the input prompt in the KV cache\. The only difference lies in the precision assigned to each token\.

### 3\.7Generation\-Token Mixed\-Precision Quantization

The mechanism above quantizes the prompt KV cache once after prefill, when the full score distribution is available and an exact top\-rrselection is possible\. Generation\-time tokens do not enjoy this property: their scores arrive one by one, and committing a bit\-width immediately upon generation requires approximating an unknown future score distribution, which we found brittle in practice\. SemKV instead applies a*block\-exact deferred*scheme, illustrated in Fig\. 6\. We call the procedure*block\-exact*because the prescribed high\-precision count is met exactly within each completed block; it does not claim equivalence to a global top\-rrranking over all generated tokens\. Newly generated tokens are appended to the cache at full precision while only their importance scores are collected; whenever a block ofBBtokens \(e\.g\.,B=64B\{=\}64\) has accumulated, exactlyround⁡\(r​B\)\\operatorname\{round\}\(rB\)tokens with the highest scores are selected within the block — whereround⁡\(⋅\)\\operatorname\{round\}\(\\cdot\)denotes nearest\-integer rounding — and only those positions are quantized tobhighb\_\{\\mathrm\{high\}\}, the remainder toblowb\_\{\\mathrm\{low\}\}\. At the end of each generation turn, the residual partial block is quantized using the same block\-wise top\-rrrule\. This guarantees the prescribed high\-precision fraction constructively \(per\-block rounding error at most half a token\), removes all threshold estimation, and keeps the most recent fewer thanBBtokens temporarily uncompressed during decoding, while also retaining the most recent local context at full precision\. In multi\-turn dialogue the same procedure is applied per turn on the accumulated conversation\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig6.png)Fig\. 6:Generation\-token mixed\-precision quantization\. Scores of newly generated tokens are accumulated and quantized block\-exactly in deferred fashion, so the prescribed high\-precision fraction is met constructively without threshold estimation, and the most recent partial block remains in FP16 during decoding; at the end of each generation turn, the residual block is quantized using the same block\-wise top\-rrrule\.
### 3\.8SemKV Algorithm

Algorithm 1SemKV Semantic Mixed\-Precision KV Cache Compression \(prefill\)0:Prompt

X=\{x1,…,xT\}X=\\\{x\_\{1\},\\ldots,x\_\{T\}\\\}, LLM

fθf\_\{\\theta\}, high\-precision fraction

rr, low bit

blowb\_\{\\mathrm\{low\}\}, high bit

bhighb\_\{\\mathrm\{high\}\}, scoring layer count

mm
0:Mixed\-precision quantized KV cache

𝒞^KV\\hat\{\\mathcal\{C\}\}\_\{\\mathrm\{KV\}\}
1:Perform prefill forward pass for input prompt

XX\.

2:Obtain KV cache

𝒞KV\(l\)=\{K\(l\),V\(l\)\}\\mathcal\{C\}\_\{\\mathrm\{KV\}\}^\{\(l\)\}=\\\{K^\{\(l\)\},V^\{\(l\)\}\\\}for each layer

ll\.

3:Compute semantic importance score

sis\_\{i\}using the last

mmlayers\.

4:Set

K=⌊r​T⌋K=\\lfloor rT\\rfloorand select

ℋ=TopK⁡\(\{si\}i=1T,K\)\\mathcal\{H\}=\\mathrm\{TopK\}\(\\\{s\_\{i\}\\\}\_\{i=1\}^\{T\},K\)\.

5:foreach token

iido

6:if

i∈ℋi\\in\\mathcal\{H\}then

7:

bi←bhighb\_\{i\}\\leftarrow b\_\{\\mathrm\{high\}\}
8:else

9:

bi←blowb\_\{i\}\\leftarrow b\_\{\\mathrm\{low\}\}
10:endif

11:endfor

12:Quantize

Ki\(l\)K\_\{i\}^\{\(l\)\}and

Vi\(l\)V\_\{i\}^\{\(l\)\}using

Qbi​\(⋅\)Q\_\{b\_\{i\}\}\(\\cdot\)\.

13:Use

𝒞^KV\\hat\{\\mathcal\{C\}\}\_\{\\mathrm\{KV\}\}for decoding; generated tokens are handled by Algorithm 2\.

Algorithm 2Block\-Exact Quantization of Generated Tokens \(Sec\. 3\.7\)0:Quantized prompt cache

𝒞^KV\\hat\{\\mathcal\{C\}\}\_\{\\mathrm\{KV\}\}from Algorithm 1, high\-precision fraction

rr, bits

bhigh,blowb\_\{\\mathrm\{high\}\},b\_\{\\mathrm\{low\}\}, block size

BB
1:Initialize score buffer

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset\(scores stored with their token indices\)\.

2:whilegeneratingdo

3:Generate token

tgt\_\{g\}; append

Kg\(l\),Vg\(l\)K\_\{g\}^\{\(l\)\},V\_\{g\}^\{\(l\)\}to

𝒞^KV\\hat\{\\mathcal\{C\}\}\_\{\\mathrm\{KV\}\}at full precision\.

4:Compute score

sgs\_\{g\}with the same indicator;

𝒮←𝒮∪\{\(g,sg\)\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\\{\(g,s\_\{g\}\)\\\}\. \{no per\-token decision\}

5:if

\|𝒮\|=B\|\\mathcal\{S\}\|=Bthen

6:

ℋblk←TopKIndices⁡\(\{sg:\(g,sg\)∈𝒮\},round⁡\(r​B\)\)\\mathcal\{H\}\_\{\\mathrm\{blk\}\}\\leftarrow\\mathrm\{TopKIndices\}\(\\\{s\_\{g\}:\(g,s\_\{g\}\)\\in\\mathcal\{S\}\\\},\\ \\operatorname\{round\}\(rB\)\)\{exact count within the block\}

7:For each token

ggin the block:

bg←bhighb\_\{g\}\\leftarrow b\_\{\\mathrm\{high\}\}if

g∈ℋblkg\\in\\mathcal\{H\}\_\{\\mathrm\{blk\}\}else

blowb\_\{\\mathrm\{low\}\}\.

8:Quantize the block’s

Kg\(l\),Vg\(l\)K\_\{g\}^\{\(l\)\},V\_\{g\}^\{\(l\)\}in place with

Qbg​\(⋅\)Q\_\{b\_\{g\}\}\(\\cdot\);

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset\.

9:endif

10:endwhile

11:During decoding, the most recent partial block \(

<B<Btokens\) stays at full precision\.

12:At the end of each generation turn, if

\|𝒮\|\>0\|\\mathcal\{S\}\|\>0, flush the residual partial block with the same rule:

ℋblk←TopKIndices⁡\(\{sg:\(g,sg\)∈𝒮\},round⁡\(r​\|𝒮\|\)\)\\mathcal\{H\}\_\{\\mathrm\{blk\}\}\\leftarrow\\mathrm\{TopKIndices\}\(\\\{s\_\{g\}:\(g,s\_\{g\}\)\\in\\mathcal\{S\}\\\},\\ \\operatorname\{round\}\(r\|\\mathcal\{S\}\|\)\), then quantize in place\.

## 4Experiments and Results

This section evaluates SemKV along the narrative established above: \(i\) the quality cliff of uniform KV quantization on a fractional\-bit grid, \(ii\) grid interpolation by all\-token\-preserving mixed precision and its indicator ablation, \(iii\) generation\-token quantization, \(iv\) scoring cost, \(v\) multi\-turn behavior, \(vi\) deletion versus low\-precision preservation, \(vii\) a below\-cliff stress test, and \(viii\) cross\-model and context\-length transfer\.

### 4\.1Experimental Setup and Statistical Protocol

Models and benchmarks\.The main backbone is Llama\-3\.1\-8B\-Instruct; transfer experiments use Mistral\-7B\-Instruct\-v0\.3\. Single\-turn evaluation follows our LongBench protocol on ten subsets \(qasper, multifieldqa\_en, hotpotqa, 2wikimqa, gov\_report, multi\_news, triviaqa, samsum, passage\_retrieval\_en, lcc\) with 30 samples per subset and a 7,500\-token context budget\. The composite score is the item\-average of each task’s official metric \(QA\-F1, ROUGE, accuracy, or code similarity, as specified by LongBench\), not QA\-F1 alone\. Multi\-turn evaluation uses MT\-Eval \[80\] with ROUGE\-L per turn\.

Quantization\.All quantized settings use integer\-level scalar quantization with a Hadamard rotation; fractional bit\-widths denote non\-power\-of\-two level counts \(e\.g\.,2\.322=log2⁡52\.322=\\log\_\{2\}5,2\.585=log2⁡62\.585=\\log\_\{2\}6bits\)\. SemKV uses two\-level allocation between adjacent above\-cliff grid points,bhigh=2\.585b\_\{\\mathrm\{high\}\}\{=\}2\.585andblow=2\.322b\_\{\\mathrm\{low\}\}\{=\}2\.322, with a high\-precision fraction ofr=25%r\{=\}25\\%for single\-turn prompts \(average 2\.39 bits\) andr=35%r\{=\}35\\%including generation tokens in multi\-turn settings \(average 2\.41 bits\)\.

Memory accounting\.Reported bit\-widths are code bits; storage adds two components\. First, non\-power\-of\-two level counts are stored by mixed\-radix packing: three five\-level codes pack into 7 bits \(2\.3332\.333bits/value,\+0\.49%\+0\.49\\%overlog2⁡5\\log\_\{2\}5\) and five six\-level codes into 13 bits \(2\.6002\.600bits/value,\+0\.58%\+0\.58\\%overlog2⁡6\\log\_\{2\}6\)\. Second, the per\-row asymmetric affine quantizer stores two FP16 parameters \(offset and scale\) per head–token row of dimension 128, adding0\.250\.25bits/value; the per\-token precision flag contributes 1 bit per token \(<10−4<10^\{\-4\}bits/value\) and is negligible\. The effective storage cost of the 2\.39\-bit operating point is therefore2\.652\.65bits/value, i\.e\., a6\.04×6\.04\\timesreduction versus FP16\. Quality experiments use fake quantization \(quantize–dequantize with FP16 compute\), so these storage figures are validated separately by a packed\-buffer microbenchmark that allocates the exact packed sizes on GPU \(Sec\. 4\.5\); measured allocations match this accounting to within allocator page rounding \(<0\.9%<0\.9\\%\), and the compression ratio is independent of context length\. The reported packed\-storage ratios exclude the transient FP16 tail of fewer thanBBgenerated tokens \(Sec\. 3\.7\); this overhead is bounded byB−1B\{\-\}1tokens and becomes negligible for long contexts\.

Statistical conventions\.Throughout the experiments,Δ\\Deltadenotes the paired score difference relative to the full\-precision \(FP16\) baseline;Δ=0\\Delta\{=\}0indicates equal measured scores\. Parenthetical values arepp\-values from the stated paired test, and we usep<0\.05p\{<\}0\.05as evidence of a statistically detectable difference\.*n\.s\.*means that the test did not detect a significant difference; it does not establish equality\. Where shown, error bars represent 95% confidence intervals of the paired difference\. Bonferroni correction is applied to prespecified families of multiple comparisons and is noted where used\.

Statistical protocol \(prespecified\)\.Every headline condition is run with three seeds \(42/43/44\); exploratory sweeps use seed 42\. The primary test is a pairedtt\-test over items keyed by \(seed, item, task\); the eight indicator\-vs\-full\-KV tests and the 28 pairwise indicator tests are corrected as separate Bonferroni families\. Single\-seed exploratory findings were not treated as confirmed effects unless their direction was consistent across all three seeds\. Table 2 summarizes the configuration\.

Three conventions matter for interpreting what follows\. First, atn=900n\{=\}900the paired 95% CI of the LongBench composite is approximately±0\.011\\pm 0\.011, so absolute deficits substantially smaller than about0\.0110\.011cannot be resolved under this protocol and are reported as n\.s\. rather than as absence of loss;*safe*accordingly means that no statistically detectable deficit is observed under the stated protocol, at this resolution \(Sec\. 4\.9 gives a concrete instance in which a more sensitive protocol resolves a deficit that is n\.s\. here\)\. Second, the*measured cliff bracket*is the interval between two adjacent tested grid points across which degradation increases sharply, with the lower grid point showing a large statistically detectable deficit and the direction consistent across seeds; a grid point above the cliff may still exhibit a smaller residual deficit — in particular in full\-cache multi\-turn evaluation \(Sec\. 4\.6\) — so lying above the collapse cliff does not by itself imply the absence of any deficit, and the cliff claim rests on the contrast between a flat region of n\.s\. differences and an adjacent collapse significant by several orders of magnitude\. Third, because each benchmark item is evaluated under three seeds, then=900n\{=\}900paired samples share item\-level structure; we therefore verified the headline single\-turn and indicator\-ablation conclusions using item\-level seed averages \(n=300n\{=\}300paired items\), and the significance pattern is unchanged \(SemKV at 2\.39 bits n\.s\.,p=0\.85p\{=\}0\.85; uniform 2\.585/2\.322 bits n\.s\.,p=0\.07p\{=\}0\.07/0\.210\.21; the 2\.0\-bit collapse significant,p<10−6p\{<\}10^\{\-6\}; all eight indicators n\.s\. vs\. full KV and all 28 pairwise comparisons n\.s\. after correction\)\.

Table 2:SemKV experimental configuration\.
### 4\.2The Quality Cliff of Uniform KV Quantization

Sweeping uniform quantization over the fractional\-bit grid reveals that quality does not decay smoothly \(Fig\. 7a\)\. On Llama\-3\.1\-8B, every uniform setting in\[2\.322,3\.0\]\[2\.322,3\.0\]bits — including the fractional points2\.807=log2⁡72\.807=\\log\_\{2\}7,2\.585=log2⁡62\.585=\\log\_\{2\}6, and2\.322=log2⁡52\.322=\\log\_\{2\}5— is statistically indistinguishable from full KV under the paired test \(\|Δ\|≤0\.0094\|\\Delta\|\\leq 0\.0094, allp≥0\.09p\\geq 0\.09,n=900n\{=\}900\), whereas uniform 2\.0 bits collapses \(Δ=−0\.0448\\Delta\{=\}\-0\.0448,p<10−7p\{<\}10^\{\-7\}\), with the direction consistent in all three seeds\. The cliff is therefore located in\(2\.0,2\.322\]\(2\.0,2\.322\], and under this base the*same collapse interval*reappears in generation\-token quantization \(Sec\. 4\.4\) and in multi\-turn dialogue \(Sec\. 4\.6, where a small residual above\-cliff floor additionally exists\), and transfers to Mistral \(Sec\. 4\.9\)\. Cliff*depth*, unlike location, is seed\-dependent; all cliff claims are pooled three\-seed results\.

The affine cliff interior is not safely navigable under the tested mixtures\.Integer\-level scalar quantization admits*no uniform grid point*strictly between 2\.0 bits \(4 levels\) and 2\.322 bits \(5 levels\); the only way to probe the interior is mixing\. Fig\. 7b reports three\-seed interior probes that mix the two boundary precisions \(2\.322/2\.0\) at protection ratiosr∈\{25,50,75\}%r\\in\\\{25,50,75\\\}\\%, i\.e\., average 2\.081/2\.161/2\.242 bits\. Recovery is monotone inrrbut strikingly slow: protecting 25% at 2\.322 bits leaves quality at the u2\.0 collapse level \(Δ=−0\.0446\\Delta\{=\}\-0\.0446vs\.−0\.0448\-0\.0448\), and even 75% protection remains significantly degraded \(Δ=−0\.0266\\Delta\{=\}\-0\.0266,p<10−4p\{<\}10^\{\-4\}\)\. Moreover, inside the cliff, importance\-concentrated protection*underperforms*spread\-out protection: delta\-norm selection atr=50%r\{=\}50\\%is significantly worse than random selection at the same budget \(Δ=−0\.0189\\Delta\{=\}\-0\.0189,p<0\.01p\{<\}0\.01, three\-seed consistent\)\. Below the cliff, degradation is broad\-based rather than confined to a small evidence set, so concentrating the protection budget on high\-scoring tokens systematically exposes the rest\. Both observations sharpen the operating rule that motivates SemKV: mixed precision should*interpolate strictly between above\-cliff grid points*, not average across the cliff\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig7.png)Fig\. 7:The quality cliff \(Llama\-3\.1\-8B, LongBench protocol, 3 seeds; error bars: 95% CI of the paired difference\)\. \(a\) All uniform settings in\[2\.322,3\.0\]\[2\.322,3\.0\]bits are statistically indistinguishable from full KV under this protocol; 2\.0 bits collapses\. \(b\) Interior probes mixing 2\.322/2\.0 bits: recovery is monotone but even 75% protection remains significantly degraded, and importance\-concentrated selection underperforms spread\-out selection inside the cliff\.
### 4\.3Grid Interpolation Above the Cliff and Indicator Indifference

The cliff delimits where mixed precision is useful: uniform quantization can only occupy grid points, while ranking\-based mixed precision interpolates between them\. Fig\. 8 and Table 3 show the interpolation result at 2\.39 average bits \(25% at 2\.585, 75% at 2\.322\) on our LongBench protocol\. Across*eight*model\-internal indicators spanning four signal families — hidden\-state \(delta norm, norm, variance, cosine delta\), KV\-norm \(key norm, value norm\), logit \(self\-information\), and attention \(accumulated attention\) — every SemKV variant is statistically indistinguishable from full KV, and all pairwise indicator comparisons are non\-significant after Bonferroni correction\. In the flat regime the benefit is thus*structural*: preserving all tokens at above\-cliff precisions suffices, and at this Llama operating point indicator choice among the tested model\-internal signals does not measurably affect quality\. This motivates choosing the indicator by*cost*\(Sec\. 4\.5\) rather than by quality\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig8.png)Fig\. 8:Grid interpolation at 2\.39 average bits on our LongBench protocol \(Llama\-3\.1\-8B, 10 subsets×\\times30 items×\\times3 seeds\)\. Paired difference from full KV with 95% confidence intervals for all eight indicator variants of SemKV at the 2\.39\-bit operating point \(n=900n\{=\}900, 3 seeds\)\. Every interval contains zero, and all 28 pairwise indicator comparisons are n\.s\. \(Bonferroni\-corrected\) — interpolation quality at this operating point is indicator\-independent\.Table 3:Grid interpolation at 2\.39 bits: paired difference vs\. full KV \(Llama\-3\.1\-8B,n=900n\{=\}900\)\. No variant differs significantly from full KV; all pairwise indicator differences are n\.s\. \(Bonferroni\-corrected\)\. Compression is measured KV storage vs\. FP16, metadata included \(Sec\. 4\.1\)\.
### 4\.4Generation\-Token Quantization: Affine\-Base Collapse and a Favorable Operating Point

Applying the block\-exact scheme of Sec\. 3\.7 to generation tokens reproduces the cliff on the generation side \(Fig\. 9\)\. Quantizing generated tokens at 2\.322 bits is a favorable operating point: quality is statistically indistinguishable from FP16 generation \(Δ=−0\.0041\\Delta\{=\}\-0\.0041,p=0\.14p\{=\}0\.14,n=900n\{=\}900\), and the max\-length termination rate is significantly*reduced*relative to FP16 \(McNemarp<10−4p\{<\}10^\{\-4\}\)\. In contrast, any configuration whose low\-bit side is 2\.0 — uniform 2\.0\-bit generation or mixed generation withblow=2\.0b\_\{\\mathrm\{low\}\}\{=\}2\.0, even when paired with 2\.585\- or 3\.0\-bit high precision — degrades significantly \(Δ\\Deltabetween−0\.0127\-0\.0127and−0\.0157\-0\.0157, allp<10−3p\{<\}10^\{\-3\}, three\-seed consistent\) and elevates the max\-length termination rate \(McNemarp=0\.015p\{=\}0\.015for uniform 2\.0\)\. The asymmetry is the generation\-side signature of the cliff: a below\-cliff component cannot be compensated by mixing in higher precision, mirroring the interior result of Sec\. 4\.2\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig9.png)Fig\. 9:Generation\-token quantization \(Llama\-3\.1\-8B, 3 seeds\)\. \(a\) 2\.322\-bit generation is statistically indistinguishable from FP16 generation; every configuration containing a 2\.0\-bit generation component degrades significantly, regardless of the paired high precision\. \(b\) The max\-length termination rate is reduced at 2\.322 bits and elevated only at 2\.0 bits\. Max\-length termination rate: the fraction of outputs that reach the configured generation limit without emitting EOS\. We use this rate as an operational proxy for runaway\-generation behavior\.
### 4\.5Cost: Scoring Overhead and Memory Footprint

At the primary Llama single\-turn operating point the eight tested indicators are statistically indistinguishable, so computational cost becomes the practical selection criterion \(Fig\. 10\)\. Hidden\-state scoring isO⁡\(1\)O\(1\)per decoding step and essentially flat in context length \(\+3\.1%\+3\.1\\%from 1k to 32k tokens in our microbenchmark\), whereas attention\-score recomputation isO⁡\(T\)O\(T\)per step, growing to3\.2×3\.2\\timesthe 1k cost by 32k and8\.3×8\.3\\timesthe hidden\-state cost at that length\. Attention scoring additionally requires materializing attention weights \(eager attention\), itself a∼\\sim5% decoding\-throughput penalty in our measurement\. On the LongBench workload the end\-to\-end scoring share of generation time was 0\.39% for hidden\-state scoring versus 1\.11% for attention recomputation \(2\.9×2\.9\\times\)\. Across the tested model\-internal indicators, a per\-prompt profiling of the prefill scoring pass shows a further spread: value\-norm, which reads the cached value tensors directly, costs 0\.2 ms and 22 MiB of peak activation memory per prompt, versus 1\.1–1\.2 ms / 263 MiB for hidden\-norm and variance, 3\.0 ms / 350 MiB for delta\-norm, and 3\.3 ms / 438 MiB for cosine\-delta — value\-norm is roughly15×15\\timescheaper in time and16×16\\timeslower in peak activation memory than delta\-norm because it avoids materializing multi\-layer hidden states\. Since the eight indicators are quality\-indistinguishable at this operating point, value\-norm is the lowest\-cost indicator among those profiled in this comparison \(the four hidden\-state indicators and value\-norm\) when scoring memory is at a premium\. SemKV nonetheless keeps delta\-norm as the reference configuration throughout this paper: it is a conservative, relatively high\-cost model\-internal indicator \(although cosine\-delta incurs slightly higher scoring time and peak activation memory\), so the reported reference overhead represents a conservative high\-cost configuration, though not the absolute maximum among all tested indicators\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig10.png)Fig\. 10:Scoring cost \(Llama\-3\.1\-8B\)\. \(a\) Per\-step scoring cost versus context length: hidden\-state scoring is flat \(O⁡\(1\)O\(1\)\), attention recomputation grows withTT\. \(b\) End\-to\-end scoring share of generation time on LongBench\. Complexity labels are with respect to context lengthTT: hidden\-state scoring isO⁡\(1\)O\(1\)per decode step inTT, while attention recomputation grows withTT\.Measured memory footprint\.Fig\. 11 reports the packed\-storage microbenchmark of Sec\. 4\.1: KV\-shaped buffers at the exact packed sizes \(code bits plus metadata\) are allocated on GPU and measured directly, with the FP16 baseline allocated as per\-layer K/V tensors\. Both backbones share the KV shape \(32 layers, 8 KV heads, head dimension 128\), so one measurement covers both\. Across context lengths from 4k to 128k tokens, SemKV at the 2\.39\-bit operating point occupies6\.0×6\.0\\timesless storage than full FP16 KV in every measurement \(6\.026\.02–6\.04×6\.04\\times; e\.g\., 16\.0 GiB→\\rightarrow2\.65 GiB at 128k\), the multi\-turn 2\.41\-bit point is6\.0×6\.0\\timesas well, and the measured ratio is constant in context length, as the accounting predicts\. The FP16 keep\-25% pruning baseline of Sec\. 4\.7 measures4\.0×4\.0\\times, so SemKV outperforms it while using a1\.5×1\.5\\timessmaller footprint\. Measured allocations agree with the analytical accounting to within CUDA allocator page rounding \(<0\.9%<0\.9\\%, largest at the shortest context\)\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig11.png)Fig\. 11:Measured KV storage \(packed\-buffer microbenchmark; both backbones share this KV shape\)\. \(a\) KV storage versus context length: SemKV at 2\.39 bits tracks6\.0×6\.0\\timesbelow full FP16 KV at every length\. \(b\) Measured compression at 128k tokens; the pruning baseline that SemKV outperforms in Sec\. 4\.7 occupies a1\.5×1\.5\\timeslarger footprint\.
### 4\.6Multi\-Turn Dialogue

On MT\-Eval, catastrophic degradation again appears at a single grid step: only uniform 2\.0\-bit collapses \(Δ=−0\.082\\Delta\{=\}\-0\.082,p<10−6p\{<\}10^\{\-6\}\), and collapse depth tracks how strongly a task depends on prior content \(expansion−0\.181\-0\.181\>\>follow\-up−0\.109\-0\.109\>\>refinement−0\.067\-0\.067\>\>recollection−0\.024\-0\.024, n\.s\.\)\. Above the cliff the multi\-turn picture is not perfectly flat: with adequate statistical power, full\-cache multi\-turn quantization leaves a small but significant residual deficit under this affine base — uniform 2\.322\-bit at−0\.021\-0\.021in both the three\-seed follow\-up run \(p<10−3p\{<\}10^\{\-3\},n=90n\{=\}90\) and the original five\-task affine run \(p<10−3p\{<\}10^\{\-3\},n=80n\{=\}80\), with uniform 2\.585/2\.807 similar \(−0\.012\-0\.012/−0\.015\-0\.015\), i\.e\. the deficit is insensitive to bits well above the cliff\. SemKV at 2\.41 average bits, with generation tokens quantized block\-exactly, sits at−0\.008\-0\.008to−0\.009\-0\.009versus full KV \(borderline,p=0\.03p\{=\}0\.03–0\.060\.06depending on the run\) — it significantly improves on uniform 2\.322\-bit \(\+0\.0137\+0\.0137,p=0\.006p\{=\}0\.006\) and is statistically indistinguishable from uniform 2\.585\-bit quality while using0\.170\.17fewer code bits, but it does not beat this small protocol floor; Sec\. 4\.9 shows the same structure, with a2\.5×2\.5\\timessmaller floor, under the TurboQuant base\. The deficit concentrates in the task most dependent on prior generated content \(expansion\), mirroring the collapse ordering above, and does not track the raw number of generated tokens — consistent with errors compounding through the dialogue text itself \(each turn conditions on previously generated, slightly perturbed answers\) rather than with per\-step cache reuse\. Fig\. 12 reports the multi\-turn indicator ablation at the operating point\. The hidden\-state indicators remain close to full KV and are mutually indistinguishable; after Bonferroni correction across indicators none of them differs significantly from full KV, although delta\-norm is nominally significant before correction \(Δ=−0\.0075\\Delta\{=\}\-0\.0075, rawp=0\.029p\{=\}0\.029\)\. Attention scoring shows the largest and most consistently detectable deficit \(Δ=−0\.0115\\Delta\{=\}\-0\.0115,p<0\.001p\{<\}0\.001, significant after correction; direct attention\-vs\-hidden comparisons are n\.s\.\)\. Together with its higher computational cost \(Sec\. 4\.5\), attention scoring offers no observed advantage for generation\-time allocation in this setting\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig12.png)Fig\. 12:Multi\-turn indicator ablation \(MT\-Eval follow\-up,n=90n\{=\}90paired conversations, 3 seeds\): paired difference vs\. full KV with 95% CIs\. Hidden\-state indicators are mutually indistinguishable, and after correction none differs significantly from full KV\. These hidden\-state indicators lie near the small multi\-turn floor of this protocol \(Secs\. 4\.6, 4\.9\)\. Attention scoring exhibits a larger deficit that remains statistically detectable after correction\.
### 4\.7Deletion versus Low\-Precision Preservation

Fig\. 13 and Table 4 use hidden\-norm for the primary*matched\-indicator*comparison between pruning and SemKV — isolating the structural variable \(delete vs\. preserve\-at\-low\-precision\) — and include additional pruning indicators as robustness controls for the selection rule\. This is a controlled structural comparison, not a reproduction of engineered eviction systems with recency windows \(e\.g\., H2O, SnapKV\)\. FP16 pruning that keeps 25% of tokens is a 4\-bit\-equivalent memory budget —*more*memory than SemKV at 2\.39 bits — yet collapses to 0\.063–0\.149 depending on the indicator, while SemKV scores 0\.432–0\.442\. Two observations sharpen the conclusion\. First,*random*pruning is statistically indistinguishable from indicator\-based pruning \(paired difference n\.s\.\) and shows the same qualitative collapse, suggesting that deletion itself, rather than selection quality, is the dominant source of failure in this controlled comparison\. Second, collapse depth tracks evidence sparsity across tasks \(synthetic retrieval and few\-shot collapse hardest; summarization and code least\), consistent with the evidence\-token\-loss mechanism\. In the earlier full sweep, collapse was also independent of the keep ratio from 15% to 50%\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig13.png)Fig\. 13:Deletion versus low\-precision preservation under the same indicator \(Llama\-3\.1\-8B, 3 seeds\)\. \(a\) SemKV at 2\.39 bits shows no statistically detectable deficit from full KV while FP16 keep\-25% pruning \(a strictly larger, 4\-bit\-equivalent memory budget\) collapses; random pruning is statistically indistinguishable from indicator\-based pruning\. \(b\) Per\-task collapse depth tracks evidence sparsity\. The pruning condition is a controlled structural baseline using the same indicator and budget accounting, not an implementation of an engineered eviction system such as H2O or SnapKV\.Table 4:Structural comparison between SemKV and FP16 pruning \(LongBench composite,n=900n\{=\}900\)\. The primary matched\-indicator comparison uses hidden\-norm; additional pruning indicators test robustness to the selection rule\. Compression is measured KV storage vs\. FP16; SemKV occupies a1\.5×1\.5\\timessmaller footprint than the pruning baselines it outperforms\. For SemKV rows the bit figure is the nominal average code rate on the level grid; for FP16 keep\-kkpruning it is the memory\-equivalent rate16​k16k— both are converted to measured storage for the compression column\.
### 4\.8Below\-Cliff Stress: When Indicator Quality Matters

The primary SemKV operating points and indicator ablations above use precision pairs on the high\-precision side of the cliff, where the tested indicators are statistically indistinguishable\. To probe when selection matters, we stress the system on the task with the deepest multi\-turn collapse \(MT\-Eval expansion\) and*widen the precision gap*to 3\.0/2\.0 bits atr=35%r\{=\}35\\%\(average 2\.30 bits\): now 65% of tokens sit*below*the cliff, so performance therefore depends strongly on which 35% of tokens receive high precision\. Fig\. 14 shows the outcome\. At the standard operating point \(2\.585/2\.322, average 2\.40 bits\), SemKV scores 0\.394, significantly above uniform 2\.322\-bit \(0\.332,p<0\.05p\{<\}0\.05\) and uniform 2\.0\-bit \(0\.254,p<0\.01p\{<\}0\.01\), while suppressing runaway generation \(2/10 conversations vs\. 4/10 and 7/10\)\. In the widened\-gap setting, the model\-internal indicators separate into a clear performance spectrum \(Fig\. 15\): value\-norm, attention, cosine\-delta, self\-information, and delta\-norm allocate at or above the random\-selection baseline, whereas hidden\-norm and key\-norm fall below it — i\.e\., in this regime a bad indicator is worse than no indicator\. Given the small stress sample \(10 conversations\), we report pooled three\-seed statistics with per\-seed direction consistency and treat individual indicator\-vs\-random gaps as directional; the value\-norm\-vs\-hidden\-norm spread itself is significant \(p<0\.01p\{<\}0\.01\)\. Although the stress sample is small, the pooled and per\-seed trends consistently support the qualitative conclusion:*in the tested Llama setting, structure dominates at the primary above\-cliff point, whereas selection becomes decisive under below\-cliff stress*\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig14.png)Fig\. 14:Below\-cliff stress \(MT\-Eval expansion, 10 conversations×\\times3 seeds\)\. \(a\) At the operating point SemKV significantly outperforms both uniform anchors in this stress setting\. \(b\) Runaway generation is suppressed by SemKV relative to the uniform anchors\.![Refer to caption](https://arxiv.org/html/2608.28911v1/fig15.png)Fig\. 15:Widened\-gap indicator spectrum under stress \(3\.0/2\.0 bits, average 2\.30 bits, MT\-Eval expansion\)\. In this Llama stress setting, indicator quality separates when the low\-bit side lies below the cliff: informative selection reduces catastrophic degradation when most tokens receive a below\-cliff precision; indicators below the random line are worse than random\.
### 4\.9Cross\-Model, Context\-Length, and Base\-Quantizer Transfer

Fig\. 16 and Table 5 evaluate transfer to Mistral\-7B\-Instruct\-v0\.3 under the identical protocol \(10 subsets×\\times30×\\times3 seeds\)\. Three properties transfer\. \(i\)*Cliff location*: only uniform 2\.0\-bit degrades significantly \(Δ=−0\.0249\\Delta\{=\}\-0\.0249,p<10−3p\{<\}10^\{\-3\}\); 2\.322–3\.0 bits are indistinguishable from full KV\. \(ii\)*Grid interpolation*: SemKV at 2\.39 bits shows no statistically detectable deficit from full KV for every informative indicator \(\|Δ\|≤0\.012\|\\Delta\|\\leq 0\.012, all n\.s\. after correction\)\. \(iii\)*Indicator indifference among informative indicators*: all pairwise comparisons between the eight model\-internal indicators are n\.s\. \(Bonferroni\-corrected\)\. Notably, on Mistral the mixing safety is*not*indicator\-free in the strong sense observed on Llama: uninformative \(random\) token selection at the same 2\.39\-bit budget degrades significantly \(Δ=−0\.0213\\Delta\{=\}\-0\.0213,p<10−3p\{<\}10^\{\-3\}, three\-seed consistent\), concentrated on retrieval\-style tasks, while every informative indicator remains safe\. Semantic selection is thus what makes aggressive mixing transferable across backbones\. A three\-seed context\-budget sweep \(2k/4k/7\.5k tokens; Fig\. 17\) adds a further datapoint: at the 2k and 7\.5k budgets only uniform 2\.0\-bit degrades, but at the mid budget \(4k\) uniform 2\.322\-bit also degrades significantly \(Δ=−0\.0243\\Delta\{=\}\-0\.0243,p<10−4p\{<\}10^\{\-4\}\), and SemKV buffers this to roughly half the deficit \(Δ=−0\.0139\\Delta\{=\}\-0\.0139; SemKV vs\. uniform 2\.322\-bit:\+0\.0104\+0\.0104,p=0\.027p\{=\}0\.027\)\. SemKV is the setting closest to full KV at every budget\. Absolute scores at smaller budgets include truncation effects and only within\-budget differences are interpreted\.

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig16.png)Fig\. 16:Transfer to Mistral\-7B\-Instruct\-v0\.3 \(10 subsets×\\times30×\\times3 seeds\)\. \(a\) The cliff transfers; SemKV at 2\.39 bits remains statistically indistinguishable from full KV, whereas random selection at the same budget does not\. \(b\) Per\-indicator deltas vs\. full KV\. On Mistral, informative indicators remain mutually interchangeable, but random allocation is not safe — indicator freedom is an operating\-point\-specific, not universal, result\.Table 5:Cross\-model transfer \(pairedΔ\\Deltavs\. full KV,n=900n\{=\}900per model\)\. Absolute scores are not comparable across models\. Compression is measured KV storage vs\. FP16; the two backbones share the same KV shape, so the ratios apply to both\.![Refer to caption](https://arxiv.org/html/2608.28911v1/fig17.png)Fig\. 17:Context\-budget sweep on Mistral \(3 seeds\)\. Within each budget, SemKV stays closest to full KV among the quantized settings; absolute levels include truncation effects\. The measured bracket can depend on context\-budget composition even when the primary 7\.5k protocol is stable\.Base\-quantizer transfer\.The cliff of Sec\. 4\.2 was mapped with a per\-row asymmetric affine quantizer under Hadamard rotation\. To test whether the cliff is a property of the model alone or of the*model–quantizer pair*, we replace the base quantizer with a reimplementation of TurboQuant\-MSE \[9\] — random rotation followed by a Gaussian\-optimal Lloyd–Max codebook, extended here to non\-power\-of\-two level counts so the fractional grid is preserved — and repeat the uniform sweep and the SemKV operating point on the single\-turn protocol \(10 subsets×\\times30×\\times3 seeds, shared full\-KV baseline\)\. Affine anchors run in the same sweep reproduce Sec\. 4\.2 exactly \(uniform 2\.322\-bit n\.s\.; uniform 2\.0\-bitΔ=−0\.0448\\Delta\{=\}\-0\.0448,p<10−7p\{<\}10^\{\-7\}\), validating the port\. Table 6 and Fig\. 18 show the outcome: with the distortion\-optimized base,*the entire tested range\[2\.0,3\.0\]\[2\.0,3\.0\]bits is flat*\. Uniform 2\.0\-bit under TurboQuant is statistically indistinguishable from full KV \(Δ=−0\.0069\\Delta\{=\}\-0\.0069,p=0\.24p\{=\}0\.24\), a direct paired recovery of\+0\.0379\+0\.0379over affine 2\.0\-bit \(p<10−5p\{<\}10^\{\-5\}\), and the runaway signature of the cliff disappears with it: the max\-length generation rate at 2\.0 bits drops from 28\.8% under affine quantization to 19\.9% under TurboQuant \(affine vs\. TurboQuant: McNemarp<10−11p\{<\}10^\{\-11\}\), statistically indistinguishable from the full\-KV baseline of this run \(21\.8%, McNemarp=0\.09p\{=\}0\.09\)\. Residual depth at 2\.0 bits is seed\-dependent \(one seed shows−0\.027\-0\.027; the other two are null\), mirroring the depth seed\-dependence noted in Sec\. 4\.2; a marginal uncorrected deficit at 3\.0 bits \(p=0\.02p\{=\}0\.02\) does not survive Bonferroni correction and is non\-monotone in bits, so we treat it as noise\. Two further observations complete the picture\. First,*orthogonality*: SemKV’s token\-axis interpolation on the TurboQuant base is statistically indistinguishable from full KV \(Δ=−0\.0065\\Delta\{=\}\-0\.0065, n\.s\.\), is indistinguishable from uniform TurboQuant 2\.322\-bit, and closely matches the affine uniform 2\.322\-bit anchor \(Δ=\+0\.0004\\Delta\{=\}\+0\.0004\) — the allocation mechanism composes with a better quantizer unchanged\. Second, a same\-protocol comparison with TurboQuant’s own*channel\-axis*mixed\-precision recipe \(outlier channels at 3 bits, the rest at 2 bits; 2\.5 code bits\): quality is statistically indistinguishable from token\-axis SemKV \(Δ=−0\.0042\\Delta\{=\}\-0\.0042,p=0\.48p\{=\}0\.48\), while SemKV’s effective storage is smaller \(2\.525 vs\. 2\.750 bits/value, as TurboQuant\-MSE stores one scalar per row against two for the channel\-split configuration\)\. The practical reading is not that either mixing axis dominates, but that the cliff location must be measured for the target deployment setting: with a stronger base quantizer the flat region extends downward — here to at least 2\.0 bits, i\.e\., a measured7\.5×7\.5\\timescompression with no statistically detectable deficit on this protocol — and SemKV’s operating rule, interpolate above the measured cliff, carries over unchanged\. Two follow\-up sweeps complete the picture\.Locating the TurboQuant cliff\.Extending the uniform sweep below 2\.0 bits \(3\-level and 2\-level Lloyd–Max grids, same protocol and seeds\) locates the cliff rather than merely bounding it: uniform 1\.585\-bit collapses \(Δ=−0\.0379\\Delta\{=\}\-0\.0379,p<10−5p\{<\}10^\{\-5\}, all three seeds consistent\) and 1\.0\-bit is catastrophic \(−0\.267\-0\.267\), so the TurboQuant cliff sits at\(1\.585,2\.0\]\(1\.585,2\.0\]—*exactly one grid step below*the affine cliff of Sec\. 4\.2\. The collapse decouples from the runaway signature: in the matched generation\-side runs, max\-length rates stay at the full\-KV level of the sub\-2\-bit run \(21\.8%\) at both 1\.585 bits \(22\.0%, McNemarp=0\.92p\{=\}0\.92\) and 1\.0 bit \(21\.4%, McNemarp=0\.87p\{=\}0\.87\)\. The interior is shallower than the affine interior reported in Sec\. 4\.2: protecting 75% of tokens at 2\.0 bits \(code 1\.9 bits, effective 2\.025,7\.9×7\.9\\times\) returns to statistical indistinguishability from full KV \(Δ=−0\.0091\\Delta\{=\}\-0\.0091,p=0\.17p\{=\}0\.17\), and mixing significantly outperforms the lower uniform grid point \(\+0\.018\+0\.018atr=50%r\{=\}50\\%,p=0\.005p\{=\}0\.005;\+0\.029\+0\.029atr=75%r\{=\}75\\%,p<10−3p\{<\}10^\{\-3\}\), while delta\-norm and random selection remain statistically indistinguishable inside the cliff \(p=0\.072p\{=\}0\.072\) — in contrast to the affine interior of Sec\. 4\.2, where delta\-norm selection atr=50%r\{=\}50\\%was significantly*worse*than random\.

Generation\-time and multi\-turn transfer\.Repeating the generation\-quantization protocol of Sec\. 4\.4 and the multi\-turn protocol of Sec\. 4\.6 with the TurboQuant base \(3 seeds each; affine anchors reproduce Secs\. 4\.4 and 4\.6 in the same runs: generation−0\.0157\-0\.0157,p<10−5p\{<\}10^\{\-5\}; multi\-turn−0\.070\-0\.070,p<10−48p\{<\}10^\{\-48\}\) shows that the boundary shift is protocol\-wide \(Table 7, Fig\. 19\)\. On the generation side the cliff is fully bracketed: TurboQuant is safe at 2\.322, 2\.0, and 1\.585 bits \(all n\.s\. vs\. FP16\-generation; largest deficit−0\.0031\-0\.0031\), and collapses at 1\.0 bit \(Δ=−0\.0329\\Delta\{=\}\-0\.0329,p<10−18p\{<\}10^\{\-18\}, three seeds consistent\) — the generation\-side cliff sits at\(1\.0,1\.585\]\(1\.0,1\.585\], at least one grid step below its affine location, and the collapse again arrives without the runaway signature \(max\-length rates match FP16 generation, McNemarp=0\.87p\{=\}0\.87\)\. In multi\-turn dialogue the collapse point moves exactly one grid step, mirroring prefill: uniform 1\.585\-bit collapses by−0\.0912\-0\.0912\(p<10−79p\{<\}10^\{\-79\}, deeper than the affine 2\.0\-bit collapse in the same run, pairedΔ=−0\.0212\\Delta\{=\}\-0\.0212\), while 2\.0\-bit removes68%68\\%of the affine collapse \(−0\.0222\-0\.0222vs\.−0\.070\-0\.070\)\.

Above the cliff, multi\-turn retains the small residual floor already documented for the affine base in Sec\. 4\.6, and a dedicated same\-run comparison shows it is*not*specific to either quantizer: affine uniform 2\.322\-bit sits at−0\.0209\-0\.0209\(p<10−10p\{<\}10^\{\-10\}\) and TurboQuant at−0\.0083\-0\.0083\(p<10−3p\{<\}10^\{\-3\}\) in the same run — the better base shrinks the floor by2\.5×2\.5\\times\(paired\+0\.0126\+0\.0126,p<10−6p\{<\}10^\{\-6\}\) but does not remove it, and raising TurboQuant to 2\.585 bits leaves the floor unchanged \(−0\.0077\-0\.0077; vs\. 2\.322\-bitp=0\.77p\{=\}0\.77\), so it is not fixable by spending more bits in this range\. Selection behaves identically on both bases: the SemKV operating point is statistically indistinguishable from the upper uniform grid point \(p=0\.68p\{=\}0\.68under TurboQuant\) — selection reaches the ceiling the protocol admits — and above\-cliff interpolation composes in multi\-turn as well: mixing 2\.322/2\.0 atr=35%r\{=\}35\\%\(token\-weighted effective 2\.236 bits/value,7\.16×7\.16\\times\) significantly improves on uniform 2\.0\-bit \(\+0\.0110\+0\.0110,p<10−3p\{<\}10^\{\-3\}\) and is statistically indistinguishable from uniform 2\.322\-bit quality while using0\.100\.10fewer code bits, exactly as the affine mix behaves relative to uniform 2\.585\-bit in Sec\. 4\.6\. We do not treat the multi\-turn mix as a no\-detectable\-loss operating point, because its residual deficit relative to full KV \(−0\.0111\-0\.0111\) remains significant under this protocol\.

This decomposition also reconciles our sweep with the original TurboQuant report, which finds quality neutrality at 3\.5 bits and*marginal degradation already at 2\.5 bits*: that claim concerns quantization of the entire cache including decode\-side tokens, whereas the flat single\-turn sweep above quantizes the prefill cache only\. In our own full\-cache protocol \(multi\-turn\), both bases show exactly such small deficits above the cliff — there is no contradiction, only a decomposition of where the loss lives\. Across all three manifestations the conclusion is uniform:*replacing the base quantizer moves the collapse boundary down by at least one grid step and leaves the operating rule — map the cliff, interpolate above it — unchanged\.*Transfer to other backbones under the TurboQuant base is left untested \(Sec\. 5\)\.

Table 6:Base\-quantizer transfer \(Llama\-3\.1\-8B, single\-turn,n=900n\{=\}900\)\. Effective bits include packing and per\-row metadata \(Sec\. 4\.1: affine and channel\-split store two scalars per row, TurboQuant\-MSE one\)\. Affine anchors reproduce Sec\. 4\.2 in the same run\. The sub\-2\-bit rows \(separate 3\-seed run, shared protocol\) locate the TurboQuant cliff at\(1\.585,2\.0\]\(1\.585,2\.0\]; ther=75%r\{=\}75\\%mix sits inside the affine\-inaccessible gap between the 4\-level and 3\-level grids\.†Does not survive Bonferroni correction \(×4⇒p=0\.08\\times 4\\Rightarrow p\{=\}0\.08\) and is non\-monotone in bits\.

Table 7:Generation\-time and multi\-turn transfer of the TurboQuant base \(3 seeds; generation: LongBench vs\. FP16\-generation baseline,n=900n\{=\}900; multi\-turn: MT\-Eval five task types vs\. full KV in a dedicated transfer run,n=324n\{=\}324paired conversation evaluations: 108 conversations×\\times3 seeds\)\. Affine anchors reproduce Secs\. 4\.4/4\.6 in the same runs\. Multi\-turn effective bits include per\-row metadata; for mixed rows the reported rate uses the realized token\-weighted high\-precision fraction over the evaluated conversations rather than the nominalrralone\. Rows at generation 1\.0\-bit and multi\-turn affine\-2\.322/TQ\-2\.585 come from a dedicated add\-on run whose continuity anchors reproduce the main run exactly\.ProtocolMethodEff\. bitsΔ\\Delta\(pp\)NoteGen\.Affine gen 2\.0b2\.250−0\.0157\-0\.0157\(p<10−5p\{<\}10^\{\-5\}\)anchor \(Sec\. 4\.4\)Gen\.TQ gen 2\.322b2\.458−0\.0031\-0\.0031\(0\.13\)safeGen\.TQ gen 2\.0b2\.125\+0\.0004\+0\.0004\(0\.88\)safeGen\.TQ gen 1\.585b1\.725−0\.0031\-0\.0031\(0\.34\)safeGen\.TQ gen 1\.0b1\.125−0\.0329\-0\.0329\(p<10−18p\{<\}10^\{\-18\}\)gen cliff:\(1\.0,1\.585\]\(1\.0,1\.585\]MTAffine uniform 2\.0b2\.250−0\.0700\-0\.0700\(p<10−48p\{<\}10^\{\-48\}\)anchor \(Sec\. 4\.6\)MTAffine uniform 2\.322b2\.583−0\.0209\-0\.0209\(p<10−10p\{<\}10^\{\-10\}\)above\-cliff floor \(affine\)MTTQ uniform 2\.322b2\.458−0\.0083\-0\.0083\(p<10−3p\{<\}10^\{\-3\}\)floor,2\.5×2\.5\\timessmaller than affineMTTQ uniform 2\.585b2\.725−0\.0077\-0\.0077\(p<10−3p\{<\}10^\{\-3\}\)floor flat in bits \(p=0\.77p\{=\}0\.77\)MTTQ uniform 2\.0b2\.125−0\.0222\-0\.0222\(p<10−11p\{<\}10^\{\-11\}\)68%68\\%of collapse removedMTTQ uniform 1\.585b1\.725−0\.0912\-0\.0912\(p<10−79p\{<\}10^\{\-79\}\)cliff \(one step below affine\)MTSemKV \(TQ\) 2\.585/2\.322 r352\.537−0\.0074\-0\.0074\(p<10−3p\{<\}10^\{\-3\}\)==u2\.322 ceiling \(p=0\.68p\{=\}0\.68\)MTSemKV \(TQ\) 2\.322/2\.0 r352\.236−0\.0111\-0\.0111\(p<10−5p\{<\}10^\{\-5\}\)\+0\.011\+0\.011vs\. u2\.0 \(p<10−3p\{<\}10^\{\-3\}\)

![Refer to caption](https://arxiv.org/html/2608.28911v1/fig18.png)Fig\. 18:Base\-quantizer transfer: single\-turn uniform sweep from 3\.0 down to 1\.0 code bits \(3 seeds; error bars: 95% CI of the paired difference\)\. The affine cliff at\(2\.0,2\.322\]\(2\.0,2\.322\]moves one grid step down to\(1\.585,2\.0\]\(1\.585,2\.0\]under TurboQuant\-MSE; the SemKVr=75%r\{=\}75\\%mix occupies the affine\-inaccessible gap at code 1\.9 bits, statistically indistinguishable from full KV\.![Refer to caption](https://arxiv.org/html/2608.28911v1/fig19.png)Fig\. 19:Base\-quantizer transfer across protocols \(3 seeds; error bars: 95% CI\)\. The downward shift transfers across all three protocols: TurboQuant collapses at 1\.585 bits in prefill and multi\-turn and at 1\.0 bit in generation \(n/m: multi\-turn not measured at 1\.0 bit\)\. Dotted blue markers give same\-protocol affine anchors — the 2\.0\-bit collapse anchors of each protocol, and the multi\-turn affine 2\.322\-bit floor, which TurboQuant shrinks2\.5×2\.5\\times\.

## 5Discussion

What mixed precision is for\.The cliff map resolves an apparent tension in the literature: importance\-aware allocation sometimes helps greatly and sometimes not at all\. In the flat regime above the cliff, preserving all tokens at above\-cliff precisions is sufficient and the tested indicators are empirically interchangeable \(a Llama operating\-point result; Sec\. 4\.9 shows that random allocation is not safe on Mistral\) — there, the value of SemKV is*grid interpolation*, reaching average precisions uniform quantization cannot realize\. At and below the cliff, structure alone no longer protects quality and selection becomes decisive, to the point that a poorly aligned indicator is worse than random\. The interior probes make the below\-cliff half of this picture concrete: recovery is monotone but slow in the protected fraction, and concentrating protection on high\-importance tokens is worse than spreading it, because below\-cliff degradation is broad\-based rather than evidence\-localized\. This two\-regime view predicts, and our interior, stress, and transfer results confirm, that the robust operating recipe is: map the collapse cliff for the target*deployment setting*and interpolate on its high\-precision side; whether the resulting point is statistically “safe” must still be evaluated under the target protocol\. If an aggressive configuration includes a below\-cliff component, informative selection becomes important for limiting degradation\. The base\-quantizer transfer of Sec\. 4\.9 makes the pair\-dependence concrete: a distortion\-optimized base \(TurboQuant\-MSE\) moves the collapse boundary down by one grid step in prefill and multi\-turn — and by at least that much on the generation side — while the interpolation mechanism and its quality carry over unchanged; the one qualification is multi\-turn, where a small above\-cliff floor persists under*either*base — the affine floor is2\.5×2\.5\\timesdeeper than TurboQuant’s, more bits do not remove it, and its task profile mirrors the collapse ordering — setting a ceiling that selection reaches \(remaining statistically indistinguishable from the upper grid point while using fewer bits on both bases\) but cannot exceed\. This floor is also what reconciles the flat prefill\-only sweep with the original TurboQuant report of marginal degradation at 2\.5 bits on the full cache\. The cliff map is thus not a fixed constant of the model but a measurement the recipe prescribes; better quantizers lower the cliff and thereby raise the compression available above it\.

Deletion versus precision redistribution\.SemKV redistributes precision instead of deleting tokens, preserving the attention graph\. The controlled comparison shows that deletion fails even with a larger memory budget and across the tested selection indicators; low\-precision preservation of every token is the operative mechanism, consistent with observations that retaining evicted pairs at low precision recovers most eviction damage \[71\]\.

Limitations\.Under the affine base, the primary 7\.5k evaluations show a consistent catastrophic transition at 2\.0 bits across the tested protocols and two backbones; however, the measured bracket can vary with context\-budget composition \(the 4k setting in Sec\. 4\.9\), evaluation protocol \(the generation\-side TurboQuant boundary sits one step lower\), and base quantizer, and the cliff*depth*is seed\- and task\-dependent\. Moreover, the stress evidence for indicator separation rests on a small below\-cliff sample\. The comparison to pruning is a structural control under a shared indicator, not a benchmark against engineered eviction systems with recency windows and sink handling\. Scoring\-cost and storage measurements are microbenchmarks on a single GPU class, and the storage figures assume the mixed\-radix packed layout of Sec\. 4\.1 rather than an optimized fused kernel\. The base\-quantizer transfer \(Sec\. 4\.9\) now covers prefill, generation\-time, and multi\-turn protocols on Llama, but not other backbones; the TurboQuant comparison uses our reimplementation with documented choices where the original leaves details unspecified \(outlier\-channel selection; rotation sharing\)\. Full\-cache multi\-turn quantization retains a small above\-cliff floor under both bases that no allocation policy we tested removes; the compounding\-through\-dialogue interpretation of this floor is inferred from its task profile and bits\-insensitivity rather than directly ablated; and the 2\.322/2\.0 multi\-turn interpolation point, while superior to uniform 2\.0\-bit, is not treated as a no\-detectable\-loss operating point\. Extending the cliff map to more backbones and to KV\-reuse serving stacks is left to future work\.

## 6Conclusion

This paper proposed SemKV, an all\-token\-preserving mixed\-precision KV cache compression framework grounded in an empirical quality\-cliff map of uniform KV quantization on a fractional\-bit grid\. On Llama\-3\.1\-8B under the affine base, the cliff lies in\(2\.0,2\.322\]\(2\.0,2\.322\]bits and co\-occurs with runaway generation; SemKV interpolates strictly above it and shows no statistically detectable deficit from full KV at 2\.39 average bits on our LongBench protocol — a measured6\.0×6\.0\\timesKV\-cache storage reduction versus FP16, metadata included — for every one of eight model\-internal importance indicators\. The block\-exact extension carries the result to generation\-time tokens and, up to a small above\-cliff multi\-turn floor that persists under both quantizers \(with magnitude2\.5×2\.5\\timessmaller under TurboQuant\-MSE\), to multi\-turn dialogue; a controlled comparison shows deletion\-based compression failing at a1\.5×1\.5\\timeslarger measured memory budget where low\-precision preservation succeeds; a below\-cliff stress test shows where indicator quality begins to matter; and the cliff, the interpolation property, and the statistically indistinguishable mixed\-precision operating point transfer to Mistral\-7B, where informative selection is what keeps aggressive mixing safe\. The cliff, finally, is an empirical property of the target deployment setting rather than a constant of the model: swapping the affine base for TurboQuant\-MSE lowers the collapse boundary by at least one grid step in prefill, generation, and multi\-turn alike, and SemKV converts the extra headroom directly into compression — interpolating at 2\.0/1\.585 bits inside the grid gap that the affine base cannot even reach, it is statistically indistinguishable from full KV at an effective 2\.025 bits/value, improving the no\-detectable\-loss operating point from6\.0×6\.0\\timesto7\.9×7\.9\\times\. These results suggest that better base quantizers are inputs to SemKV rather than competitors: when a stronger base lowers the measured cliff, the same rule — map the cliff, interpolate above it — converts that shift into additional memory savings\.

## Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: A patent application related to the method described in this manuscript has been filed by ETRI \(Korean patent application, 2026\)\.

## Declaration of Generative AI and AI\-assisted technologies in the writing process

During the preparation of this work, the author used generative AI and AI\-assisted technologies solely to assist with English grammar, wording, and readability\. The author reviewed and edited all AI\-assisted suggestions and takes full responsibility for the content of the publication\.

## Data availability

The benchmark datasets used in this study \(LongBench, MT\-Eval\) are publicly available\. Code and experiment configurations will be made available upon reasonable request\.

## Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(RS\-2024\-00336738, Development of Complex Task Planning Technologies for Autonomous Agents\)\.

## References

- \[1\]Hooper et al\., “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[2\]Liu et al\., “KIVI: A Tuning\-Free Asymmetric 2bit Quantization for KV Cache,” Proceedings of Machine Learning and Systems \(MLSys\), 2025\.
- \[3\]Xiao et al\., “SmoothQuant: Accurate and Efficient Post\-Training Quantization for Large Language Models,” International Conference on Machine Learning \(ICML\), 2023\.
- \[4\]Frantar et al\., “GPTQ: Accurate Post\-Training Quantization for Generative Pre\-trained Transformers,” International Conference on Learning Representations \(ICLR\), 2023\.
- \[5\]Lin et al\., “AWQ: Activation\-aware Weight Quantization for LLM Compression and Acceleration,” Proceedings of Machine Learning and Systems \(MLSys\), 2024\.
- \[6\]Dettmers et al\., “LLM\.int8\(\): 8\-bit Matrix Multiplication for Transformers at Scale,” Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[7\]Yao et al\., “ZeroQuant: Efficient and Affordable Post\-Training Quantization for Large\-Scale Transformers,” Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[8\]Lin et al\., “QServe: W4A8KV4 Quantization and System Co\-design for Efficient LLM Serving,” Proceedings of Machine Learning and Systems \(MLSys\), 2024\.
- \[9\]Zandieh et al\., “TurboQuant: Online Vector Quantization with Near\-optimal Distortion Rate,” International Conference on Learning Representations \(ICLR\), 2026\. arXiv:2504\.19874\.
- \[10\]Ashkboos et al\., “QuaRot: Outlier\-Free 4\-Bit Inference in Rotated LLMs,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[11\]Zhao et al\., “Atom: Low\-Bit Quantization for Efficient and Accurate LLM Serving,” Proceedings of Machine Learning and Systems \(MLSys\), 2024\.
- \[12\]Sheng et al\., “FlexGen: High\-Throughput Generative Inference of Large Language Models with a Single GPU,” International Conference on Machine Learning \(ICML\), 2023\.
- \[13\]Zhang et al\., “H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models,” Advances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[14\]Li et al\., “SnapKV: LLM Knows What You are Looking for Before Generation,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[15\]Cai et al\., “PyramidKV: Dynamic KV Cache Compression Based on Pyramidal Information Funnel,” Conference on Language Modeling \(COLM\), 2025\.
- \[16\]Xiao et al\., “Efficient Streaming Language Models with Attention Sinks,” International Conference on Learning Representations \(ICLR\), 2024\.
- \[17\]Ge et al\., “Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs,” International Conference on Learning Representations \(ICLR\) 2024\.
- \[18\]Liu et al\., “ChunkKV: Semantic\-Preserving KV Cache Compression for Efficient Long\-Context LLM Inference,” Advances in Neural Information Processing Systems \(NeurIPS\), 2025\.
- \[19\]Xiao et al\., “InfLLM: Training\-Free Long\-Context Extrapolation for LLMs with an Efficient Context Memory,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[20\]Liu et al\., “RetrievalAttention: Accelerating Long\-Context LLM Inference via Vector Retrieval,” arXiv preprint, 2024\.
- \[21\]Fu et al\., “LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference,” arXiv preprint, 2024\.
- \[22\]Rao et al\., “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification,” Advances in Neural Information Processing Systems \(NeurIPS\), 2021\.
- \[23\]Bolya et al\., “Token Merging: Your ViT But Faster,” International Conference on Learning Representations \(ICLR\), 2023\.
- \[24\]Ryoo et al\., “TokenLearner: Adaptive Space\-Time Tokenization for Videos,” Advances in Neural Information Processing Systems \(NeurIPS\), 2021\.
- \[25\]Child et al\., “Generating Long Sequences with Sparse Transformers,” arXiv preprint arXiv:1904\.10509, 2019\.
- \[26\]Beltagy et al\., “Longformer: The Long\-Document Transformer,” arXiv preprint arXiv:2004\.05150, 2020\.
- \[27\]Zaheer et al\., “Big Bird: Transformers for Longer Sequences,” Advances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[28\]Dao et al\., “FlashAttention: Fast and Memory\-Efficient Exact Attention with IO\-Awareness,” Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[29\]Dao, “FlashAttention\-2: Faster Attention with Better Parallelism and Work Partitioning,” International Conference on Learning Representations \(ICLR\), 2024\.
- \[30\]Su et al\., “RoFormer: Enhanced Transformer with Rotary Position Embedding,” Neurocomputing, vol\. 568, 2024\.
- \[31\]Vaswani et al\., “Attention Is All You Need,” Advances in Neural Information Processing Systems \(NeurIPS\), 2017\.
- \[32\]Brown et al\., “Language Models are Few\-Shot Learners,” Advances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[33\]Touvron et al\., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302\.13971, 2023\.
- \[34\]Touvron et al\., “Llama 2: Open Foundation and Fine\-Tuned Chat Models,” arXiv preprint arXiv:2307\.09288, 2023\.
- \[35\]AI@Meta, “Llama 3 Model Card,” 2024\.
- \[36\]Jiang et al\., “Mistral 7B,” arXiv preprint arXiv:2310\.06825, 2023\.
- \[37\]Shazeer, “Fast Transformer Decoding: One Write\-Head is All You Need,” arXiv preprint arXiv:1911\.02150, 2019\.
- \[38\]Pope et al\., “Efficiently Scaling Transformer Inference,” Proceedings of Machine Learning and Systems \(MLSys\), 2023\.
- \[39\]Devlin et al\., “BERT: Pre\-training of Deep Bidirectional Transformers for Language Understanding,” NAACL\-HLT, 2019\.
- \[40\]Bai et al\., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL, 2024\.
- \[41\]Liu et al\., “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics \(TACL\), 2024\.
- \[42\]Bai et al\., “LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long\-context Multitasks,” arXiv preprint, 2024\.
- \[43\]Zhang et al\., “InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens,” arXiv preprint, 2024\.
- \[44\]OpenAI, “GPT\-4 Technical Report,” arXiv preprint arXiv:2303\.08774, 2023\.
- \[45\]Anthropic, “Claude 3 Model Card,” 2024\.
- \[46\]Google DeepMind, “Gemini: A Family of Highly Capable Multimodal Models,” arXiv preprint, 2023\.
- \[47\]Tay et al\., “Efficient Transformers: A Survey,” ACM Computing Surveys, 2022\.
- \[48\]Micikevicius et al\., “Mixed Precision Training,” International Conference on Learning Representations \(ICLR\), 2018\.
- \[49\]Rhu et al\., “vDNN: Virtualized Deep Neural Networks for Scalable, Memory\-Efficient Neural Network Design,” International Symposium on Microarchitecture \(MICRO\), 2016\.
- \[50\]Rajbhandari et al\., “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” SC Conference, 2020\.
- \[51\]Rajbhandari et al\., “ZeRO\-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning,” SC Conference, 2021\.
- \[52\]Shoeybi et al\., “Megatron\-LM: Training Multi\-Billion Parameter Language Models Using Model Parallelism,” arXiv preprint arXiv:1909\.08053, 2019\.
- \[53\]Narayanan et al\., “Efficient Large\-Scale Language Model Training on GPU Clusters Using Megatron\-LM,” SC Conference, 2021\.
- \[54\]Korthikanti et al\., “Reducing Activation Recomputation in Large Transformer Models,” Proceedings of Machine Learning and Systems \(MLSys\), 2023\.
- \[55\]Yin et al\., “A\-ViT: Adaptive Tokens for Efficient Vision Transformer,” IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2022\.
- \[56\]Li et al\., “BRECQ: Pushing the Limit of Post\-Training Quantization by Block Reconstruction,” International Conference on Learning Representations \(ICLR\), 2021\.
- \[57\]Banner et al\., “Post Training 4\-bit Quantization of Convolutional Networks for Rapid\-Deployment,” Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.
- \[58\]Choi et al\., “Towards the Limit of Network Quantization,” International Conference on Learning Representations \(ICLR\), 2017\.
- \[59\]Nagel et al\., “Data\-Free Quantization Through Weight Equalization and Bias Correction,” International Conference on Computer Vision \(ICCV\), 2019\.
- \[60\]Krizhevsky, “One Weird Trick for Parallelizing Convolutional Neural Networks,” arXiv preprint arXiv:1404\.5997, 2014\.
- \[61\]Lepikhin et al\., “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding,” International Conference on Learning Representations \(ICLR\), 2021\.
- \[62\]Fedus et al\., “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” Journal of Machine Learning Research \(JMLR\), 2022\.
- \[63\]Chowdhery et al\., “PaLM: Scaling Language Modeling with Pathways,” Journal of Machine Learning Research \(JMLR\), 2023\.
- \[64\]Hoffmann et al\., “Training Compute\-Optimal Large Language Models,” Advances in Neural Information Processing Systems \(NeurIPS\), 2022\.
- \[65\]Press et al\., “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation,” International Conference on Learning Representations \(ICLR\), 2022\.
- \[66\]Mu et al\., “Learning to Compress Prompts with Gist Tokens,” Advances in Neural Information Processing Systems \(NeurIPS\), 2023\.
- \[67\]Liu et al\., “Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[68\]Sun et al\., “HAWQ: Hessian Aware Quantization of Neural Networks with Mixed\-Precision,” International Conference on Computer Vision \(ICCV\), 2019\.
- \[69\]He et al\., “Deep Residual Learning for Image Recognition,” IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2016\.
- \[70\]Kingma and Ba, “Adam: A Method for Stochastic Optimization,” International Conference on Learning Representations \(ICLR\), 2015\.
- \[71\]Yang et al\., “No Token Left Behind: Reliable KV Cache Compression via Importance\-Aware Mixed Precision Quantization,” arXiv preprint arXiv:2402\.18096, 2024\.
- \[72\]Dong et al\., “QAQ: Quality Adaptive Quantization for LLM KV Cache,” arXiv preprint arXiv:2403\.04643, 2024\.
- \[73\]Duanmu et al\., “SKVQ: Sliding\-Window Key and Value Cache Quantization for Large Language Models,” Conference on Language Modeling \(COLM\), 2024\.
- \[74\]He et al\., “ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification,” Advances in Neural Information Processing Systems \(NeurIPS\), 2024\.
- \[75\]Kang et al\., “GEAR: An Efficient KV Cache Compression Recipe for Near\-Lossless Generative Inference of LLM,” arXiv preprint arXiv:2403\.05527, 2024\.
- \[76\]Yue et al\., “WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More,” arXiv preprint arXiv:2402\.12065, 2024\.
- \[77\]Su et al\., “RotateKV: Accurate and Robust 2\-Bit KV Cache Quantization for LLMs via Outlier\-Aware Adaptive Rotations,” arXiv preprint arXiv:2501\.16383, 2025\.
- \[78\]Liu et al\., “IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact,” Findings of the Association for Computational Linguistics \(ACL Findings\), 2024\.
- \[79\]Zhang et al\., “More Tokens, Lower Precision: Towards the Optimal Token\-Precision Trade\-off in KV Cache Compression,” arXiv preprint arXiv:2412\.12706, 2024\.
- \[80\]Kwan et al\., “MT\-Eval: A Multi\-Turn Capabilities Evaluation Benchmark for Large Language Models,” Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\), 2024\.
- \[81\]Kim et al\., “KVzip: Query\-Agnostic KV Cache Compression with Context Reconstruction,” Advances in Neural Information Processing Systems \(NeurIPS\), 2025\.