@anemll: Effect of V-compression (int8) in KV-cache on prefill Qwen 3.8-27B, ANE M6 ANEMLL-FORGE. Compression (Quant-dequant) is…
Summary
ANEMLL-FORGE reports on applying int8 V-compression to the KV-cache of Qwen 3.8-27B during prefill on Apple's ANE M6, with compression and dequantization performed on ANE to yield memory I/O savings.
View Cached Full Text
Cached at: 10/02/26, 06:45 PM
Effect of V-compression (int8) in KV-cache on prefill Qwen 3.8-27B, ANE M6 ANEMLL-FORGE. Compression (Quant-dequant) is done on ANE. savings are memory I/O https://t.co/55Wwj3thIK
Similar Articles
Qwen3.8-27b q8 KV cache does seem to actually hurt model performance
The article discusses how on-the-fly KV cache quantization can reduce long-context model performance due to compounding errors, based on experiments with Qwen3.8-27B.
Here are my KV cache quantization benchmarks: TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM
A detailed benchmark comparing KV cache quantization methods (TurboQuant, TCQ, q4, q5, q8) using PPL and KLD metrics on Qwen 3.6 27B, finding that TCQ improves low-bit quantization, asymmetric KV beats symmetric at same size, and q8 is often overkill. Includes analysis and data in linked article.
I tested Qwen3.8 27B KLD for 21 Models at 9 different KV quants that fit on 16GB VRAM (189 benchmarks).
A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.
You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown
A technique to offload the KV cache of Qwen3.8-Flash-Next to system RAM is demonstrated, allowing long-context inference with minimal decode slowdown by leveraging the model's efficient architecture.
Someone apparently managed to kind of replicate what V4.1 flash does on KV for fast prefill on Qwen
Someone has replicated a technique similar to V4.1 flash for fast prefill using KV caching on Qwen models, with a demo available for testing on the 27B version.