@QuixiAI: LESSON LEARNED: Always use BF16 kv cache. I was using turboquant. Yeah the VRAM consumption sucks - so another trick is…
Summary
The author shares a technical lesson on using BF16 KV cache instead of turboquant for AI model optimization and implements CPU offloading in SlimServe for Qwen models to manage VRAM consumption.
View Cached Full Text
Cached at: 08/29/26, 12:04 PM
LESSON LEARNED:
Always use BF16 kv cache.
I was using turboquant.
Yeah the VRAM consumption sucks - so another trick is to CPU Offload the kv cache.
Implemented in SlimServe for 3090 and Qwen (it requires extensive testing to get it right)
I have further work to universalize it for all models.
I am now evaluating Quen3.8-Flash-Next on Clearwing
Eric Hartford (@QuixiAI): My experience with Qwen3.8-Flash-Next is that it can’t even keep track of a multi-turn conversation. It answers questions I asked 2 turns ago. This is at FP8
Similar Articles
16 GB VRAM purgatory discussion thread
A discussion thread sharing configurations and tips for running AI models like Qwen3.8-27B on 16 GB VRAM Windows systems, focusing on memory optimization techniques.
@DavidFSWD: THE MOST Amzing thing in Local AI this week is this optimized Qwen 3.8 model: ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF…
This tweet highlights an optimized version of the Qwen 3.8 model that can run efficiently on local hardware with under 16GB VRAM, achieving 30-80 tokens per second on older GPU setups.
Qwen3.8-27b q8 KV cache does seem to actually hurt model performance
The article discusses how on-the-fly KV cache quantization can reduce long-context model performance due to compounding errors, based on experiments with Qwen3.8-27B.
You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown
A technique to offload the KV cache of Qwen3.8-Flash-Next to system RAM is demonstrated, allowing long-context inference with minimal decode slowdown by leveraging the model's efficient architecture.
@sakurayukiai: Counting the draft model's KV cache in bytes instead of hiding it inside a flat VRAM cushion is how Unsloth pushed Qwen…
Unsloth improved Qwen3.6-27B Q6_K context length from 23K to 64K on a single 32GB card by accurately counting the draft model's KV cache in bytes instead of using a flat VRAM cushion.