@QuixiAI: LESSON LEARNED: Always use BF16 kv cache. I was using turboquant. Yeah the VRAM consumption sucks - so another trick is…
Summary
The author shares a technical lesson on using BF16 KV cache instead of turboquant for AI model optimization and implements CPU offloading in SlimServe for Qwen models to manage VRAM consumption.
View Cached Full Text
Cached at: 08/29/26, 12:04 PM
LESSON LEARNED:
Always use BF16 kv cache.
I was using turboquant.
Yeah the VRAM consumption sucks - so another trick is to CPU Offload the kv cache.
Implemented in SlimServe for 3090 and Qwen (it requires extensive testing to get it right)
I have further work to universalize it for all models.
I am now evaluating Quen3.8-Flash-Next on Clearwing
Eric Hartford (@QuixiAI): My experience with Qwen3.8-Flash-Next is that it can’t even keep track of a multi-turn conversation. It answers questions I asked 2 turns ago. This is at FP8
Similar Articles
16 GB VRAM purgatory discussion thread
A discussion thread sharing configurations and tips for running AI models like Qwen3.8-27B on 16 GB VRAM Windows systems, focusing on memory optimization techniques.
Qwen3.8-27b q8 KV cache does seem to actually hurt model performance
The article discusses how on-the-fly KV cache quantization can reduce long-context model performance due to compounding errors, based on experiments with Qwen3.8-27B.
@sakurayukiai: Counting the draft model's KV cache in bytes instead of hiding it inside a flat VRAM cushion is how Unsloth pushed Qwen…
Unsloth improved Qwen3.6-27B Q6_K context length from 23K to 64K on a single 32GB card by accurately counting the draft model's KV cache in bytes instead of using a flat VRAM cushion.
@TheAhmadOsman: Been playing with @PrismML's new model that turned Qwen 3.5 27B into a sub-4GB and sub-6GB weights and I am impressed C…
PrismML released a compressed version of Qwen 3.5 27B that fits in sub-4GB and sub-6GB memory, enabling impressive local AI performance.
@TraffAlex: AI MODELS FOR 32GB VRAM — TOP 17 CHEAT SHEET Hit the HuggingFace API, grabbed real .gguf Q4 sizes. Every link = direct …
A cheat sheet listing top AI models optimized for 32GB VRAM using GGUF Q4 quantization, with direct download links from HuggingFace. Includes models from Qwen, DeepSeek, Llama, and Mistral families, with tips on quantization and context settings.