Tag
A hobbyist researcher proposes a speculative decoding scheme where a 2-bit model copy on GPU drafts tokens while the full residual (6-8 bit) in system RAM verifies them, only exchanging activations over PCIe. Early measurements suggest 2-bit + 8-bit residual outperforms Q8_0 quality and the 2-bit drafter beats Gemma 4's official drafter on acceptance rate.
The author shares a technical lesson on using BF16 KV cache instead of turboquant for AI model optimization and implements CPU offloading in SlimServe for Qwen models to manage VRAM consumption.
Technical write-up on running DeepSeek-V4-Flash-0731 with full 1M context on a single RTX 5090 + 256GB DDR5 desktop using vLLM with CPU/RAM offloading, achieving ~800 tps prefill and ~15 tps decode.
The author argues that releasing smaller Qwen models (27B, 35B, 122B, 397B) would better serve the local AI community than focusing on trillion-parameter behemoths, which are impractical for most users.
DeepSeek V4 Flash (98GB) now runs up to 7 tokens per second on a single RTX 4060 Ti with CPU offloading, a 3x speed improvement over the previous week's 2 t/s.