You can offload most of Qwen3.8-Flash-Next's KV cache to RAM with little decode slowdown

I'm pretty sure it can be done with any model based on qwen4exp, which Qwen's next local models will be based on. You can use a quant that barely fits in VRAM and still run at the model's maximum context length without kv cache quantization, since most of the KV cache can live in system RAM. I actually made it working on vLLM and now I get 1M context with 3x 3090. I get ~80 tok/s at short context, dropping to ~60 tok/s once QSA reaches its 2048-token budget, after which decode speed stays flat as total context grows. The throughput is pretty good too, and I get like 150tk/s @ 4 concurrent requests. Prefill at 248k reaches 3,701 tok/s. (The patches and the model are available on my huggingface page if you're interested) Decode speed is a bandwidth problem. Each decode step produces one token, and to produce it the GPU reads every weight and every piece of attention state that the step needs. On a single stream the card spends most of the step waiting for memory rather than computing. So the size of that per-step read sets the token rate. This is why a normal model keeps its KV cache in VRAM. Take Qwen3.8-27B, which is built on the Qwen3-Next architecture and shares most of its properties with Qwen3.8-Flash-Next (`qwen4_exp`). It still has one full attention layer every few layers, and a full attention layer reads its entire KV cache on every step. That read grows with the context, so decode gets slower as the conversation gets longer. It also grows past what any host link (such as PCIe) can carry, so the cache has to sit next to the compute. The numbers of this model show the size of the problem. One QSA layer holds 2 key/value heads of 256 dimensions, as K and as V, in 2 bytes each, which is 2,048 B per token. At 262,144 tokens that is 512 MiB for one layer, and 6 GiB for all 12 layers on every single step. A PCIe 4.0 x16 slot carries about 32 GiB/s, so a host-resident cache of that shape allows about 5 tokens per second. Here's an interesting part, Qwen3.8-Flash-Next avoids this in two ways: Only 12 of the 48 layers have a KV cache at all. The other 36 layers are gated delta-net layers, a linear attention whose recurrent state has a fixed size. That state does not grow with the context. Those 12 layers also do not attend over the whole context. QSA runs a cheap indexer over a pooled, compressed key, where `indexer_head_dim=128` divided by `indexer_compress_ratio=4` gives the pooled width. The indexer selects at most `indexer_budget=2048` positions. The layer reads the main KV rows only for the positions that the indexer selects. So `indexer_budget` bounds the bytes that a decode step reads, and the context length does not: ``` 2048 selected x 2 kv heads x 256 dim x 2 (K and V) x 2 B = 4 MiB per layer x 12 layers = 48 MiB per token ``` Take an example, at 80 tok/s that is about 3.9 GB/s across the link. It is a small fraction of a PCIe 4.0 x16 slot, and most of it overlaps with compute. Only few things need to stay on the GPU. The model itself, and a 2-byte slot plus the pooled index key, which is `1 x (128 / 4) x 2 B = 64 B`. Together they are 66 B per token per layer, against 2,048 B for a full row.