Tag
The article expresses hope for future AI models with optimizations like KVCache and Engram to reduce memory usage, enabling larger models to run on consumer GPUs with limited VRAM.
The article explains RadixAttention, a technique in SGLang that uses a radix tree to efficiently cache and reuse overlapping KV caches across branching requests in AI serving.
Announced release of Luce KVFlash, a tool to run models inside Lucebox at 256k context without KVCache OOM, achieving up to 2.9x faster decoding at long context using speculative prefill and dynamic offloading.
A custom digital chip designed gate-by-gate achieves over 56,000 tokens/sec running a Transformer with KV cache at just 80 MHz, prototyped on an FPGA.
A detailed benchmark comparing ByteShape and Unsloth quantizations of Qwen3.6-35B-A3B on tool calling performance, KV cache quantization effects, and long context degradation using llama.cpp and tool-eval-bench.
ZCube is a new network architecture that flattens the topology and mixes single/multi-rail access to optimize KV Cache transmission in long-context and PD separation scenarios. In the GLM-5.1 production cluster, it achieved a 33% reduction in switch/optical module costs, a 15% increase in GPU inference throughput, and a 40.6% decrease in TTFT P99.