@no_stp_on_snek: anyone still talking about mlx-swift-lm? said i was taking the day off... cleaned the chicken coop, got a workout in, f…

X AI KOLs Following Tools
kv-cache-compression mlx swift metal turboquant decode-optimization apple

Summary

The author describes implementing TurboQuant KV-cache compression into Apple's mlx-swift-lm, achieving 2.7x compression with quality on par with 8-bit, and 3-4x decode speed improvements via a fused Metal kernel.

anyone still talking about mlx-swift-lm? said i was taking the day off... cleaned the chicken coop, got a workout in, felt like a balanced human for about two hours. then i opened the laptop to check one thing. it is never one thing. spent the "day off" getting TurboQuant KV-cache compression into Apple's mlx-swift-lm. it's an asymmetric scheme by design. keys barely tolerate compression (softmax amplifies their error) so you keep K near-lossless, and values compress almost for free, so you squeeze V hard. result: 2.7x KV compression at quality on par with 8-bit, validated across 6 model families (1.7B up to a 30B MoE and a 32B) by KL-divergence against the fp16 cache. measured, not vibes. the fun part was decode speed. first pass ran at ~20% of fp16, and the bottleneck wasn't the math. it was re-materializing the entire cache every single token: full-cache f32 casts, GQA tensor expansion, 17 GPU dispatches where fp16 does 3. so i ported my llama.cpp fork's fused single-kernel decode into Metal. score, softmax, and inline dequant in one SIMD-parallel dispatch, K read once in its native basis. decode jumped 3 to 4x. now fp16-competitive at a quarter of the KV bytes. been slowly trickling in some swift work upstream. this one is a larger pr.
Original Article
View Cached Full Text

Cached at: 07/16/26, 06:17 PM

anyone still talking about mlx-swift-lm?

said i was taking the day off… cleaned the chicken coop, got a workout in, felt like a balanced human for about two hours. then i opened the laptop to check one thing. it is never one thing.

spent the “day off” getting TurboQuant KV-cache compression into Apple’s mlx-swift-lm.

it’s an asymmetric scheme by design. keys barely tolerate compression (softmax amplifies their error) so you keep K near-lossless, and values compress almost for free, so you squeeze V hard. result: 2.7x KV compression at quality on par with 8-bit, validated across 6 model families (1.7B up to a 30B MoE and a 32B) by KL-divergence against the fp16 cache. measured, not vibes.

the fun part was decode speed. first pass ran at ~20% of fp16, and the bottleneck wasn’t the math. it was re-materializing the entire cache every single token: full-cache f32 casts, GQA tensor expansion, 17 GPU dispatches where fp16 does 3.

so i ported my llama.cpp fork’s fused single-kernel decode into Metal. score, softmax, and inline dequant in one SIMD-parallel dispatch, K read once in its native basis. decode jumped 3 to 4x. now fp16-competitive at a quarter of the KV bytes.

been slowly trickling in some swift work upstream. this one is a larger pr.

Similar Articles

@no_stp_on_snek: got it here if ya want to try it out:

X AI KOLs Following

A fork of llama.cpp integrating TurboQuant+ for advanced KV-cache and weight quantization, with cross-backend kernel support (Apple Silicon, NVIDIA CUDA, AMD ROCm, Vulkan) and used in production by LocalAI, Chronara, and AtomicChat.

Dynamic KV Cache Quantization and Load-on-demand mmproj/MTP: my llama.cpp wishlist

Reddit r/LocalLLaMA

A developer has implemented a proof-of-concept PR for llama.cpp that adds dynamic KV cache quantization via an HTTP endpoint, allowing users to requantize their KV cache on-demand without fully reloading the model. The post also outlines a wishlist including load-on-demand mmproj/MTP swapping and an automatic --fit flag for context optimization.