@jun_song: If we ever figure out how to load ONLY the active params of an MoE into the GPU instead of the full weights, it's game …
Summary
The author speculates that loading only active parameters of MoE models onto GPUs could drastically improve efficiency and allow running large models like Kimi locally, though acknowledges this is currently impractical.
Similar Articles
@techNmak: The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy. L…
KTransformers is a framework that optimizes inference and fine-tuning of large Mixture-of-Experts models by dynamically placing only active experts on the GPU while keeping the rest in CPU memory, enabling large models like DeepSeek-V3 to run on limited consumer GPU memory.
Multi Tier MoE Caching
Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.
Smaller, faster, safer: running Kimi and GLM at scale
Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.
@nrehiew_: > LatentMoE > 16 activated experts out of 896 > Kimi Delta Attention and AttnRes > 2.5x more efficient scaling This is …
Discussion of LatentMoE architecture with extreme sparsity (16/896 experts) and Kimi Delta Attention, claiming 2.5x more efficient scaling, and speculation about Kimi K3 model capabilities.
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)
A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.