@jun_song: If we ever figure out how to load ONLY the active params of an MoE into the GPU instead of the full weights, it's game …

X AI KOLs Following News

Summary

The author speculates that loading only active parameters of MoE models onto GPUs could drastically improve efficiency and allow running large models like Kimi locally, though acknowledges this is currently impractical.

If we ever figure out how to load ONLY the active params of an MoE into the GPU instead of the full weights, it's game over. Data centers would see a 100x efficiency boost. And we could literally run 1T models like Kimi locally on just 32GB VRAM. Yeah I know it's basically impossible right now, but who knows what the future holds. Let me dream.
Original Article

Similar Articles

An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk

Reddit r/LocalLLaMA

作者将上周的 MoE 专家按需从磁盘流式加载的方案接入 llama.cpp 的 WebGPU 后端,使 80B MoE 模型(Qwen3-Next,48GB GGUF)也能在 Chrome 标签页中运行,无需为每个模型手写引擎。该方案新增了 ggml-webgpu 的缓冲类型、slot 映射的 MUL_MAT_ID 着色器和驻留逻辑,实测在 M4 Pro MacBook 上以 2.9 tok/s 解码 80B 模型,较小模型可达 22 tok/s。

Multi Tier MoE Caching

Reddit r/LocalLLaMA

Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.

Smaller, faster, safer: running Kimi and GLM at scale

Hacker News Top

Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.