@jun_song: If we ever figure out how to load ONLY the active params of an MoE into the GPU instead of the full weights, it's game …
Summary
The author speculates that loading only active parameters of MoE models onto GPUs could drastically improve efficiency and allow running large models like Kimi locally, though acknowledges this is currently impractical.
Similar Articles
@techNmak: The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy. L…
KTransformers is a framework that optimizes inference and fine-tuning of large Mixture-of-Experts models by dynamically placing only active experts on the GPU while keeping the rest in CPU memory, enabling large models like DeepSeek-V3 to run on limited consumer GPU memory.
An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk
作者将上周的 MoE 专家按需从磁盘流式加载的方案接入 llama.cpp 的 WebGPU 后端,使 80B MoE 模型(Qwen3-Next,48GB GGUF)也能在 Chrome 标签页中运行,无需为每个模型手写引擎。该方案新增了 ggml-webgpu 的缓冲类型、slot 映射的 MUL_MAT_ID 着色器和驻留逻辑,实测在 M4 Pro MacBook 上以 2.9 tok/s 解码 80B 模型,较小模型可达 22 tok/s。
Multi Tier MoE Caching
Discusses multi-tier caching strategies for MoE models to improve inference speed by keeping frequently activated experts on GPU, referencing existing implementations like PowerInfer and llama.cpp branches.
LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes
LRMoE is a memory-aware MoE training system that jointly recalibrates backward demand and balances retained workload on memory-constrained GPU nodes, reducing cumulative MoE-layer training time by 21.2-44.8% on Qwen3-30B-A3B and GLM4.5-106B-A12B compared to SYMI.
Smaller, faster, safer: running Kimi and GLM at scale
Cloudflare details how it serves large open-source MoE models like Kimi K2.6 and GLM 5.2 efficiently using FP8 KV cache quantization and weight compression, boosting throughput and cutting costs without accuracy loss.