tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)
Summary
Explores predicting which MoE experts will be used for the next token to improve CPU/GPU offloading, achieving 30->150-200 tg/s speedup, and questions the feasibility of implementation.
Similar Articles
Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)
A developer benchmarks Qwen3.6-35B-A3B Q6 on an RTX 3090, showing that offloading eight MoE expert layers to CPU and increasing batch sizes improves prompt processing by 2.36× (564→1330 tok/s) with no decode speed regression, using evolutionary search to find the tuning config.
Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment
The paper presents Apex, a CPU-GPU collaborative MoE inference system that adaptively assigns GPU-missed experts to either CPU or GPU based on token load, improving prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline.
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Specter 提出了一种通过量化实现的多步推测性专家卸载方法，在单 GPU 上加速 MoE 推理，相比最先进系统平均获得 1.50 倍（最高 1.88 倍）的 TPOT 加速。
Why can't we make MoE routers predict experts needed in the next 5-10 tokens?
The user questions whether Mixture of Experts (MoE) routers can be designed to predict future expert needs for token sequences to enable faster caching between RAM and VRAM, or if a separate neural network could be trained for this purpose.
@NVIDIARTXSpark: Want more tokens per second on your NVIDIA DGX Spark? Explore three techniques to accelerate MoE inference, from optimi…
@NVIDIARTXSpark shares three techniques to accelerate Mixture-of-Experts (MoE) inference on the NVIDIA DGX Spark, covering optimized kernels, FP8 quantization, and multi-token prediction to boost tokens per second.