Tag
The paper presents Apex, a CPU-GPU collaborative MoE inference system that adaptively assigns GPU-missed experts to either CPU or GPU based on token load, improving prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline.
Appreciation post for tcclaviger adding expert RAM offloading support to vLLM, enabling running large AI models like DeepSeek-V4-Flash-Vision-Exp on local setups with multiple GPUs.
An implementation of expert lookahead achieves over 10% performance improvement for MoE models running on low-memory devices using slotstream, with additional gains from a correction model.
Luce Spark is an open-source tool that enables running 35B MoE models on 16GB GPUs by intelligently caching hot experts on the GPU while keeping the rest in system RAM, using a calibrated placement and bounded async cache to maintain high throughput without the usual offload speed cliff.