MoE SSD streaming on a 64 GB Mac mini: GPU still waits 27% of decode on experts. Ideas?
Summary
A developer shares performance numbers and optimization ideas for streaming Qwen Flash Next Q4 MoE experts from SSD on a 64 GB Mac mini M5, achieving 17.5 tok/s decode with the GPU idle 27% of the time waiting on expert loads, and asks for ideas to improve expert prefetch accuracy and hit rates.
Similar Articles
Based on advice from LocalLLama implemented a n+2 lookahead prediction for ssd streaming Qwen Flash Next - and it worked!
A developer implemented an n+2 lookahead expert-prediction optimization for SSD streaming Qwen Flash Next on a Mac Mini M5, achieving +8% token throughput on chat and +4.2% on code, with the code shared on GitHub.
Qwen3.8-Flash-Next (95.5 GiB) on a 64GB Mac at ~27 tok/s, checkpoint + fork
The author optimized the Qwen3.8-Flash-Next model to run on a 64GB Mac using expert streaming and other techniques, achieving ~27 tok/s by publishing a checkpoint and a llama.cpp fork.
I run 35B–480B coding models on my 36 GB MacBook by streaming MoE experts from SSD — self-contained app, and I publish the benchmarks that *failed* too
Slipstream streams MoE expert weights from SSD instead of RAM, enabling large coding models (35B–480B) on 36 GB MacBooks. Benchmarks show ~13–19 tok/s for 35B models and ~2.8 tok/s for 118B, with honest reporting of failed approaches.
Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 → 25-29 t/s decode with the expert cache PR
User benchmarks and details a GPU-resident expert cache PR in llama.cpp that boosts decode speed for Qwen3.8-Flash-Next on a dual RTX 3090 system from 17 to 25-29 tokens per second.
Running 95.5 GiB Qwen3.8-Flash-Next at 41–52 tok/s on a 64GB Mac (1.76x faster than llama.cpp): Slipstream release, 130k context scaling, + Swift variant
作者发布 Slipstream——一款针对 Apple Silicon 的 C++ Metal 推理引擎，支持原生 SSD 专家流式加载与投机草稿解码，可在 64GB Mac 上以 41–52 tok/s 运行 95.5 GiB 的 Qwen3.8-Flash-Next 模型，比 llama.cpp 快 1.76 倍，且 130k 长上下文下速度不衰减。