Tag
A new checkpoint for Qwen3.8 Flash-Next offloads the ngram lookup table to SSD, reducing RAM usage by 48 GB while maintaining inference speed in SGLang.
A developer showcases Mference, a new inference engine that runs MoE models like DeepSeek-V4-Flash on just ~5.3GB of memory by streaming experts from SSD, with a native Mac app and OpenAI-compatible server.
Slipstream streams MoE expert weights from SSD instead of RAM, enabling large coding models (35B–480B) on 36 GB MacBooks. Benchmarks show ~13–19 tok/s for 35B models and ~2.8 tok/s for 118B, with honest reporting of failed approaches.
GLM 5.2 model runs with Q2_K quantized routed experts (effective 2.6 bits) using SSD streaming on an M5 Max 128GB computer.
A guide on running DeepSeek 4 flash on a Mac M3 Max with 96GB RAM using Antirez's ds4 engine and SSD streaming, achieving ~12 tokens/second inference speed.
DeepSeek v4 PRO, a 1.6 trillion parameter model, is running via SSD streaming on a 128GB MacBook m5 max, demonstrating local inference of a massive model.