Faster than Light in Air: 8-22 tg/s Qwen3.8-Flash-Next (Q4/Q4ish) on a 32GB M4 MacBook Air

Reddit r/LocalLLaMA Tools
inference-engine apple-silicon memory-constrained quantization expert-streaming ai-optimization

Summary

Cherenkov is a new inference engine for Apple Silicon that enables efficient memory-constrained inference of large AI models like Qwen3.8-Flash-Next using predictive expert streaming and mixed-precision execution.

I believe I may currently hold the record for memory constrained inference for Qwen3.8–Flash-Next on Apple Silicon — needing only about 21GB of allocations. Introducing Cherenkov, an inference engine for Apple Silicon combining predictive expert streaming with optional mixed-precision execution. It works by keeping a bounded working set of experts in unified memory, rather than loading the entire model. A one-layer lookahead predicts which experts will be needed next and initiates SSD reads. If there isn’t enough time to load the full target expert, Cherenkov optionally falls back to just in time loading of smaller Q3/Q2 quantizations.
Original Article

Similar Articles

Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro

Reddit r/LocalLLaMA

Inco Splash is an open-source inference engine optimized for Apple silicon, offering significant speed improvements for running AI models like Qwen3.8-27B on M-series MacBooks.

Qwen3.8-Flash-Next optimised for Macs

Reddit r/LocalLLaMA

The article details custom optimizations for running the Qwen3.8-Flash-Next AI model on Mac M1 Max hardware, including SSD streaming, custom quantizations, and a sparse attention mechanism to improve performance.

Qwen Flash Q4_K_M on 4080 + 64GB DDR5 at ~8tk/s 98304 CTX.

Reddit r/LocalLLaMA

The author demonstrates how to run a 182B Qwen model on a 4080 GPU with 64GB DDR5 by offloading ngrams to SSD, achieving faster and more intelligent performance than a 27B model, making large model inference accessible on consumer hardware.