It's unbelievable! I used the mmap function in llama.cpp to fit Qwen3.8-Flash-Next IQ3_XSS into 16G+64G RAM, and the speed still reached 26t/s.
Summary
A user successfully used the mmap function in llama.cpp to fit the Qwen3.8-Flash-Next IQ3_XSS model into 16GB+64GB RAM, achieving a speed of 26 tokens per second, which outperforms a larger non-MOE 30B model.
Similar Articles
Qwen3.8-Flash-Next, on 5090+64gb, with Llama.cpp - Seems to not use ram?
User shares performance results and setup details for running the Qwen3.8-Flash-Next model with Llama.cpp on a system featuring an NVIDIA 5090 GPU and 64GB RAM, noting unexpected low RAM usage during inference.
Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)
A developer showcases a custom `--moe-direct-io` feature for llama.cpp that achieves 10–15x speedup (20–24 tok/s) running the 125B MoE Qwen3.5-Flash Next (IQ2_XS, 68GB) on just an RTX 3060 12GB with 16GB RAM, while keeping output bit-exact to stock.
TQwen 3.8 flash next ud1s on 6gb vram and 16 gb system ram
A user shares their experience running the Qwen 3.8 flash next model on a system with 6GB VRAM and 16GB RAM using llama.cpp, achieving 6-7 tokens per second with 1-bit quantization, and asks for recommendations on quantization variants.
Maxing out 64GB of RAM - Qwen3.5 122B A10B at UD-Q2_K_XL w/ MTP fully replaced Qwen3 Next 80B at UD-Q4_K_XL for me
A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
Qwen Flash Q4_K_M on 4080 + 64GB DDR5 at ~8tk/s 98304 CTX.
The author demonstrates how to run a 182B Qwen model on a 4080 GPU with 64GB DDR5 by offloading ngrams to SSD, achieving faster and more intelligent performance than a 27B model, making large model inference accessible on consumer hardware.