A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
I've got 64GB of regular RAM, and I ran Qwen3 Next 80B at UD-Q4_K_XL, enjoying both nice performance and a lot better internal knowledge than Qwen3.5 35B A3B, which itself is better than Qwen3.6 35B A3B. With DDR4, I get 8.5 tok/s on average, However, Qwen3.5 122B A10B with the UD-Q2-K_XL quants make it possible to fit a ~120B model into the same 64GB RAM. Contrary to popular belief, UD-Q2_K_XL weights, despite being Q2 at their core, are still pretty good, even with MTP enabled. I get way higher quality responses in terms of internal knowledge, even if there's a major drop to just ~2.9 tok/s, and more specifically, a huge drop in prompt processing speed while running on CPU (yeah, I have to run both models on CPU). Supposedly, if I were to have DDR5-5600 RAM in my laptop instead, I would've already bumped up the generation speed to whopping 6 tok/s, which is already pretty usable. I find it that it is worth waiting for high quality responses, which seem to be brute-forced by the regular MoE architecture of Qwen3.5 122B having 10B active parameters, as opposed to the essentially outdated Qwen3 Next 80B's Sparse MoE architecture, which, despite the impressive 80B total parameter count, still only has ~3.5B active parameters, and that makes it miss some potentially important facts, if not completely make things up when prompting it to explain a niche subject. There are drawbacks to this, though. The huge drop in prompt processing speed makes it essentially impossible to meaningfully integrate it into agentic workloads where the goal is to look some info up in a Wikipedia archive, for instance. Without a ~60B model to cover the 32-48GB RAM range and strike a sweetspot between Qwen3.6 35B A3B's speed and Qwen3.5 122B A10B's sucklessness, I think that we would be unable to have a digital encyclopedia in only ~100GB of storage space. We'd only be able to make do with Qwen3.5/3.6 35B A3B, provided that there's even enough memory to fit it - otherwise it would be Gemma 4 26B A4B, or the dense models, especially when running an LLM on a dGPU instead of unified memory. Though, if you have the likes of an AMD Strix Halo mini-PC with 64GB or 96GB of RAM, I imagine that you're not going to have such issues. P.S. I've already tried REAP variants of Qwen3.5 122B A10B and those are completely dead in the water, they're somehow much dumber and the core problem of very slow prompt processing speed remains unresolved.
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
Demonstrates running an 80B Qwen model in just 4.3 GB of RAM on a Mac and a 35B model on an iPhone, showcasing extreme memory optimization for local LLM inference.
A user shares their experience running the Qwen 3.8 flash next model on a system with 6GB VRAM and 16GB RAM using llama.cpp, achieving 6-7 tokens per second with 1-bit quantization, and asks for recommendations on quantization variants.
A Reddit user shares how they successfully ran the Qwen 3.8 Next MoE model on a system with 16GB VRAM and 32GB RAM using aggressive quantization and specific llama.cpp settings, achieving usable performance for large models on limited hardware.
Demonstrates running the Qwen 3.6 35B MoE model in Q4_K_M quantization on a Xiaomi 12 Pro with 12GB RAM, showing local AI inference on a mobile device.