A user compares running quantized Qwen3 Next 80B and Qwen3.5 122B on a 64GB RAM system, noting the trade-offs in speed, quality, and memory usage for local LLM inference.
I've got 64GB of regular RAM, and I ran Qwen3 Next 80B at UD-Q4_K_XL, enjoying both nice performance and a lot better internal knowledge than Qwen3.5 35B A3B, which itself is better than Qwen3.6 35B A3B. With DDR4, I get 8.5 tok/s on average, However, Qwen3.5 122B A10B with the UD-Q2-K_XL quants make it possible to fit a ~120B model into the same 64GB RAM. Contrary to popular belief, UD-Q2_K_XL weights, despite being Q2 at their core, are still pretty good, even with MTP enabled. I get way higher quality responses in terms of internal knowledge, even if there's a major drop to just ~2.9 tok/s, and more specifically, a huge drop in prompt processing speed while running on CPU (yeah, I have to run both models on CPU). Supposedly, if I were to have DDR5-5600 RAM in my laptop instead, I would've already bumped up the generation speed to whopping 6 tok/s, which is already pretty usable. I find it that it is worth waiting for high quality responses, which seem to be brute-forced by the regular MoE architecture of Qwen3.5 122B having 10B active parameters, as opposed to the essentially outdated Qwen3 Next 80B's Sparse MoE architecture, which, despite the impressive 80B total parameter count, still only has ~3.5B active parameters, and that makes it miss some potentially important facts, if not completely make things up when prompting it to explain a niche subject. There are drawbacks to this, though. The huge drop in prompt processing speed makes it essentially impossible to meaningfully integrate it into agentic workloads where the goal is to look some info up in a Wikipedia archive, for instance. Without a ~60B model to cover the 32-48GB RAM range and strike a sweetspot between Qwen3.6 35B A3B's speed and Qwen3.5 122B A10B's sucklessness, I think that we would be unable to have a digital encyclopedia in only ~100GB of storage space. We'd only be able to make do with Qwen3.5/3.6 35B A3B, provided that there's even enough memory to fit it - otherwise it would be Gemma 4 26B A4B, or the dense models, especially when running an LLM on a dGPU instead of unified memory. Though, if you have the likes of an AMD Strix Halo mini-PC with 64GB or 96GB of RAM, I imagine that you're not going to have such issues. P.S. I've already tried REAP variants of Qwen3.5 122B A10B and those are completely dead in the water, they're somehow much dumber and the core problem of very slow prompt processing speed remains unresolved.
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
A developer benchmarked 21 local LLMs on MacBook Air M5 using HumanEval+ and found Qwen 3.6 35B-A3B (MoE) leads at 89.6% with 16.9 tok/s, while Qwen 2.5 Coder 7B offers the best RAM-to-performance ratio at 84.2% in 4.5 GB. Notably, Gemma 4 models significantly underperformed expectations (31.1% for 31B), possibly due to Q4_K_M quantization effects.
A user shares their experience running Qwen3-35B-A3B quantized model on an M2 MacBook Pro with 32GB RAM for coding tasks via opencode and llama.cpp, finding that the 32K context window limit causes critical memory loss during compaction, making complex coding tasks impractical. They conclude that meaningful agentic coding with this model likely requires at least 128K context, exceeding what their hardware can support.
A user shares their attempts and configurations to achieve up to 115K context on a Q8-quantized Qwen3.6-27B model using 32GB VRAM on an RTX 5090, with benchmark results and trade-offs between context length and kv-cache quantization.