The author finds that running Qwen 3.8 Next Flash on Exllama3 at 3.05 bpw on 3x 3090 GPUs delivers exceptional performance and quality for local LLM usage, outperforming other quantizations.
I have been anti-low quant. I can feel the difference, I swear. I am also anti-quantized KV cache. I have been burned there - I think the tradeoffs compound, especially at longer contexts. Due to this, I have been running Qwen 3.8 27B (UD-Q8_K_L) on 2x of the 3090s in TP. I swear by it. It is such an amazing little model and I can get nearly max context at full FP16 KV cache and it feels great and is an amazing coding agent. I've built whole projects with this thing. Unlike the 3.6 version I can run 3.8 into the 250k+ context with zero sense of degradation. Using 3.6 I only felt comfortable up to like 150k before I thought about compacting or starting a new session. However - I have this hanger-on 3rd 3090. What I have been doing is using it for experimenting with LoRAs, other GPU-usable models like Datadog's Toto. Sometimes I use it to load a bigger model but the tradeoff in speed and quantization has rarely been worth it. Until now. I loaded up the UD-Q4_K_XL of Qwen 3.8 Flash Next and I'll be honest - it ran way slower that I would have liked. Not all of the model layers fit on VRAM - some were offloaded to RAM. I am sure there was something undertuned in my system, but I got about 50t/s max output, average of like 38, and PP was abysmal - in the low hundreds. After living life at 75-100 t/s+ and 1,000 t/s PP this was not something I could deal with. Except I saw the output. Holy mother of god. I have the stupidest benchmark I run 1st thing on every single model I try. I make it make me flappy bird. I use a basic prompt - nothing special. Just in a chat session, so no tools. I know it sounds ridiculous, but because I have ran this prompt hundreds of times I can read a LOT of things about a model by what it produces. All of the little details, play feel, if it includes sound, graphical 'juice', scrolling backgrounds, etc. And because it is always the same I get a feel for how fast the model runs. I have never seen any flappy bird like this one. It has an entire UI built around the game window - every single FB i have ever made only had the game in the middle. It has the coolest animations. It has MULTIPLE mechanics: - a 'bonus point' score if you get close to a pipe without hitting it (seen it before on good models) - THE PIPES START GOING UP AND DOWN <---- totally unique and very well done You might say to yourself, "Well that is just in the training data." And I say back to you - I KNOW! And it rocks. If it is in the training data of every model then you'd expect all of them to be about the same. Many of them are but this one is special. It obviously has way more going on in terms of world knowledge than the 3.8 27B model. Anyway - that is great and all but I cannot use a model like that at <250 t/s PP and 35 t/s or slower at long context. Oh well. But I have been playing around with the EXL3 quants, too (not worth it for 27B on 2x 3090s, btw, lccp just as good or better), so I noticed that I could get the 3.05bpw Qwen 3.8 Next Flash model running entirely in VRAM at full context and FP16 KV Cache with only the engram table offloaded to RAM. So I gave it a shot. I am telling you right now - if you gave me this model and asked me what quant it is I would say 100% it is a Q8 quant. This is the most coherent, capable, knowledgeable, tool-callingest 3-bit model I have ever used. Zero difference between it and the Q4_K_XL model except way faster on VRAM. Maybe the Unsloth Q3 quants are just as good - check them out and let me know. It slices. It dices. It runs at 110+ t/s with PP over 1,500 t/s. If you think those numbers are bad on 3x 3090s, they might be. I am running all of them at 250-265W and one is on a TB4 eGPU. Not ideal and I still get these numbers. Insanity for the quality. TL;DR: I ranted for like 8 paragraphs. Use Qwen 3.8 Next Flash on Exllama3 (exl3) at 3.05 bpw if you have 3x 3090s.
The article compares llama.cpp backends for running Qwen 3.6 27B on an RTX 3090 24GB, finding ik_llama.cpp with IQ4_KS quantization yields the best performance (1261 tok/s prefill, 72.9 tok/s decode).
A user details their first attempt at tuning the Qwen3.8-27B model with Q4_K_M quantization on an RTX 5080 16GB, achieving about 13.2 tokens per second at 50-61K context by selectively offloading FFN tensors to CPU to improve performance.
Detailed breakdown of running Qwen 3.5 122B MoE on a single RTX 3090 at 35 t/s using a custom llama.cpp fork (ik_llama.cpp) with fused MoE operations and expert offloading to CPU RAM, significantly outperforming stock llama.cpp MTP.
A detailed guide on running the Qwen3.6-35B-A3B APEX model on an RTX 3090, comparing two llama.cpp forks and quantization methods for optimal speed and quality.