yall are sleeping on qwen 3.8 27b q2 + q2 dflash + q5 kv
Summary
A user shares their experience running a quantized Qwen 3.8 27B model using QAT Q2 and Q5 KV, achieving high performance on a 12GB GPU with up to 200K token context, surpassing models like Sonnet 4.6.
Similar Articles
Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s.
The article describes a method to run the Qwen3.8-Flash-Next model with quantized n-grams to achieve over 170K token context on a single 96GB GPU card, with performance up to 110 tokens per second using INT4 quantization and memory-mapped disk access.
@cyrilXBT: UNREAL. Qwen 3.8 27B is now happily running on an RTX 4060 locally with only 8GB of VRAM. You’re looking at a 64k conte…
A user shares that the Qwen 3.8 27B model runs locally on an RTX 4060 with only 8GB VRAM using Unsloth's IQ4_XS quantization, achieving a 64k context window and impressive performance metrics.
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context
A guide on optimizing VRAM usage on an AMD 7900XTX to run a 27B Qwen model with Q6K quantization and 131k context by compiling llama.cpp with OpenBLAS and CUDA_FA_ALL_QUANTS, and using kvcache quantization at q5_0/q4_0.
TQwen 3.8 flash next ud1s on 6gb vram and 16 gb system ram
A user shares their experience running the Qwen 3.8 flash next model on a system with 6GB VRAM and 16GB RAM using llama.cpp, achieving 6-7 tokens per second with 1-bit quantization, and asks for recommendations on quantization variants.