A user details their first attempt at tuning the Qwen3.8-27B model with Q4_K_M quantization on an RTX 5080 16GB, achieving about 13.2 tokens per second at 50-61K context by selectively offloading FFN tensors to CPU to improve performance.
This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were much slower than reports here. Hardware: RTX 5080 16 GB i5-14600K 64 GB DDR5-5600 (4 DIMMs) Windows Final model/runtime: Unsloth Qwen3.8-27B UD-Q4_K_M, unmodified (16.46 GB) official llama.cpp b10760 CUDA 13.3 build 65,536 context, one slot Q4_0 K/V cache, Flash Attention medium thinking, text only Pi as the coding agent Results: 49,738 input tokens: 13.247 / 13.260 / 13.261 tok/s across three runs 61,238 input tokens: 13.055 tok/s 4/4 retrieval in every run Pi read a broken implementation plus a separate test, edited only the implementation, ran PowerShell, and got PASS The useful change was selective FFN placement. I kept attention/KV and most tensors on the GPU, but moved the 16 largest FFN tensor groups (about 2.764 GiB) to CPU. Whole-layer offload in LM Studio gave me only 6.633 tok/s around 50K. MTP was surprisingly worse on this machine at deep context. MTP1 reached 8.654 tok/s and MTP3 7.810 tok/s, while disabling MTP reached 13.256 tok/s. My guess is that the CPU-side draft competed for RAM bandwidth with the spilled FFNs. I originally chased the recent ~75 tok/s 5080 post, but the linked 13.5 GB custom quant uses IQ3_S for its FFN tensors. That is a valid speed tradeoff, but I specifically wanted true Q4 weights and a deep-context measurement. I published the exact Windows launcher, tensor override, Pi config, benchmark harness, raw results, model SHA, failed profiles, and methodology here: https://github.com/johnconnor2020/qwen38-27b-rtx5080-16gb Caveats: the recall prompt is synthetic, the Pi task is a practical smoke test rather than LiveCodeBench/SWE-bench, and runs 2/3 reused prompt cache for ingestion (decode speed stayed the same). This is also likely sensitive to RAM bandwidth and llama.cpp version. I would be very interested in comparable true-Q4 50K+ results from other 16 GB cards, or suggestions for a better coding-quality benchmark that is practical to run locally.
This article demonstrates the practical setup and performance benchmarks for running the Qwen3.8-27B model on a single NVIDIA RTX 5090 with a 262K token context window using vLLM, including detailed metrics and configuration steps.
A user shares achieving 80 tok/s on a Qwen3.6 35B model with Q4_K_M quantization and 100k context on a single Nvidia P40 using TheTom's TurboQuant fork of llama.cpp, highlighting various optimizations.
Running Qwen3.6 27B on an RTX 5090, achieving 6.4k tokens per second after tuning MTP and cache settings, demonstrating optimization techniques for inference.
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
A user shares their experience running a quantized Qwen 3.8 27B model using QAT Q2 and Q5 KV, achieving high performance on a 12GB GPU with up to 200K token context, surpassing models like Sonnet 4.6.