A new vLLM fork introduces comprehensive INT8 optimization for Qwen3.8 27B on older AMD MI100 GPUs, achieving up to 972 tokens per second throughput with rigorous accuracy validation.
Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2. Its not just another vibed autoresearch loop. No, vLLM ships with very little int8 support, and this stack adds INT8 into every crevice of Qwen3.8 including in dependent libraries and new fused kernels. So no longer are your old INT8-centric cards relegated to second rate algos and suboptimal dtypes. My system runs at 15 tok/s TG on stock vLLM and is now running at 972 tok/s TG on this branch. This stack adds thorough and complete tensor parallel INT8 support across the entire Qwen3 vLLM arch stack, plus significant and thorough tuning for a 4x MI100 system. My system is benchmarking 972 TG / 5,680 PP on this model at near-reference quality. Itemized work includes: Complete W8A8 INT8 GEMM library tuned for MI100 and usage everywhere INT8 KV cache INT8 AITER Unified Attention (and fallback Triton impl) -- faster than Flash Attention INT8 Mamba and GDN attention INT8 custom allreduce / allgather optimized for XGMI interlinks INT8 Embedding Many new fused INT8 kernels More INT8s than you can count The stack's performance has been vetted for near-reference quality/accuracy by measuring KLD not just for every token, but for every GEMM, every attention block, every KV lookup, and every layer. I've left all of the diagnostic scripts for this accuracy vetting process in the vLLM branch so you can verify it yourself. This fork has an extreme level of testing that isn't present even in stock vLLM. Check your card's specs for whether its INT8 TOPS exceeds its FP8 FLOPS, and if it does, this project is for you. Much of the work is generalizable to all older GPUs (AMD and Nvidia) that don't have native FP8. In stock vLLM they will mostly use BF16/FP16 which is very slow! While I specifically tuned for MI100, this fork has many INT8 triton fallbacks, which are hardware agnostic, so those should run less optimally, but still much faster than stock vLLM for any card. So if you are technically proficient and have some of these older cards, you should be able to blow away any other high-concurrency serving options with this fork. This project should also generalize to other model architectures, but none of them have been polished up with love like Qwen3.8. This stack should work almost out of box and especially well for older AMD cards like MI50 and MI210 (you just need to build aiter for your arches). Here are the projects and models of this stack: https://github.com/curvedinf/int8-vllm https://github.com/curvedinf/int8-aiter https://huggingface.co/curvedinf/Qwen3.8-27B-GPTQ-INT8-W8A8-GS128 https://huggingface.co/curvedinf/Qwen3.8-27B-DFlash2-GPTQ-INT8-W8A8-GS128 Please submit feedback here or in the github issues of vllm. Tuning for other cards/model arches is welcome in PRs.
Benchmark results for running Qwen 3.6 27B on AMD MI50 GPUs using a custom vllm fork, achieving 52.8 tokens/s TG and 1569 tokens/s PP without quantization or MTP, demonstrating usability for agentic tasks on 2018 hardware.
User shares an optimized recipe for running Qwen 3.5 122B Int4 on a single DGX Spark with vLLM, achieving over 40 tokens per second. They invite others to try and further optimize it.
A user shares achieving 80 tok/s on a Qwen3.6 35B model with Q4_K_M quantization and 100k context on a single Nvidia P40 using TheTom's TurboQuant fork of llama.cpp, highlighting various optimizations.
A user demonstrates running the Qwen 27b AI model quantized to Q3_K_M on two RX 580 GPUs, achieving 7.39 tokens per second using old DDR3 hardware for under $100.
Running Qwen3.6 27B on an RTX 5090, achieving 6.4k tokens per second after tuning MTP and cache settings, demonstrating optimization techniques for inference.