366 t/s Qwen3.6 27B NVFP4 on v100s
Summary
Introduces v100-skinny, a custom kernel library enabling fast NVFP4 inference on V100 (sm70) GPUs, achieving 366 t/s for Qwen3.6 27B in best-case extraction, with lower speeds for structured generation and code.
Similar Articles
2.5x faster Qwen3.6 NVFP4 Unsloth quants
Unsloth releases quantized Qwen3.6 models using NVFP4 format, achieving 2.5x faster inference speeds.
Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100
RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100, enabling full context operation on 4xV100 GPUs with performance metrics showing high throughput and context handling up to 256k tokens.
NVFP4 on VOLTA! Despite being built for Blackwell, I made four 2017 V100s run Qwen 3.8 NVFP4 natively and match my $6000 RTX 5090.
The author achieved performance parity between four 2017 Tesla V100 GPUs and a modern RTX 5090 when running the Qwen 3.8 model with NVFP4 precision, using custom software optimizations like the QPN kernel for efficient inference.
@witcheer: everyone says NVFP4 makes blackwell cards "faster." I benchmarked Qwen3.6-27B three ways on my 5090: >NVFP4 >plain Q4_K…
A benchmark of NVFP4 on an RTX 5090 with Qwen3.6-27B shows prefill speed gains of 32-42% over equal-bit Q4_K_M and 52-68% over Q6_K, but decode gains are modest (+9% vs Q4) as decode is memory-bandwidth bound. The quality loss compared to Q6 is minimal (-0.8 average), making NVFP4 a good choice for local inference.
2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps V3.0
A user shares kernel optimizations for Tesla P100 GPUs to improve performance when running the Qwen 3.8 model with llama.cpp, achieving significant speedups in inference.