Ninfer and a 5090 with 3.8 27B is making me cry tears of joy it's so good.
Summary
A user reports achieving high token throughput with the Ninfer tool on an NVIDIA RTX 5090 GPU using a Qwen 3.8B model, significantly outperforming llama.cpp.
Similar Articles
Nifer is insane. 700t/s with Qwen 3.6 35B (no thinking). Purpose build for RTX5090. Full 250k context too.
Nifer is a tool that achieves 700 tokens per second inference on Qwen 3.6 35B without thinking, specifically optimized for the RTX 5090, with support for full 250k context.
Ninfer Stahp
A user benchmarks the new Ninfer inference backend on a Blackwell MaxQ GPU, reporting 130-180 t/s on qwen3.8-27b-nvfp4 with minimal slowdown under parallel decoding — claiming vLLM-level concurrency and far faster throughput than llama-server, even after installing it on Windows 10.
NInfer day0 support for Qwen3.8 27b: ~200 tok/s generation, with tons of engine improvments
NInfer adds day-0 support for the Qwen3.8-27B model, achieving around 200 tokens per second on a single RTX 5090 with speculative decoding, and includes engine improvements like concurrent requests and kernel optimizations.
I built Ninfer 4080 for 16GB class GPUs
A community developer released Ninfer 4080, an optimized local inference runtime that runs the Qwen 3.8 27B GSQ model at 100k context on a 16GB RTX 4080, achieving up to ~2720 tok/s prefill and 262 tok/s decode via DFlash2 speculative decoding — significantly faster than general-purpose engines like llama.cpp or vLLM.
(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s
A developer forked NInfer, a C++20/CUDA inference engine, to add tensor-parallelism and YaRN rope scaling, enabling Qwen3.8-27B to run with a 1M token context on dual 5090 GPUs and outperforming vLLM in specific decode scenarios.