NInfer-2080ti-22g now supports Windows-native and is only 0.28 GB

Reddit r/LocalLLaMA Tools
inference-engine local-llm qwen cuda open-source consumer-gpu

Summary

NInfer v0.3.0, an inference engine for running Qwen3.6/8-27B on a modded 22 GB RTX 2080 Ti, adds native Windows support and prebuilt releases, cutting binaries to 0.28 GB while delivering up to 47.5 tok/s decode with MTP speculative decoding.

v0.3.0 of NInfer for Turing is out, and this is all about streamlining the UX. For context, this is an engine for running Qwen3.6/8-27B (and ONLY Qwen3.6/8-27B) on a single modded 22 GB RTX 2080 Ti. It is designed to ONLY use VRAM, no system RAM offloading, hence it does not support 35B-A3B. Fine-tunes and abliterations etc are supported as well, just use the .ninfer checkpoint converter. What's new - Native Windows. Previously this required Linux, WSL2, or Docker. Now Windows users don't need to worry about partitioning/releasing resources (WSL or Docker instances require system RAM allocation even if your use case inside them doesn't). - Prebuilt releases. You can still compile it yourself, requiring CUDA, Visual Studio and FFmpeg and takes a while, but now you can also just download a pre-compiled zip and run a single script. - Streamlined. Removing "useless" (for us) code paths for RTX 5090, FP8, FP4, and 35B-A3B (fits fully in 32GB VRAM but not in 22GB) cut the built binaries from 1.56GB to 0.28 GB. Builds now take minutes, not hours. Outputs remain bit-identical. What you get - OpenAI-compatible and Anthropic-compatible APIs, plus a command-line tool. - Image and video input, thinking mode, and CUDA Graph decode. - 64k context by default, using an INT8 KV cache. The weights take 15.4 GiB, so you need the 22 GB card. - MTP speculative decoding, on by default. Numbers (one 2080 Ti 22 GB, Windows 11, single request) Test Result prefill, 512-token prompt 616 tok/s prefill, 64k-token prompt 458 tok/s decode, no MTP 27.0 tok/s decode, MTP, real code prompt 47.5 tok/s (26.3 without MTP) decode, MTP, real prose prompt 28.8 tok/s (26.0 without MTP) (Apache-2.0, derived from Neroued/ninfer)
Original Article

Similar Articles

I built Ninfer 4080 for 16GB class GPUs

Reddit r/LocalLLaMA

A community developer released Ninfer 4080, an optimized local inference runtime that runs the Qwen 3.8 27B GSQ model at 100k context on a 16GB RTX 4080, achieving up to ~2720 tok/s prefill and 262 tok/s decode via DFlash2 speculative decoding — significantly faster than general-purpose engines like llama.cpp or vLLM.

Ninfer Stahp

Reddit r/LocalLLaMA

A user benchmarks the new Ninfer inference backend on a Blackwell MaxQ GPU, reporting 130-180 t/s on qwen3.8-27b-nvfp4 with minimal slowdown under parallel decoding — claiming vLLM-level concurrency and far faster throughput than llama-server, even after installing it on Windows 10.