@TheAhmadOsman: Why do I focus on Inference Engines/Software Stacks for your hardware? - 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving t…
Summary
Comparison of inference engine performance on different hardware: moving from baseline to vLLM with TP=2 on 2x RTX 3090s improves from ~14.5 tok/s to ~64 tok/s, and on RTX PRO 6000 moving to Sglang improves from ~32 tok/s to ~110 tok/s. Recommends vLLM/Sglang for CUDA/multi-GPU and llama.cpp for edge devices.
View Cached Full Text
Cached at: 06/22/26, 05:31 AM
Why do I focus on Inference Engines/Software Stacks for your hardware?
-
2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving to vLLM w/ TP=2
-
RTX PRO 6000: ~32 tok/s → ~110 tok/s moving to Sglang
So:
-
CUDA/2+ GPUs: ExLlamaV3/vLLM/Sglang > llama.cpp
-
Edge: llama.cpp > Ollama https://t.co/5WXSlPrrOB
Similar Articles
Comparing dual-GPU inference speed between llama.cpp row/tensor split and ik_llama graph split
A user benchmarks dual-GPU inference speed on two RTX 3080 20GB using llama.cpp (row/tensor split) and ik_llama (graph split) with a Qwen3.6-27B GGUF model, comparing token generation and prompt processing speeds.
The Rise of Overfit Inference Engines
A blog post argues that hyper-specialized, hardware- and model-specific inference runtimes (Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo) are now outperforming general-purpose engines like llama.cpp, vLLM, and Ollama by 2.5-4x on fixed consumer hardware, with Strata reaching 53-90 tok/s decoding Qwen3.8-Flash-Next on a single 12GB RTX 4070.
If you have a 3090, or other 30xx for local LLMs, I have something for you
The article presents a custom fork of llama.cpp optimized for NVIDIA Ampere architecture GPUs like the 3090, achieving over 90 tokens per second for 27B parameter models with large context windows, making local inference faster than API speeds.
GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode)
Speed test results for GLM-5.2 running on llama.cpp with RTX 5090 and RTX 3090 Ti, showing prefill speeds up to 579 t/s at 8k context and decode at ~10.6 t/s.
@lxfater: Someone built their own inference engine using an RTX 3090, running Qwen 27B at a blazing 273 tokens per second! This d…
A developer (coffeecup2020) built a custom inference engine called tqllm that runs Qwen3-27B (TQ3_4S-v2 quantization) on an RTX 3090 at an impressive 273 tokens per second — the post contextualizes that speed against human reading rates and notes a broader recent wave of lightweight LLM inference releases, including Strata running quantized Qwen3.8-Flash-Next on 12GB VRAM.