@TheAhmadOsman: Why do I focus on Inference Engines/Software Stacks for your hardware? - 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving t…

X AI KOLs Following News

Summary

Comparison of inference engine performance on different hardware: moving from baseline to vLLM with TP=2 on 2x RTX 3090s improves from ~14.5 tok/s to ~64 tok/s, and on RTX PRO 6000 moving to Sglang improves from ~32 tok/s to ~110 tok/s. Recommends vLLM/Sglang for CUDA/multi-GPU and llama.cpp for edge devices.

Why do I focus on Inference Engines/Software Stacks for your hardware? - 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving to vLLM w/ TP=2 - RTX PRO 6000: ~32 tok/s → ~110 tok/s moving to Sglang So: - CUDA/2+ GPUs: ExLlamaV3/vLLM/Sglang > llama.cpp - Edge: llama.cpp > Ollama https://t.co/5WXSlPrrOB
Original Article
View Cached Full Text

Cached at: 06/22/26, 05:31 AM

Why do I focus on Inference Engines/Software Stacks for your hardware?

  • 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving to vLLM w/ TP=2

  • RTX PRO 6000: ~32 tok/s → ~110 tok/s moving to Sglang

So:

  • CUDA/2+ GPUs: ExLlamaV3/vLLM/Sglang > llama.cpp

  • Edge: llama.cpp > Ollama https://t.co/5WXSlPrrOB

Similar Articles

The Rise of Overfit Inference Engines

Reddit r/LocalLLaMA

A blog post argues that hyper-specialized, hardware- and model-specific inference runtimes (Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo) are now outperforming general-purpose engines like llama.cpp, vLLM, and Ollama by 2.5-4x on fixed consumer hardware, with Strata reaching 53-90 tok/s decoding Qwen3.8-Flash-Next on a single 12GB RTX 4070.

@lxfater: Someone built their own inference engine using an RTX 3090, running Qwen 27B at a blazing 273 tokens per second! This d…

X AI KOLs Timeline

A developer (coffeecup2020) built a custom inference engine called tqllm that runs Qwen3-27B (TQ3_4S-v2 quantization) on an RTX 3090 at an impressive 273 tokens per second — the post contextualizes that speed against human reading rates and notes a broader recent wave of lightweight LLM inference releases, including Strata running quantized Qwen3.8-Flash-Next on 12GB VRAM.