@no_stp_on_snek: while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)... went and teste…

X AI KOLs Timeline Tools
performance benchmark inference quantization vllm llama-cpp unsloth-ai nvfp4 qwen

Summary

A detailed comparison of Unsloth's NVFP4 quantized model inference performance between vLLM and llama.cpp, highlighting prefill speed advantages for vLLM but decode and caching advantages for llama.cpp in single-stream agent workloads.

while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)... went and tested @UnslothAI 's new NVFP4 model to test their claims. unsloth's NVFP4 checkpoint on vLLM is about 2x faster than my llama.cpp GGUF at prefill. i'm keeping the GGUF anyway, and the reason turned out to be nothing like what i expected. qwen3.6-27b NVFP4, single 5090 (32GB), WSL2. fair warning: my llama.cpp side is the Blackwell-native NVFP4 kernel branch, not stock, so a stock build won't reproduce these numbers. prefill: vLLM ~6,600 tok/s vs llama.cpp ~2,800-3,400. call it ~2x for vLLM (my llama.cpp figure is server-side timing, vLLM is wall-clock, so i'm not going to defend a precise ratio). decode: llama.cpp 109 tok/s vs vLLM 97.9 with MTP spec-decode, 63.8 without. llama.cpp wins. weights: 16.6GB vs 20.5GB. cold start: 11 seconds vs minutes. first, a correction on myself, because i nearly posted the wrong conclusion. i believed vLLM capped me at 32K context. it did not. that cap was MINE, set conservatively during an OOM fight and never re-probed (thanks claude). the KV pool actually held ~94k tokens. my sessions would have fit fine. the eval is always where you fool yourself, and i fooled myself. the real reason is less obvious and more interesting: vLLM auto-disables prefix caching for hybrid mamba/DeltaNet architectures. so every single agent turn re-prefills the entire conversation from scratch, roughly 5.5s at 36k tokens. llama.cpp checkpoints the recurrent state and hits 97-100% cache on my real traffic. that's the whole ballgame. my workload is one agent taking sequential turns, re-sending a growing conversation. vLLM's 2x prefill advantage gets spent redoing work that llama.cpp simply never does, while llama.cpp's decode edge applies to every token generated. and yes, single request. that's the point. vLLM is built for concurrent serving and single-stream is its worst case. but single-stream IS my workload, which is the entire thesis here. the serving saga, if you're attempting this: 7 attempts. four OOM-killed at an identical ~52GB + 20GB swap peak, invariant to compile-parallelism caps, multimodal off, tiny batch, 8k context, even full eager. model load was never the problem (20.5GB VRAM in 14s every time). the spike is post-load, in profiling and graph capture, and looks specific to the hybrid gated-DeltaNet arch in vLLM 0.24. it wants 70-100GB of HOST ram. fixed by raising WSL to 58GB + 48GB swap. this is a WSL2 memory-ceiling problem, native linux may never hit it. to be fair to unsloth: it genuinely wins cold long-context one-shots, roughly 2x faster time to first token. and their "2.5x" is measured against other vLLM NVFP4 quants, so that claim can be completely true and it can still lose decode to llama.cpp + MTP. i checked both. not contradictory. tldr: benchmark your own workload. the headline number is almost never the number that matters for you.
Original Article
View Cached Full Text

Cached at: 07/11/26, 05:27 PM

while everyone is talking about @SpaceXAI , @AnthropicAI , and @OpenAI updates (but where @GoogleAI?)…

went and tested @UnslothAI ’s new NVFP4 model to test their claims.

unsloth’s NVFP4 checkpoint on vLLM is about 2x faster than my llama.cpp GGUF at prefill. i’m keeping the GGUF anyway, and the reason turned out to be nothing like what i expected.

qwen3.6-27b NVFP4, single 5090 (32GB), WSL2. fair warning: my llama.cpp side is the Blackwell-native NVFP4 kernel branch, not stock, so a stock build won’t reproduce these numbers.

prefill: vLLM ~6,600 tok/s vs llama.cpp ~2,800-3,400. call it ~2x for vLLM (my llama.cpp figure is server-side timing, vLLM is wall-clock, so i’m not going to defend a precise ratio). decode: llama.cpp 109 tok/s vs vLLM 97.9 with MTP spec-decode, 63.8 without. llama.cpp wins. weights: 16.6GB vs 20.5GB. cold start: 11 seconds vs minutes.

first, a correction on myself, because i nearly posted the wrong conclusion. i believed vLLM capped me at 32K context. it did not. that cap was MINE, set conservatively during an OOM fight and never re-probed (thanks claude). the KV pool actually held ~94k tokens. my sessions would have fit fine. the eval is always where you fool yourself, and i fooled myself.

the real reason is less obvious and more interesting: vLLM auto-disables prefix caching for hybrid mamba/DeltaNet architectures. so every single agent turn re-prefills the entire conversation from scratch, roughly 5.5s at 36k tokens. llama.cpp checkpoints the recurrent state and hits 97-100% cache on my real traffic.

that’s the whole ballgame. my workload is one agent taking sequential turns, re-sending a growing conversation. vLLM’s 2x prefill advantage gets spent redoing work that llama.cpp simply never does, while llama.cpp’s decode edge applies to every token generated.

and yes, single request. that’s the point. vLLM is built for concurrent serving and single-stream is its worst case. but single-stream IS my workload, which is the entire thesis here.

the serving saga, if you’re attempting this: 7 attempts. four OOM-killed at an identical ~52GB + 20GB swap peak, invariant to compile-parallelism caps, multimodal off, tiny batch, 8k context, even full eager. model load was never the problem (20.5GB VRAM in 14s every time). the spike is post-load, in profiling and graph capture, and looks specific to the hybrid gated-DeltaNet arch in vLLM 0.24. it wants 70-100GB of HOST ram. fixed by raising WSL to 58GB + 48GB swap. this is a WSL2 memory-ceiling problem, native linux may never hit it.

to be fair to unsloth: it genuinely wins cold long-context one-shots, roughly 2x faster time to first token. and their “2.5x” is measured against other vLLM NVFP4 quants, so that claim can be completely true and it can still lose decode to llama.cpp + MTP. i checked both. not contradictory.

tldr: benchmark your own workload. the headline number is almost never the number that matters for you.

Similar Articles

ExLlamaV3 is underrated

Reddit r/LocalLLaMA

The author advocates for ExLlamaV3, an underrated local AI inference tool that outperforms llama.cpp on NVIDIA hardware, with recent updates like CPU MoE offload enhancing its performance.

@_avichawla: https://x.com/_avichawla/status/2077653695123378321

X AI KOLs Timeline

This article argues that vLLM and similar serving frameworks are inefficient for running multiple small AI models on a single GPU due to design limitations. It introduces the SIE open-source inference engine as a solution for serving many models together to reduce costs.

Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000

Reddit r/LocalLLaMA

A community member benchmarks Ninfer6000 (a Ninfer fork tuned for RTX Pro 6000 Blackwell and Qwen3.8-Flash-Next) against vLLM and llama.cpp, finding ~1.6x decode speedup over vLLM and 2.2x over llama.cpp with near-identical accuracy, but reports crippling real-world tool-calling malformations that keep them on vLLM.