Need maybe say "Use llama.cpp"
Summary
A user reports that a newly hyped inference engine degrades into a repetitive 'doom loop' when running an IQ3_S quantized model, while the same model runs coherently under llama.cpp — suggesting the new engine may have issues with small quantized open-weights models.
Similar Articles
Multi-hour llama.cpp optimization experiments on Qwen MoE models, patches, benchmarks, and reproduction guides
This article presents a multi-hour investigation into speeding up local inference for Qwen MoE models using llama.cpp, including experimental results, patches, benchmarks, and reproduction guides, with some workloads showing substantial gains.
@akshay_pachaar: This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a…
Benchmark thread showing Uzu, an open-source inference engine from Mirai, running Qwen3.5-9B at 92.1 tokens/s on a 16GB M5 MacBook Pro — about 4x faster than llama.cpp (22 tok/s) and MLX (25.1 tok/s) — by using tree-based speculative decoding with a Weaver drafter to break the memory-bandwidth ceiling of ordinary local decoding.
Qwen3.6 27B more dumb in vLLM compared to llama.cpp
A user reports that the Qwen3.6-27B model performs better and more reliably with llama.cpp than with vLLM, citing tool call errors and 'lobotomized' behavior in vLLM despite extensive configuration.
We really need llama.cpp to support changing thinking amount
The user advocates for llama.cpp to support adjustable thinking modes for AI models like Qwen 3.8 27b to enhance task-specific performance.
Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000
A community member benchmarks Ninfer6000 (a Ninfer fork tuned for RTX Pro 6000 Blackwell and Qwen3.8-Flash-Next) against vLLM and llama.cpp, finding ~1.6x decode speedup over vLLM and 2.2x over llama.cpp with near-identical accuracy, but reports crippling real-world tool-calling malformations that keep them on vLLM.