Gufo performance .... 70tps Qwen 3.8 27b but you need to read the fine print.

Reddit r/LocalLLaMA Tools

Summary

作者对开源 LLM 推理服务 Gufo 0.4.0 进行实测复现:其宣传的 70 tok/s 确实存在,但主要来自重复词 prompt 下的投机解码命中率,普通 prompt 上约 39 tok/s;与 halogen 相比,日常生成速度慢约 13-18%,但长 prompt 处理更快。

So everyone has been yelling about how I should be using Gufo instead of halogen because it's open source and it's "just as good or better". Checking in on their GitHub (GitHub.com/gufo-org/gufo) got me immediately .. "Qwen 27B Q4: 70.56 tok/s single user, 123 tok/s with 8 users" on a strix halo device? Yes please ... So I broke down and tried it today ... Setup: gufo 0.4.0 from their podman image, Qwen3.8 27B UD-Q4_K_XL from Unsloth plus the DFlash2 Q4_K_M draft model, using their own benchmark script and their own settings (greedy, thinking off, 128 output tokens, prompt cache off). If you want the short version ... yeah I got 70.22 tok/s. So the number is real. But the prompt that produces it is "Write the word red exactly 1000 times". But it's also not real. In that figure all the speed comes from the speculative decoding. The draft model guesses like 7 tokens ahead, the 27b checks them in one pass and keeps what it agrees with. When the output is the same word over and over the draft is right every time. On a real prompt it's right maybe half the time. Their benchmark has a second set of nine ordinary prompts (some C++, a word problem, a summary, Italian, Chinese, JSON, a bit of fiction, a debugging checklist). On those: | | repeat-a-word prompt | normal prompts | |---|---|---| | 1 user | 70.2 tok/s | 39.4 tok/s median, anywhere from 22 to 52 depending on the prompt | | 8 users, their "aggregated" number | 122.6 | 82.4 | | 8 users, tokens actually delivered per second | 82 | 52 | About that last row. The "123 tok/s aggregated" figure is each request's decode speed added together, with prompt processing and queue time left out. If you just count tokens coming out of the box per second of wall clock it's 82, or 52 on normal. prompts. To be fair to the gufo people, none of this is hidden. Their benchmark docs have separate "mixed" and "repetitive" columns and the mixed numbers they publish match what I got. It's only the repo description and the top of the README that lead with the best case. And 39 tok/s from a 27B at Q4 on an APU is still really good. Without the draft model their docs put it around 12. The other thing I wanted to know was how it compares to halogen (peonist-ai/halogen-flash-server), which is what I normally run. Both can serve Qwen3.8 Flash-Next, so I put that on both and sent the same prompts to each. Two boxes, same hardware, same OS image. Greedy, thinking off, 256 tokens. | | halogen 0.13.8 | gufo 0.4.0 | |---|---|---| | nine normal prompts, average decode | 43.9 tok/s | 38.2 tok/s | | 4 users at once, end to end | 76.6 tok/s | 63.1 tok/s | | cold prompt processing, ~9.7k tokens | 1288 tok/s | 1495 tok/s | | the "red" prompt | 56.9 tok/s | 87.4 tok/s | So for everyday generation halogen was about 13% faster for one user and about 18% faster with four. gufo was 16% faster at chewing through a long prompt and a lot faster on the repetitive one. I'll add this just in case, because someone will ask or at least try to poke about it in the comments - I know the weights aren't the same. halogen uses its own 4-bit format, gufo uses the Unsloth GGUF. I only measured speed. I did not compare output quality at all. - They were two different machines but identical hardware and software, and my boxes have agreed within 1% on other benchmarks, but it's still two machines. - One run each was done for the head to head. The reproduction of their numbers was 3 reps. - gufo has shipped four releases over the last four day, so this could all be stale by next week. There was quite a lot of stuff that I liked about gufo that isn't performance related. It takes plain GGUFs, it's MIT, the 27B loads in about 3 seconds (Flash-Next in 13), the per-request log line tells you draft acceptance and cache hits, and it does 8 batched sessions. It also has ASR, TTS and image models that I haven't touched. Their benchmark hashes the output with and without the draft model and it was identical every time, so the speculative path isn't changing what the model says. One thing to keep in mind if you try it out is that it reserves memory per session up front. Flash-Next with 4 sessions at 64k context took 94GB. So it isn't smoke and mirrors exactly. Everything I checked reproduced. Just know that the 70 is a ceiling you'll only hit if your workload is incredibly predictable text, and plan around the 30s for the 27B on normal stuff. I kept this all setup to tinker with on actual output quality over the next few days, I'm happy to run other prompts or try it with different settings if anyone wants to see something specific.
Original Article

Similar Articles

@nicekate8888: For the past twenty days, I've been obsessing over one thing — how to make Qwen3.6-27B run fast and well on my Mac. I started with Unsloth Q5, got 18 tok/s, and the fan was roaring. Then I switched to MLX 6bit + DFlash, hitting 22 tok/s, still not fast enough. Eventually I found MTPLX 4bit: 43 tok/s with good quality.

X AI KOLs Timeline

The user shares their experience optimizing Qwen3.6-27B inference speed on a Mac using different quantization methods (Unsloth Q5, MLX 6bit + DFlash, MTPLX 4bit), ultimately reaching 43 tok/s.