Would extremely high decode tok/s even be useful?

Reddit r/LocalLLaMA News
inference-speed decode-tokens large-language-models performance discussion

Summary

A discussion questioning whether extremely high decode speeds (1k-10k tok/s) for large models like Qwen 3.5 397B or GLM-5.2 would unlock new use cases or be better spent on loading larger models.

If you were able to get an inference machine that could do decode at 1k toks/s or even 10k tok/s, would that even be helpful? Would it unlock any new use cases? Let’s assume that this is for actually useful models and fairly large models like Qwen 3.5 397B, GLM-5.2, etc Or at that speed would it just make better send to load much larger models? In which case, question still applies. E.g. Kimi K3 at the high speeds
Original Article

Similar Articles

2X tk/s (from 19.4 -> 38.1 tk/s on 1 x MI50) Playing with a hypothesis like speculative decoding.. but instead of an additional side model, exploiting that I can run multiple computations side-by-side AS IF I had Qwen3.6-27B loaded twice in memory - small quants don't use all the available compute.

Reddit r/LocalLLaMA

Packed Twin Inference (PTI) is a technique that achieves ~2× LLM throughput by running multiple token sequences in a single batch decode, exploiting weight sharing in llama.cpp without needing a draft model or additional VRAM.

Optimizing Models to Be Fast at Codegen (8 minute read)

TLDR AI

Morph LLC describes three key techniques—training a speculator on coding output, auto-searching kernels on cheap GPUs, and writing a custom interconnect—to dramatically speed up open models like Qwen and DeepSeek for coding agent workloads, achieving up to 3x speculative decoding speedup and 97-162 tok/s on a $7K GPU.