How fast is 10 tokens per second really?
Summary
Simon Willison explores the practical meaning of 10 tokens per second speed for large language models, offering context on how fast that feels and its implications for usability.
View Cached Full Text
Cached at: 05/20/26, 06:38 PM
Similar Articles
How fast is N tokens per second really?
A web tool that lets users visually experience different LLM token generation rates (e.g., 5–800 tok/s) across code, text, reasoning, and agent modes, helping internalize performance numbers from benchmarks.
Getting a feel for how fast X tokens/second really is.
The author introduces a web-based script designed to help users intuitively understand token-per-second speeds in local LLM setups by simulating text, code, and reasoning generation rates.
The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).
Presents the insight that CPU decode speed depends on active parameters per token, not total parameters, and proposes a 10B-parameter model using ternary weights and granular MoE to achieve high token rates on mid-level PCs. Reports sandbox measurements showing a speedup from 176 to 848 tok/s on an 8.3M model with minimal quality loss.
Token maxxing
Discusses strategies and techniques for maximizing token usage in large language models to improve efficiency and output quality.
GigaToken: ~1000x faster Language model tokenization
GigaToken is an ultra-fast tokenizer library that claims up to 1000x speedup over HuggingFace tokenizers, supporting most common LLM tokenizers and providing drop-in compatibility.