@akshay_pachaar: This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a…

This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. → llama.cpp reached 22.0 tokens/s → MLX reached 25.1 tokens/s → Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. → It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. → It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: https://github.com/trymirai/uzu (don't forget to star 🌟) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.