I designed a CPU-native LLM architecture that hits 100+ tok/s on a 10B parameter model (the quality is the problem)

Reddit r/LocalLLaMA Models
cpu-native llm architecture high-speed inference model-adaptation open-source

Summary

The author designed a CPU-native LLM architecture that achieves over 100 tokens per second on a 10B parameter model, but highlights quality issues; they share a GitHub repo with code and research on adapting existing models.

I've been working on an architecture designed to work with CPU memory constraints, not against it. The 10B parameters runs at 113-130 tok/s on a Ryzen 5 3600X, no GPU but the weights are garbage. To verify whether the quality would also hold at scale, I wanted to train an LM. While the training was running (8+ weeks for a 206M on T4, no budget) I had the idea to try to convert an existing LLM to the format that my engin can run. Turns out adapting a transformer donor (Qwen2.5-Coder) to an SSM/ternary/sparse format is, predictably, a pain in the ass. If you want to take a look at the repo (and the research/donor-adaptation branch): https://github.com/WildPino/SiliconLLM
Original Article

Similar Articles

The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).

Reddit r/LocalLLaMA

Presents the insight that CPU decode speed depends on active parameters per token, not total parameters, and proposes a 10B-parameter model using ternary weights and granular MoE to achieve high token rates on mid-level PCs. Reports sandbox measurements showing a speedup from 176 to 848 tok/s on an 8.3M model with minimal quality loss.