The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).

Reddit r/LocalLLaMA Papers
cpu-inference ternary-weights mixture-of-experts active-parameters model-efficiency research

Summary

Presents the insight that CPU decode speed depends on active parameters per token, not total parameters, and proposes a 10B-parameter model using ternary weights and granular MoE to achieve high token rates on mid-level PCs. Reports sandbox measurements showing a speedup from 176 to 848 tok/s on an 8.3M model with minimal quality loss.

On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters per token" (ternary weights and a granular MoE), the total capacity can grow without effecting the speed. Now the new catch: if speed is not the problem (a 105M and a 206M model run at the same tok/s (predicted 739–1309 tok/s)), will the model capacity scale with the number of params or the model will go "dumber" because of the lack of routing capacity with more experts? My measurements: On a Ryzen r5 3600X (single thread), the engine went from 176 tok/s to 848 tok/s on a 8.3M sandbox model with ternary LUT MLP, activation skip, deterministic SSM scan, two pool MoE with only a +0.00004 BPB quality cost. (Here the model is cache resident). I launched the full training on a 30M (11M active) model on the 2x T4 on Kaggle. Before the run I pushed 5 gates. Four passed and one failure: distilling from a bigger teacher using a different tokenizer lost against simply cross entropy (-0.0116 BPB, about 2.3 sigma worse), so I flipped the recipe to CE-primary. This is my idea of a 100% transparent project. (nothing above the 8.3M sandbox is trained yet, the 10B is the target) Repo in the comments
Original Article

Similar Articles

2X tk/s (from 19.4 -> 38.1 tk/s on 1 x MI50) Playing with a hypothesis like speculative decoding.. but instead of an additional side model, exploiting that I can run multiple computations side-by-side AS IF I had Qwen3.6-27B loaded twice in memory - small quants don't use all the available compute.

Reddit r/LocalLLaMA

Packed Twin Inference (PTI) is a technique that achieves ~2× LLM throughput by running multiple token sequences in a single batch decode, exploiting weight sharing in llama.cpp without needing a draft model or additional VRAM.

Would extremely high decode tok/s even be useful?

Reddit r/LocalLLaMA

A discussion questioning whether extremely high decode speeds (1k-10k tok/s) for large models like Qwen 3.5 397B or GLM-5.2 would unlock new use cases or be better spent on loading larger models.