@Yuchenj_UW: UC Berkeley open-sourced FreeToken. Wild results: A single RTX PRO 6000 runs the 753B GLM-5.2 at 14.9 tok/s! An 8GB RTX…
Summary
UC Berkeley has open-sourced FreeToken, a tool that significantly speeds up local AI inference on consumer GPUs, achieving up to 39.3 tok/s on an 8GB RTX 4060 laptop.
View Cached Full Text
Cached at: 08/24/26, 05:51 AM
UC Berkeley open-sourced FreeToken. Wild results:
A single RTX PRO 6000 runs the 753B GLM-5.2 at 14.9 tok/s!
An 8GB RTX 4060 laptop (~$1,000) runs Qwen3.6-35B at 39.3 tok/s!
FreeToken is 2–4x faster than Ollama across consumer GPUs. Local AI inference is getting very real. Great work by @Andy_ShuoYang and UC Berkeley Sky Lab!
Shuo Yang (@Andy_ShuoYang): Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization!
Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s
DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s
GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s
Similar Articles
@aehyok: Share an open-source project FreeToken, a local inference engine specifically for running ultra-large Mixture-of-Experts (MoE) models on consumer-grade computers. Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → R…
FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.
4090 + 5060 Ti + 64GB RAM: 206 t/s on a 35B-A3B, and a 122B at 37 t/s
A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.
@RiskRich: YOUR 8GB GPU ISN'T THE DEAD END IT USED TO BE FreeToken's paper reports Qwen3.6 35B running on an RTX 4060 laptop at 39…
FreeToken enables running large AI models like Qwen3.6 35B on limited hardware such as an 8GB GPU by offloading to CPU and RAM, providing a cost-effective solution for edge computing.
@rohanpaul_ai: I had to test it myself to believe this unreal inference speed. 3,000 tokens/s for 1 user on standard datacenter GPUs. …
Kog AI achieves 3,000 tokens/s inference speed on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200, leveraging a hidden efficiency gap in GPU token generation.
Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP
The article details an experiment achieving 50 tokens per second inference with Qwen3.8-27B at 256K context on a 24GB GPU using Multi-Token Prediction and custom optimizations.