Wow, this project lets your 4060 run large models??? Researchers from UC Berkeley, MIT, and UT Austin have just open-sourced FreeToken: an inference framework specifically designed for running ultra-large MoE models on personal computers. According to the official paper, several sets of data have been produced...
Summary
FreeToken is an open-source inference framework developed by researchers from UC Berkeley, MIT, and UT Austin, specifically designed for running ultra-large MoE models on personal computers, achieving efficient inference by dynamically allocating GPU, CPU, and memory resources.
View Cached Full Text
Cached at: 08/21/26, 11:16 PM
Damn, this project lets your 4060 run large models???
Researchers from UC Berkeley, MIT, and UT Austin have just open-sourced FreeToken: an inference framework specifically designed for running ultra-large MoE models on personal computers.
According to the official paper, they’ve already achieved some impressive results:
Qwen3.6 35B
→ 8GB RTX 4060 laptop
→ 39.3 tok/s
DeepSeek-V4-Flash 284B
→ RTX 5090 desktop
→ 22–25 tok/s
GLM-5.2 753B
→ RTX PRO 6000 workstation
→ ~15 tok/s
A gaming laptop with only 8GB VRAM can already run a 35B model at nearly 40 tok/s.
This speed even surpasses the median decoding speed of 33 tok/s reported for Codex online.
FreeToken’s approach leverages the GPU, CPU, system memory, and PCIe bus to work together.
Since MoE models only activate a few experts per token, there’s no need to load hundreds of billions of parameters entirely into VRAM.
FreeToken dynamically decides:
which layers stay in VRAM,
which are temporarily moved from memory to GPU,
and which are offloaded directly to the CPU for computation.
During prefill, it prefetches the next layer’s experts; during decode, it dynamically distributes work based on actual CPU, GPU, and PCIe speeds.
FreeToken also includes an Agent State Cache.
When an agent calls tools, modifies context, or continues tasks, it can skip much of the repetitive prefill work.
This makes FreeToken’s demo especially clear:
Writing code, reading emails, calling tools—even playing games.
You can easily run an open-source large model on your gaming PC and let agents like Claude Code or Codex handle tasks through it.
Previously, when running local models, people used to wonder: “Can my GPU even load this model?”
Now, with a 4060, we’re already thinking about how to host a 24/7 local agent.
Running cutting-edge models on consumer-grade GPUs is becoming increasingly practical.
This project is seriously awesome—can’t wait to try it out on the office computer tomorrow.
Shuo Yang (@Andy_ShuoYang):
Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization!Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s
DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22–25 tok/s
GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s
Similar Articles
@aehyok: Share an open-source project FreeToken, a local inference engine specifically for running ultra-large Mixture-of-Experts (MoE) models on consumer-grade computers. Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → R…
FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.
@AISuperDomain: Stop buying multi-GPU workstations to run large models! Open-source inference engine FreeToken integrates CPU, GPU, and memory: 8GB VRAM slim laptops run 35B MoE, home single-GPU gaming laptops handle 290B+! Completely solves the VRAM capacity issue, open-source and free: #AI #LLM …
Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.
@Phoenixyin13: This is the brand new best open-source model from the US. Thinking Machines Lab directly released a 975B parameter MoE multimodal giant, fully open-source under Apache-2.0! On standard benchmarks, it even outperforms NVIDIA's Nemotron strong model. Its core...
Thinking Machines Lab released a new 975B parameter MoE multimodal open-source model under Apache-2.0 license, surpassing NVIDIA's Nemotron on standard benchmarks, supporting text, image, and audio modalities with only 41B activated parameters, efficient and deployment-friendly.
@wlzh: Microsoft + UPenn Open-Source Multiplex Thinking: Let LLMs 'Clone' at Forks Then Merge. In a nutshell: When reasoning reaches a critical decision point, the model 'clones' into K pathfinders, each taking a different path. After one step, they merge back into a composite token and continue. With K=3, one token carries the information of three…
Microsoft and the University of Pennsylvania open-source Multiplex Thinking, which allows LLMs to split into K parallel paths during inference, explore, then merge, improving efficiency. On a 7B model, it achieves over 50% accuracy on AMC2023 (first 7B model to do so) and over 55% on AIME2025.
@NFTCPS: Running large models locally – looking at the hundreds of GB of weights and VRAM requirements, most people are discouraged immediately, and I was too. Colibri changes the game by treating VRAM, RAM, and disk as a unified hierarchy, streaming weights from disk on demand. Written in pure C with zero dependencies, it already has over 25,000 stars. Here are a few points: …
Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.