@aehyok: Share an open-source project FreeToken, a local inference engine specifically for running ultra-large Mixture-of-Experts (MoE) models on consumer-grade computers. Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s DeepSeek-V4-Flash 284B → R…
Summary
FreeToken is an open-source local inference engine designed to run large Mixture-of-Experts (MoE) models on consumer-grade computers, offering significantly faster performance than alternatives like Ollama with easy installation and native GUI.
View Cached Full Text
Cached at: 08/24/26, 03:44 AM
| Download | Paper | Developer Slack | Community Discord | Community WeChat |
Similar Articles
Wow, this project lets your 4060 run large models??? Researchers from UC Berkeley, MIT, and UT Austin have just open-sourced FreeToken: an inference framework specifically designed for running ultra-large MoE models on personal computers. According to the official paper, several sets of data have been produced...
FreeToken is an open-source inference framework developed by researchers from UC Berkeley, MIT, and UT Austin, specifically designed for running ultra-large MoE models on personal computers, achieving efficient inference by dynamically allocating GPU, CPU, and memory resources.
@AISuperDomain: Stop buying multi-GPU workstations to run large models! Open-source inference engine FreeToken integrates CPU, GPU, and memory: 8GB VRAM slim laptops run 35B MoE, home single-GPU gaming laptops handle 290B+! Completely solves the VRAM capacity issue, open-source and free: #AI #LLM …
Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
FreeToken is an edge-native serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines, enabling efficient execution of models up to 753B on a single GPU.
poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding
作者将 llama.cpp 的 server 分支为 AgrillaMoE，针对 Qwen3.6-35B-A3B MoE 模型在 16GB 显存 GPU 上优化，配合 2-bit 量化可达 ~57-60 tok/s，并支持 OpenAI/Anthropic API 兼容接口；其 MoE 扩展补丁在运行时增加每 token 激活专家数（20 vs 8），无需重训即可在 GPQA-Diamond 上提升约 2.5 分。
FreeToken: Efficient Edge-Native MoE Serving (24 minute read)
FreeToken is an edge-native Mixture of Experts serving system that efficiently runs large open-weight models on consumer hardware by adaptively managing resources across heterogeneous edge devices.