FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Summary
FreeToken is an edge-native serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines, enabling efficient execution of models up to 753B on a single GPU.
View Cached Full Text
Cached at: 08/19/26, 03:55 AM
Paper page - FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Source: https://huggingface.co/papers/2608.16157 Authors:
,
,
,
,
,
,
,
,
,
Abstract
FreeToken is an edge-native Mixture-of-Experts serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines.
Frontieropen-weight modelsare increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-nativeMoE servingsystem that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading,expert residency, CPU--GPU execution,agentic state reuse, andruntime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixedoffloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2608\.16157
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.16157 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.16157 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.16157 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
FreeToken: Efficient Edge-Native MoE Serving (24 minute read)
FreeToken is an edge-native Mixture of Experts serving system that efficiently runs large open-weight models on consumer hardware by adaptively managing resources across heterogeneous edge devices.
tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s)
Explores predicting which MoE experts will be used for the next token to improve CPU/GPU offloading, achieving 30->150-200 tg/s speedup, and questions the feasibility of implementation.
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.
I thought I’d done something extraordinary by running massive models on standard smartphones but
Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.
LFM2.5 230M running in-browser at 1,400 tok/s using custom WebGPU kernels
LFM2.5 230M model achieves 1,400 tokens per second in-browser using custom WebGPU kernels, demonstrating efficient local inference.