Freetokens project is impressive
Summary
A new open-source project called FreeToken has been released, featuring a research paper and GitHub repository, with tests showing efficient token processing speeds of around 100 tokens per second on consumer hardware.
Similar Articles
@Yuchenj_UW: UC Berkeley open-sourced FreeToken. Wild results: A single RTX PRO 6000 runs the 753B GLM-5.2 at 14.9 tok/s! An 8GB RTX…
UC Berkeley has open-sourced FreeToken, a tool that significantly speeds up local AI inference on consumer GPUs, achieving up to 39.3 tok/s on an 8GB RTX 4060 laptop.
Gigatoken: A new open source tokenizer ~100x faster than Tiktoken, -500-1000x faster than Huggingface
Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.
FlashML-org/FreeToken
FreeToken is an edge-native Mixture-of-Experts serving engine that runs frontier-scale 290B+ open-weight MoE models locally on consumer hardware, featuring bandwidth-adaptive CPU-GPU co-execution, semantic-aware caching, and elastic VRAM management with Anthropic/OpenAI-compatible APIs.
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
FreeToken is an edge-native serving system that dynamically maps computation and model state onto heterogeneous local hardware to run large open-weight models on personal machines, enabling efficient execution of models up to 753B on a single GPU.
FreeToken: Efficient Edge-Native MoE Serving (24 minute read)
FreeToken is an edge-native Mixture of Experts serving system that efficiently runs large open-weight models on consumer hardware by adaptively managing resources across heterogeneous edge devices.