@percyliang: ! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the …
Summary
Marcel Rød announces Gigatoken, a tokenizer implementation that is 500-1000x faster than HuggingFace and 100x faster than OpenAI's tiktoken, built in Rust.
View Cached Full Text
Cached at: 07/22/26, 06:19 AM
⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.
Marcel Rød (@marcelroed): Introducing the world’s fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI’s tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
Similar Articles
Gigatoken: A new open source tokenizer ~100x faster than Tiktoken, -500-1000x faster than Huggingface
Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.
Gigatoken (GitHub Repo)
Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.
GigaToken: ~1000x faster Language model tokenization
GigaToken is an ultra-fast tokenizer library that claims up to 1000x speedup over HuggingFace tokenizers, supporting most common LLM tokenizers and providing drop-in compatibility.
quicktok: a faster tokenizer (exact and byte-identical with tiktoken) [P]
quicktok is a fast and exact BPE tokenizer in C++ that is byte-identical with tiktoken, achieving 2–11x speedup over existing alternatives. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 encoders.
Balancing Image Compression and Generation with Bootstrapped Tokenization
Introduces SelfBootTok, a self-bootstrapped tokenization method that separates global and local information, reducing generator computation by ~40% and achieving a new state-of-the-art gFID of 1.56 with only 64 tokens.