tokenizers v1 (rust)

Reddit r/LocalLLaMA 09/21/26, 03:10 PM Tools

tokenization hugging-face rust nlp library version-1 multi-threading

Summary

Hugging Face has released version 1 of their tokenizers library, featuring multiple language support, multi-thread scaling, and minimal package size.

Hey all! I am Aritra from Hugging Face. I wanted to share an update on the `tokenizers` library that we have at Hugging Face. It has gone under major changes and we have finally released version 1 of it. Here are what we are most excited about: > multiple language support > multi-thread scaling > minimal package size Read: https://huggingface.co/blog/tokenizers-v1

Original Article

Similar Articles

tokenizers v1: encode, decode and scaling, measured

Hugging Face Blog

Hugging Face releases tokenizers v1, a major performance update for the tokenization library, with benchmarks showing significant speed improvements over previous versions.

@percyliang: ! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the …

X AI KOLs Following

Marcel Rød announces Gigatoken, a tokenizer implementation that is 500-1000x faster than HuggingFace and 100x faster than OpenAI's tiktoken, built in Rust.

@LysandreJik: Tokenizers v1's first release candidate is out absolutely huge release across the board: Up to 30x faster tokenization,…

X AI KOLs Following

Tokenizers v1's first release candidate delivers up to 30x faster tokenization with enhancements in speed, memory usage, and scalability, while maintaining full API compatibility with previous versions.

Gigatoken (GitHub Repo)

TLDR AI

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.

Gigatoken: A new open source tokenizer ~100x faster than Tiktoken, -500-1000x faster than Huggingface

Reddit r/LocalLLaMA

Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.