ztok — a fast multithreaded tokenizer in Zig that loads tiktoken / HF / SentencePiece and is 2–5× faster

Reddit r/LocalLLaMA 05/22/26, 08:49 AM Tools

tokenizer zig multithreaded fast local-pipelines rag dataset-tokenization

Summary

ztok 是一个用 Zig 编写的高性能多线程分词器库，支持多种格式（tiktoken、HF、SentencePiece 等），速度比现有方案快 2–5 倍，适用于 RAG 分块和数据集分词。

I built ztok, a tokenizer library focused on being fast and format-agnostic for local pipelines. \- Loads what you already have — .tiktoken, HF tokenizer.json, SentencePiece .model, TokenMonster, Mistral Tekken. Auto-detected. \- Bit-identical to tiktoken / HF / SentencePiece on the equivalence gate, so it's a drop-in. \- Faster on the same vocab + same bytes (cl100k vs tiktoken, EPYC 24c/48t): \~2× single-thread, 3.8–5.5× batched (\~291–425 MB/s vs \~78). Also faster than HF tokenizers andSentencePiece on their own vocabs. \- 8 language bindings over one C ABI — Python, Node, Ruby, Go, Rust, .NET, Java, Swift. \- Built for the boring-but-useful jobs: RAG chunking with token-cap windows + byte-accurate offsets, and dataset tokenization straight to .bin/.npy for training. Zig 0.16, AGPL-3.0, \~1100 tests. Feedback welcome, especially on vocab formats I'm missing. [https://github.com/sirus20x6/ztok](https://github.com/sirus20x6/ztok)

Original Article

Similar Articles

quicktok: a faster tokenizer (exact and byte-identical with tiktoken) [P]

Reddit r/MachineLearning

quicktok is a fast and exact BPE tokenizer in C++ that is byte-identical with tiktoken, achieving 2–11x speedup over existing alternatives. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 encoders.

Gigatoken: A new open source tokenizer ~100x faster than Tiktoken, -500-1000x faster than Huggingface

Reddit r/LocalLLaMA

Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.

Accelerating the Zig Tokenizer (with AVX-512)

Lobsters Hottest

Niles Salter’s talk introduces how to use SIMD data parallelism and bit manipulation primitives (CTZ, TZCNT, etc.) to accelerate a Zig tokenizer, achieving a throughput of 820 MB/s, and explores the potential for further speedups with AVX-512.

Gigatoken (GitHub Repo)

TLDR AI

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.

ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixes

Hugging Face Daily Papers

ZipTok3D is a 3D tokenizer that achieves high-fidelity reconstruction from extremely short token sequences by organizing geometry into compact global-token prefixes and using iterative decoding, outperforming baselines with significantly reduced token lengths.