multithreaded

Tag

#multithreaded

ztok — a fast multithreaded tokenizer in Zig that loads tiktoken / HF / SentencePiece and is 2–5× faster

Reddit r/LocalLLaMA ↗ · 2026-05-22

ztok 是一个用 Zig 编写的高性能多线程分词器库，支持多种格式（tiktoken、HF、SentencePiece 等），速度比现有方案快 2–5 倍，适用于 RAG 分块和数据集分词。

0 favorites 0 likes