tokenizer

A preview of the upcoming VocabX paper showing how specializing a small language model's vocabulary around token cost rather than frequency cut measured inference cost by 77% on a production Portuguese essay grader built on Qwen3-4B, without changing the base model.

Stanford HazyResearch 提出 Next-Concept Modeling (NCM)，一种既非自回归也非扩散的粗到细并行文本生成方法，据称比强扩散基线快 10–31 倍且训练 tokens 更少；不过实验规模仅到 725M 参数，能否扩展至通用 LLM 规模仍待验证。

We present NEPSEE, a Nepali sentiment analysis benchmark of 3,189 double-annotated Devanagari sentences, evaluating encoders, TF-IDF baselines and zero-shot LLMs, plus a tokenizer audit showing that default tokenizers of Nepali BERT models strip non-spacing Devanagari vowel signs and the virama.

Niles Salter’s talk introduces how to use SIMD data parallelism and bit manipulation primitives (CTZ, TZCNT, etc.) to accelerate a Zig tokenizer, achieving a throughput of 820 MB/s, and explores the potential for further speedups with AVX-512.

Simon Willison 发布 ttok 1.0，将默认分词器从 GPT-4 切换为 GPT-5/GPT-6 系列，依据是实验确认 GPT-6 与 GPT-5 家族使用相同 tokenizer。

该论文测量了2026年主流LLM分词器对法语及法国地方语言的token溢价（法语比英语多31%–58%，地方语言高达1.6–3.3倍），并提出法语优化的字节级BPE原型Baracoda FR v1.2，在法语上比Tekken少用11.5%的token。

StableVQ proposes practical guidelines to stabilize the training of vector-quantized tokenizers by decoupling encoder-decoder and codebook training, improving stability and codebook utilization for image generation models.

The paper proposes 'unit' as an explicit primitive in machine learning, where learning tasks declare persistent individuals, and supervised learning specializes to unit-conditioned response laws with tokenization.

The article discusses Sander Land's reproduction of Claude's tokenizer, revealing its surprisingly small vocabulary size compared to models like Qwen 3.8, and explores the theoretical implications of the softmax bottleneck in language models as described in a recent paper.

Through tokenizer comparison, error string analysis, and temp-0 output matching, the author concludes that Ox Alpha is likely based on the GLM-5.3 model, possibly a variant or updated version.

NE-BERT is a multilingual encoder model trained on 8.3 million sentences for nine Northeast Indian languages and two anchor languages, outperforming existing models like IndicBERT-V2 and MuRIL with significant gains on ultra-low-resource languages through custom tokenization.

Liquid AI ran an experiment where coding agents autonomously built a production-grade BPE tokenizer trainer called toktoktok, sharing lessons on designing effective agent loops and verification infrastructure for real-world problem-solving.

This paper investigates whether released LLM tokenizer vocabularies can support fine-grained token-level estimation of hidden pretraining corpora, proposing a Quantile-Guided Density Estimation (QGDE) method that achieves low error rates in controlled and realistic settings.

This paper introduces Sampled-BPE, a lightweight token-level auditing pipeline for web-scale Chinese corpora, and applies it to reveal widespread but uneven pollution across open Chinese datasets and Common Crawl snapshots. It also releases a hierarchical dataset of over 630k polluted token records with context and explanations.

This paper trains five GPT-2-style models from scratch to compare dedicated monolingual models for Tamil, Telugu, Kannada, and Malayalam against a joint multilingual model, finding monolingual models outperform mGPT on sentiment classification and NER with more efficient tokenizers.

This paper introduces KVAE, a family of tokenizers for audio, image, and video designed for text-conditioned generative models, claiming competitive or superior reconstruction and generation quality compared to existing open-source tokenizers. The code and training details are publicly released.

This paper introduces BPE-guided insertion for post-hoc tokenizer adaptation on byte-level BPE models, keeping vocabulary size fixed and preserving most token-ID assignments. The method reduces Ukrainian token counts by ~33-36% while minimizing impact on English and other European languages.

Mage-VL is an efficient codec-native streaming multimodal foundation model that reduces visual token consumption by over 75% using a custom tokenizer, achieving up to 3.5x inference speedup while matching or outperforming existing models on static and video tasks.

Marcel Rød announces Gigatoken, a tokenizer implementation that is 500-1000x faster than HuggingFace and 100x faster than OpenAI's tiktoken, built in Rust.

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.