tokenizer

The paper proposes 'unit' as an explicit primitive in machine learning, where learning tasks declare persistent individuals, and supervised learning specializes to unit-conditioned response laws with tokenization.

The article discusses Sander Land's reproduction of Claude's tokenizer, revealing its surprisingly small vocabulary size compared to models like Qwen 3.8, and explores the theoretical implications of the softmax bottleneck in language models as described in a recent paper.

Through tokenizer comparison, error string analysis, and temp-0 output matching, the author concludes that Ox Alpha is likely based on the GLM-5.3 model, possibly a variant or updated version.

NE-BERT is a multilingual encoder model trained on 8.3 million sentences for nine Northeast Indian languages and two anchor languages, outperforming existing models like IndicBERT-V2 and MuRIL with significant gains on ultra-low-resource languages through custom tokenization.

Liquid AI ran an experiment where coding agents autonomously built a production-grade BPE tokenizer trainer called toktoktok, sharing lessons on designing effective agent loops and verification infrastructure for real-world problem-solving.

This paper investigates whether released LLM tokenizer vocabularies can support fine-grained token-level estimation of hidden pretraining corpora, proposing a Quantile-Guided Density Estimation (QGDE) method that achieves low error rates in controlled and realistic settings.

This paper introduces Sampled-BPE, a lightweight token-level auditing pipeline for web-scale Chinese corpora, and applies it to reveal widespread but uneven pollution across open Chinese datasets and Common Crawl snapshots. It also releases a hierarchical dataset of over 630k polluted token records with context and explanations.

This paper trains five GPT-2-style models from scratch to compare dedicated monolingual models for Tamil, Telugu, Kannada, and Malayalam against a joint multilingual model, finding monolingual models outperform mGPT on sentiment classification and NER with more efficient tokenizers.

This paper introduces KVAE, a family of tokenizers for audio, image, and video designed for text-conditioned generative models, claiming competitive or superior reconstruction and generation quality compared to existing open-source tokenizers. The code and training details are publicly released.

This paper introduces BPE-guided insertion for post-hoc tokenizer adaptation on byte-level BPE models, keeping vocabulary size fixed and preserving most token-ID assignments. The method reduces Ukrainian token counts by ~33-36% while minimizing impact on English and other European languages.

Mage-VL is an efficient codec-native streaming multimodal foundation model that reduces visual token consumption by over 75% using a custom tokenizer, achieving up to 3.5x inference speedup while matching or outperforming existing models on static and video tasks.

Marcel Rød announces Gigatoken, a tokenizer implementation that is 500-1000x faster than HuggingFace and 100x faster than OpenAI's tiktoken, built in Rust.

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.

Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.

The article reveals that the actual cost of using frontier models varies significantly due to tokenizer differences, with TypeScript costing up to 73% more tokens on Claude than GPT, hidden from pricing pages.

Qwen-Music is a music generation model that produces high-fidelity songs with vocals, supporting text-to-music and cover song generation. It uses a novel Melody-Chain-of-Thought mechanism and achieves state-of-the-art results on 13 of 16 objective metrics.

This paper explores the feasibility of reconstructing the tokenizer of a closed-source large language model using only two oracle queries from its chat API, potentially revealing internal representation details.

This article argues that comparing AI models by price per million tokens is misleading due to differences in tokenizers and token efficiency. It provides a benchmark cost analysis showing that models with higher per-token prices can be cheaper per completed task, with DeepSeek V4 Pro being a strong cost-efficiency outlier.

Anthropic released Claude Sonnet 5, a model with performance near Opus 4.8 at lower prices, but featuring a new tokenizer that increases token counts for English and code by ~30%, effectively raising costs.

Anthropic released Claude Sonnet 5 with improved reasoning, tool use, and coding, but its updated tokenizer maps text to more tokens (up to 1.35×), increasing effective cost per task despite the same listed price; introductory pricing applies until August 31, 2026.