tokenizer

Marcel Rød announces Gigatoken, a tokenizer implementation that is 500-1000x faster than HuggingFace and 100x faster than OpenAI's tiktoken, built in Rust.

Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.

Gigatoken is an open-source tokenizer that achieves up to 1000x speedup over HuggingFace tokenizers and 100x over Tiktoken, using SIMD and caching optimizations. It supports drop-in replacement for existing tokenizer APIs.

The article reveals that the actual cost of using frontier models varies significantly due to tokenizer differences, with TypeScript costing up to 73% more tokens on Claude than GPT, hidden from pricing pages.

Qwen-Music is a music generation model that produces high-fidelity songs with vocals, supporting text-to-music and cover song generation. It uses a novel Melody-Chain-of-Thought mechanism and achieves state-of-the-art results on 13 of 16 objective metrics.

This paper explores the feasibility of reconstructing the tokenizer of a closed-source large language model using only two oracle queries from its chat API, potentially revealing internal representation details.

This article argues that comparing AI models by price per million tokens is misleading due to differences in tokenizers and token efficiency. It provides a benchmark cost analysis showing that models with higher per-token prices can be cheaper per completed task, with DeepSeek V4 Pro being a strong cost-efficiency outlier.

Anthropic released Claude Sonnet 5, a model with performance near Opus 4.8 at lower prices, but featuring a new tokenizer that increases token counts for English and code by ~30%, effectively raising costs.

Anthropic released Claude Sonnet 5 with improved reasoning, tool use, and coding, but its updated tokenizer maps text to more tokens (up to 1.35×), increasing effective cost per task despite the same listed price; introductory pricing applies until August 31, 2026.

Claude Sonnet 5 may be released later today, featuring a new tokenizer, high-resolution vision, and marketed as Sonnet-priced near-Opus performance, but it may not exceed Opus 4.8.

Introduces BrainG3N, a dual-purpose tokenizer for 3D brain MRI latent diffusion using a frozen masked autoencoder encoder for clinically informative embeddings and a CNN decoder for reconstruction, achieving state-of-the-art performance on a 23-task benchmark and enabling controllable generation and longitudinal forecasting.

quicktok is a fast and exact BPE tokenizer in C++ that is byte-identical with tiktoken, achieving 2–11x speedup over existing alternatives. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 encoders.

Introduces llm.istanbul, a WebGPU LLM workbench that lets you train small models, train tokenizers, and generate text entirely in the browser, no server required, fully local.

Introduces UniTok, a universal tokenizer that transforms continuous time series into discrete tokens, and UniTok-FM, a foundation model pretrained via next-token prediction that enables zero-shot and prompt-boosted forecasting as well as few-shot generation and classification through training-free in-context inference.

MiniCPM5-1B is a new small language model from OpenBMB, apparently built from scratch with its own tokenizer and distinct behavior, generating excitement as a capable 1B model.

This pull request adds tokenizer support for MiniCPM5 to llama.cpp, extending the tool's compatibility with the MiniCPM family of models.

SupraLabs released Supra-50M, a compact 50M-parameter causal language model with base and instruct versions, trained on 20B tokens from fineweb-edu, achieving competitive benchmarks against larger models like GPT-2 and SmolLM.

ztok 是一个用 Zig 编写的高性能多线程分词器库，支持多种格式（tiktoken、HF、SentencePiece 等），速度比现有方案快 2–5 倍，适用于 RAG 分块和数据集分词。

HuggingFace releases Carbon, a DNA model that is 275x faster than the previous state-of-the-art (Evo2), enabling processing of the entire human genome on a single GPU in under two days. The model uses a unique tokenizer that splits sequences into 6-base chunks while maintaining single-base resolution, and comes with an interactive demo.

A technique to make embedding models aware of number ordering by overriding tokenizer and MLM fine-tuning, achieving 59% accuracy on number sorting benchmarks.