tokenizer

Claude Sonnet 5 may be released later today, featuring a new tokenizer, high-resolution vision, and marketed as Sonnet-priced near-Opus performance, but it may not exceed Opus 4.8.

Introduces BrainG3N, a dual-purpose tokenizer for 3D brain MRI latent diffusion using a frozen masked autoencoder encoder for clinically informative embeddings and a CNN decoder for reconstruction, achieving state-of-the-art performance on a 23-task benchmark and enabling controllable generation and longitudinal forecasting.

quicktok is a fast and exact BPE tokenizer in C++ that is byte-identical with tiktoken, achieving 2–11x speedup over existing alternatives. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 encoders.

Introduces llm.istanbul, a WebGPU LLM workbench that lets you train small models, train tokenizers, and generate text entirely in the browser, no server required, fully local.

Introduces UniTok, a universal tokenizer that transforms continuous time series into discrete tokens, and UniTok-FM, a foundation model pretrained via next-token prediction that enables zero-shot and prompt-boosted forecasting as well as few-shot generation and classification through training-free in-context inference.

MiniCPM5-1B is a new small language model from OpenBMB, apparently built from scratch with its own tokenizer and distinct behavior, generating excitement as a capable 1B model.

This pull request adds tokenizer support for MiniCPM5 to llama.cpp, extending the tool's compatibility with the MiniCPM family of models.

SupraLabs released Supra-50M, a compact 50M-parameter causal language model with base and instruct versions, trained on 20B tokens from fineweb-edu, achieving competitive benchmarks against larger models like GPT-2 and SmolLM.

ztok 是一个用 Zig 编写的高性能多线程分词器库，支持多种格式（tiktoken、HF、SentencePiece 等），速度比现有方案快 2–5 倍，适用于 RAG 分块和数据集分词。

HuggingFace releases Carbon, a DNA model that is 275x faster than the previous state-of-the-art (Evo2), enabling processing of the entire human genome on a single GPU in under two days. The model uses a unique tokenizer that splits sequences into 6-base chunks while maintaining single-base resolution, and comes with an interactive demo.

A technique to make embedding models aware of number ordering by overriding tokenizer and MLM fine-tuning, achieving 59% accuracy on number sorting benchmarks.

MAML is a novel multi-modal AI model that unifies understanding of chemistry, genetics, and proteins, outperforming specialized models on 11 drug discovery benchmarks, promising to accelerate pharmaceutical research and improve success rates.

Andrew Ng's newsletter covers recent AI developments including attacks on data centers, the release of Qwen3.5 in various sizes, DeepSeek's collaboration with Huawei, and Apple's multimodal tokenizer, alongside reflections on AI-driven job uncertainty and geopolitical risks.

Kronos is an open-source foundation model for financial K-line sequences, trained on data from over 45 global exchanges. It uses a specialized tokenizer and a decoder-only Transformer, and has been accepted at AAAI 2026.