bpe

This paper introduces TokEval, a framework for evaluating language model tokenizers using intrinsic metrics that correlate with downstream task performance.

Liquid AI ran an experiment where coding agents autonomously built a production-grade BPE tokenizer trainer called toktoktok, sharing lessons on designing effective agent loops and verification infrastructure for real-world problem-solving.

This paper investigates whether released LLM tokenizer vocabularies can support fine-grained token-level estimation of hidden pretraining corpora, proposing a Quantile-Guided Density Estimation (QGDE) method that achieves low error rates in controlled and realistic settings.

This paper introduces JOLT, an integer programming approach to optimize subword tokenization for greedy left-to-right longest-match decoding (WordPiece). JOLT achieves near-optimal compression, closing most of the gap between BPE and the theoretical lower bound, reducing token count by up to 0.78% over BPE.

This paper introduces BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced multilingual corpus for classical Indian languages. Subword fertility analysis shows significant improvements in tokenization efficiency, reducing sequence length by up to 90% compared to baseline tokenizers, thereby enhancing effective context length for downstream language models.

This paper systematically compares BPE and Unigram-LM tokenization methods for SMILES strings in chemistry, finding they produce near-disjoint vocabularies and differ significantly in segmentation granularity. The results show that the choice of subword algorithm is a critical modeling decision, not a free default.

quicktok is a fast and exact BPE tokenizer in C++ that is byte-identical with tiktoken, achieving 2–11x speedup over existing alternatives. It supports cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3 encoders.