Tag
This article explores using Byte-Pair Encoding (BPE) token efficiency as a more effective alternative to entropy for detecting secrets in code, focusing on statistical rarity over randomness.
This paper proposes Pruned BPE, a post-training method that prunes low-exposure tokens from a BPE vocabulary and reallocates slots to better-exposed candidates, reducing encoded length without increasing model-visible vocabulary size. Experiments on English and Chinese corpora show approximately 0.27–0.36% encoded length reduction over standard BPE.
This blog post presents an algorithm using integer linear programming to compute optimal tokenizers for language models, drawing parallels to solving the Traveling Salesman Problem. It notes that while the result is theoretically interesting, practical tokenizers are already near-optimal and the method may not generalize well.
This paper introduces an incremental algorithm for Byte Pair Encoding (BPE) tokenization that processes each byte in O(log^2 t) time, enabling efficient partial tokenization in streaming settings and achieving speedups over existing implementations.