tokenization

A user shares an observation that Qwen and Gemma tokenize code very differently, with Qwen using far fewer tokens for the same HTML/JS input, which may explain differences in coding and language performance. They also note a potential retraining project by LiquidAI using a more efficient tokenizer.

This arXiv paper evaluates federated training of tokenized generative event models (GEMs) on ICU EHR data from three health systems, showing that federated learning preserves most centralized performance and improves cross-site transportability compared to conventional supervised models.

This paper proposes Pruned BPE, a post-training method that prunes low-exposure tokens from a BPE vocabulary and reallocates slots to better-exposed candidates, reducing encoded length without increasing model-visible vocabulary size. Experiments on English and Chinese corpora show approximately 0.27–0.36% encoded length reduction over standard BPE.

TokTier is a stateful tokenization service that cuts tokenization overhead in agentic LLM serving by reusing stable token boundaries and running exact CPU/GPU tokenization, reducing time-to-first-token by 16–34% under vLLM.

This paper proposes GenCDSR, a generative framework for cross-domain sequential recommendation with hybrid tokenization and serial-parallel decoding, achieving improved accuracy and significantly reduced inference latency compared to state-of-the-art baselines.

This paper investigates whether language models remain robust to alternative (non-canonical) tokenizations across 27 languages, finding that invariance observed in English does not generalize and that languages with higher token fragmentation show greater sensitivity. The authors demonstrate that LoRA fine-tuning with multi-tokenization data can mitigate this sensitivity.

This paper introduces JOLT, an integer programming approach to optimize subword tokenization for greedy left-to-right longest-match decoding (WordPiece). JOLT achieves near-optimal compression, closing most of the gap between BPE and the theoretical lower bound, reducing token count by up to 0.78% over BPE.

This paper introduces BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced multilingual corpus for classical Indian languages. Subword fertility analysis shows significant improvements in tokenization efficiency, reducing sequence length by up to 90% compared to baseline tokenizers, thereby enhancing effective context length for downstream language models.

Brazilian farmers tokenized 10 dairy cows on the B3 stock exchange, raising nearly $20,000 in credit backed by livestock, using AI-powered tracking collars to prevent fraud and enable movable collateral.

GigaToken is an ultra-fast tokenizer library that claims up to 1000x speedup over HuggingFace tokenizers, supporting most common LLM tokenizers and providing drop-in compatibility.

Liquid AI shares a recipe for upgrading a pretrained model's tokenizer in place, expanding LFM2.5's tokenizer from 65K to 128K to improve efficiency for languages like Thai, Vietnamese, and Hindi, resulting in up to 4x fewer tokens and 2.2-3.7x faster generation.

This paper investigates how multilingual tokenizers handle cross-lingual homographs (identical surface forms with different meanings across languages) and proposes a lightweight language-cue intervention that introduces language-specific characters to reduce token sharing. Experiments show modest improvements in machine translation, particularly with BPE tokenization.

This paper introduces a unified conceptual framework for discrete diffusion models, analyzing their design space through tokenization, state space construction, and highlighting trade-offs in training, inference, and scaling.

Ant Digital Technologies launched Anvita, a new Web3+AI brand, at the Real Up Cannes Summit to build infrastructure for the agent-to-agent (A2A) digital economy, featuring Anvita Flow for autonomous agent collaboration and Anvita TaaS for tokenization-as-a-service.

This blog post starts a series on building an APL interpreter in Python, covering tokenization and parsing of basic APL expressions with numbers, functions, and operators.

This paper systematically compares BPE and Unigram-LM tokenization methods for SMILES strings in chemistry, finding they produce near-disjoint vocabularies and differ significantly in segmentation granularity. The results show that the choice of subword algorithm is a critical modeling decision, not a free default.

This paper proposes a speaker-disentangled syllabic tokenizer that uses regression of perturbed student representations toward clean teacher targets, achieving state-of-the-art syllable boundary detection and a 7% relative improvement in speech language model understanding over SpiRit-LM.

Victoria Lin's Stanford CS25 lecture discusses the paradigm shift from language models to native multimodal intelligence, covering tokenization approaches and comparing models like Chameleon and Transfusion.

AVTok proposes a unified 1D tokenizer for audio-video generation using a dual-stream transformer with shared encoder-decoder and modal-specific queries, achieving compact latent representations and excelling in reconstruction and downstream generation tasks.

BrainJanus is the first unified brain model that integrates brain, vision, and language via a shared Omni space, enabling bidirectional mapping between neural activity and sensory stimuli through tokenized representation and autoregressive next-token prediction.