tokenization

Liquid AI shares a recipe for upgrading a pretrained model's tokenizer in place, expanding LFM2.5's tokenizer from 65K to 128K to improve efficiency for languages like Thai, Vietnamese, and Hindi, resulting in up to 4x fewer tokens and 2.2-3.7x faster generation.

This paper investigates how multilingual tokenizers handle cross-lingual homographs (identical surface forms with different meanings across languages) and proposes a lightweight language-cue intervention that introduces language-specific characters to reduce token sharing. Experiments show modest improvements in machine translation, particularly with BPE tokenization.

This paper introduces a unified conceptual framework for discrete diffusion models, analyzing their design space through tokenization, state space construction, and highlighting trade-offs in training, inference, and scaling.

Ant Digital Technologies launched Anvita, a new Web3+AI brand, at the Real Up Cannes Summit to build infrastructure for the agent-to-agent (A2A) digital economy, featuring Anvita Flow for autonomous agent collaboration and Anvita TaaS for tokenization-as-a-service.

This blog post starts a series on building an APL interpreter in Python, covering tokenization and parsing of basic APL expressions with numbers, functions, and operators.

This paper systematically compares BPE and Unigram-LM tokenization methods for SMILES strings in chemistry, finding they produce near-disjoint vocabularies and differ significantly in segmentation granularity. The results show that the choice of subword algorithm is a critical modeling decision, not a free default.

This paper proposes a speaker-disentangled syllabic tokenizer that uses regression of perturbed student representations toward clean teacher targets, achieving state-of-the-art syllable boundary detection and a 7% relative improvement in speech language model understanding over SpiRit-LM.

Victoria Lin's Stanford CS25 lecture discusses the paradigm shift from language models to native multimodal intelligence, covering tokenization approaches and comparing models like Chameleon and Transfusion.

AVTok proposes a unified 1D tokenizer for audio-video generation using a dual-stream transformer with shared encoder-decoder and modal-specific queries, achieving compact latent representations and excelling in reconstruction and downstream generation tasks.

BrainJanus is the first unified brain model that integrates brain, vision, and language via a shared Omni space, enabling bidirectional mapping between neural activity and sensory stimuli through tokenized representation and autoregressive next-token prediction.

This paper proposes a feed-forward framework that decomposes 3D scenes into instance-structured token groups from unposed multi-view images, enabling direct object-level reconstruction, segmentation, and manipulation without 3D annotations.

This paper systematically quantifies the tokenization penalty for 20 African languages across 11 frontier and open tokenizers, finding up to 8.9× inference cost and latency multipliers and as little as 11% effective context window compared to English, highlighting a structural digital divide encoded in subword vocabularies.

This paper systematically investigates the optimal order of preprocessing techniques for sentiment analysis on Twitter data, finding that tokenisation is most impactful and spelling correction least, with the best order being tokenisation, cleaning, stemming, then stopword removal.

This paper presents QuechuaTok, a benchmark for evaluating tokenization strategies for Southern Quechua, and introduces Morphological Boundary Accuracy (MorphAcc) as a necessary metric. It shows that BPE achieves low fertility but poor morphological accuracy, while a morphology-aware PRPE tokenizer achieves 83% MorphAcc, demonstrating that fertility rate alone is insufficient for agglutinative languages.

This paper compares four tokenization methods (Affine, AIM, JetFormer, VQ-VAE) for astronomical images within a unified transformer framework, using 640,000 galaxy images to evaluate reconstruction quality, physical property prediction, and morphological preservation. It finds that no single method excels across all tasks, highlighting trade-offs in representation learning.

TOTEN is a knowledge-based ontological tokenization framework that replaces statistical tokenization with declarative classification grounded in a formal ontology of engineering entities, achieving high ontological atomicity and numerical reconstruction for physical quantities and technical notation in Brazilian Portuguese.

BioMatrix is a multimodal foundation model that unifies molecular sequences, structures, and natural language in a single decoder-only architecture, achieving state-of-the-art performance on 77 out of 80 biological tasks.

A developer working on an AI agent wrapper observes that the agent's hallucinations of user responses can actually aid problem-solving, and proposes treating such hallucinations as imagined events rather than errors.

This paper introduces CADE, a framework for time-series question answering that maps each timestep directly into the LLM embedding space and uses a one-directional supervised contrastive loss to align time-series representations with frozen text anchors, outperforming existing baselines on the Time-MQA benchmark.

This paper presents Morpheus, a neural tokenizer and word embedder for Turkish that learns morpheme boundaries without string normalization, achieving lossless tokenization and competitive embeddings for lexical retrieval, while using less GPU memory than subword tokenizers.