Efficient Pre-Training with Token Superposition
Summary
Token-Superposition Training (TST) improves LLM pre-training efficiency by combining contiguous tokens into bags during a superposition phase with a multi-hot cross-entropy objective, achieving up to 2.5x reduction in training time without architectural changes.
View Cached Full Text
Cached at: 05/13/26, 04:13 PM
Paper page - Efficient Pre-Training with Token Superposition
Source: https://huggingface.co/papers/2605.06546
Abstract
Token-Superposition Training (TST) improves pre-training efficiency by combining contiguous tokens into bags during a superposition phase with multi-hot cross-entropy objective, achieving faster training times without architectural changes.
Pre-trainingof Large Language Models is often prohibitively expensive and inefficient at scale, requiring complex and invasive modifications in order to achieve highdata throughput. In this work, we presentToken-Superposition Training(TST), a simple drop-in method that significantly improves thedata throughputperFLOPsduringpre-trainingwithout modifying theparallelism,optimizer,tokenizer, data, ormodel architecture. TST is done in two phases: (i) A highly efficientsuperposition phasewhere we combine many contiguous tokens into one bag and train using amulti-hot cross-entropy(MCE) objective, and (ii) arecovery phasewhere we revert back to standard training. We extensively evaluate TST on the scale of 270M and 600M parameters and validate on 3B and a 10B A1B mixture of experts model, demonstrating that it is highly robust in different settings. Ultimately, TST consistently outperforms baseline loss and downstream evaluations, and under equal-loss settings, TST yields up to a 2.5x reduction in totalpre-trainingtime at the 10B A1B scale.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.06546 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.06546 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.06546 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Nous Research Releases Token Superposition Training to Speed Up LLM Pre-Training by Up to 2.5x Across 270M to 10B Parameter Models
Nous Research releases Token Superposition Training (TST), a method that speeds up LLM pre-training by up to 2.5x across models from 270M to 10B parameters, reducing wall-clock time without altering architecture or data.
SuperThoughts: Reasoning Tokens in Superposition
SuperThoughts compresses consecutive chain-of-thought tokens into latent representations and decodes two tokens per step, achieving ~20–30% CoT length reduction with minimal accuracy loss on math reasoning benchmarks, while doubling inference throughput.
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
SEATS is a training-free, stage-adaptive token selection method that reduces computational overhead in omni-modal LLMs by progressively pruning redundant visual and audio tokens, achieving a 9.3x FLOPs reduction and 4.8x prefill speedup while preserving 96.3% performance.
Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages
This paper investigates whether pretraining LLMs on artificial languages (pre-pretraining) consistently improves token efficiency across multiple natural languages, finding that gains are highly dependent on experimental setup and random seed, though stable gains appear for small models with the Llama tokenizer.
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
This paper investigates the impact of subword tokenization on LLM training efficiency and performance by conducting controlled byte-level pretraining experiments. It reveals key factors such as training throughput and the integration of subword boundaries as linguistic priors.