The State-Prediction Separation Hypothesis
Summary
The paper proposes the state-prediction separation hypothesis, suggesting that separating state prediction from token prediction in Transformers improves language modeling performance and efficiency across scales, with experiments showing consistent gains.
View Cached Full Text
Cached at: 07/02/26, 07:47 AM
Paper page - The State-Prediction Separation Hypothesis
Source: https://huggingface.co/papers/2607.01218
Abstract
Separating state prediction from token prediction in Transformers improves language modeling performance and efficiency across different scales.
Transformersuse the sameforward computation streamto both predict the next token and store useful state for future token predictions. We formulate thestate-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses twocomputation streamsto separate the two functions, and conductpretrainingexperiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improvingvalidation lossand outperforming standardTransformersby 2--3 percentage points on average ondownstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in thegradientsour design entails.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.01218
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.01218 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.01218 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.01218 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Comparing Transformers and Hybrid Models at the Token Level
This paper analyzes token-level prediction differences between transformers and hybrid attention-recurrent models using Olmo 3 and Olmo Hybrid, finding that hybrids improve on semantic state tracking while transformers excel at n-gram copying and syntactic bracket matching.
Hierarchical Latent Prediction for Language Models
This paper introduces HiLP, a hierarchical representation training method that adds multi-scale self-predictive learning to transformer pretraining, aiming to reduce compounding error and improve long-horizon reasoning and speculative decoding efficiency.
@rohanpaul_ai: New Microsoft paper argues that transformers generalize better when they learn compact internal states, not just next t…
Microsoft's NextLat paper proposes a self-supervised training method where transformers predict their next hidden state instead of just the next token, leading to more compact world models, better planning and reasoning, and up to 3.3x faster generation.
Next-Latent Prediction Transformers [R]
Microsoft Research introduces Next-Latent Prediction (NextLat), a self-supervised method that trains transformers to predict their own next latent state, enabling compact world models for reasoning and planning and achieving up to 3.3x faster inference via self-speculative decoding.
Geometric and Behavioral Stratification in Transformer Residual Streams
This paper investigates the residual stream geometry of trained transformers, showing that the prediction direction acts as a privileged anchor that stratifies residual-stream variation into narrow, readout-relevant and broad, computational regions across 18 models. The findings have implications for interpretability and evaluation.