Hierarchical Latent Prediction for Language Models

# Hierarchical Latent Prediction for Language Models
Source: [https://arxiv.org/html/2608.05806](https://arxiv.org/html/2608.05806)
Chang Shi1,Tim Pearce2,Manan Tomar2,Siddhartha Sen2,John Langford2 1University of Texas at Austin,2Microsoft ResearchWork done during internship at Microsoft Research\. Correspondence to: Chang Shi[<changshi@utexas\.edu\>](https://arxiv.org/html/2608.05806v1/[email protected])

###### Abstract

While standard Next\-Token Prediction \(NTP\) lays the foundation of language model pretraining, its teacher\-forced training paradigm may not be optimal for long\-horizon reasoning and planning\. Recent works such as Multi\-Token Prediction \(MTP\) and Next\-Latent prediction \(NextLat\) try to mitigate the problem through predicting multiple future tokens and self\-supervised prediction in the latent space\. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi\-step rollout\. We introduceHierarchicalLatentPrediction\(HiLP\), which introduces an auxiliary higher\-level abstract latent to help reduce the error accumulation effect in latent\-space rollouts\. Experiments show that HiLP can lead to longer\-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi\-step reasoning benchmarks, and offers more speculative decoding efficiency\.

Hierarchical Latent Prediction for Language Models

Chang Shi1††thanks:Work done during internship at Microsoft Research\. Correspondence to: Chang Shi[<changshi@utexas\.edu\>](https://arxiv.org/html/2608.05806v1/[email protected]), Tim Pearce2, Manan Tomar2, Siddhartha Sen2, John Langford21University of Texas at Austin,2Microsoft Research

## 1Introduction

Next\-token prediction \(NTP\) with teacher forcing has yielded transformers of remarkable capability, yet it also introduces a training\-inference mismatch known as exposure bias: during inference time, the auto\-regressive token generation relies on its own previous outputs, leading to compounding errors which could degrade long\-range generation quality\.

Recent work NextLat\(Teohet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib31)\)partially mitigates the issue through enforcing a latent space self\-prediction loss\. However, its latent dynamics model predicts one step ahead\. This means that the self\-predictive learning signal, while effective in inducing local transition consistency, provides only indirect gradient pressure to form representations that capture structure at longer temporal scales\. In practice, long\-horizon dependencies still remain attenuated by sequential dynamics unrolling\.

Multi\-scale temporal abstraction methods have a long history in sequence modeling and reinforcement learning\(Suttonet al\.,[1999](https://arxiv.org/html/2608.05806#bib.bib29); Baconet al\.,[2017](https://arxiv.org/html/2608.05806#bib.bib12)\)\. Inherently, language incorporates hierarchical structures at multiple scales, from characters, phrases to sentences, paragraphs and discourses\. Therefore, it is reasonable to extend NextLat with hierarchical learning signals at different granularities, for the sake of richer latent transition dynamics\. This observation motivates a natural question:can we introduce temporal hierarchy directly into the latent\-space self\-prediction objective, shaping the representation of a language model transformer to encode multi\-scale predictive structure during pretraining?

In this paper, we answer affirmatively by introducingHierarchical Latent Prediction \(HiLP\)\. Our contributions are as follows:

1. 1\.We propose HiLP, a hierarchical representation training method that introduces multi\-scale self\-predictive learning into transformer pretraining via sliding\-window attention over latents, a higher\-level dynamics model, and a combined NTP head\.
2. 2\.We evaluate the resulting models on downstream benchmark accuracy and speculative decoding efficiency, comparing HiLP with MTP and NextLat\.
3. 3\.We demonstrate that the entire hierarchical apparatus can be removed at inference time with no architectural overhead, preserving NextLat’s option of being a pure training\-time intervention\.

## 2Methodology

We useX1:TX\_\{1:T\}to denote the token sequence prefix\. A transformerGθG\_\{\\theta\}produces*lower\-level*latents𝐡t=Gθ​\(X1:t\)\\mathbf\{h\}\_\{t\}=G\_\{\\theta\}\(X\_\{1:t\}\)\. A sliding\-window attention \(SWA\) module with windowkkproduces*higher\-level*abstract latents𝐮t=SWA​\(𝐡t−k\+1:t\)\\mathbf\{u\}\_\{t\}=\\mathrm\{SWA\}\(\\mathbf\{h\}\_\{t\-k\+1:t\}\), a deterministic function of the lastkklower states\.

A higher\-level dynamics model is trained to predictkksteps ahead in this abstract space\. This explicit coarser\-scale prediction objective encourages lookahead planning beyond single\-step latent transitions\. A combined NTP head then conditions the next\-token prediction on both level latents, taking advantage of abstract lookahead information to improve subsequent predictions\. Four parameterised maps are jointly learned: output headpθp\_\{\\theta\}, lower dynamicspψp\_\{\\psi\}, upper dynamicspϕp\_\{\\phi\}, and combined headpρp\_\{\\rho\}\. An illustration of the architecture is shown in Fig\.[1](https://arxiv.org/html/2608.05806#S2.F1)\.

#### Training objectives\.

The total training objective is then a weighted sum of five terms: the standard NTP loss, the NextLat \(lower\-level\) transition consistency loss, the KL term, the higher\-level transition consistency loss, and the combined NTP loss\. At inference time, only the standard NTP head is used, the entire hierarchical apparatus serves purely as an auxiliary training signal\.

ℒntp\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{ntp\}\}=𝔼t<T​\[−log⁡pθ​\(Xt\+1∣𝐡t\)\]\\displaystyle=\\mathbb\{E\}\_\{t<T\}\\\!\\left\[\-\\log p\_\{\\theta\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\}\)\\right\]\(A\)ℒh\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{h\}\}=𝔼t​\[1d​∑i=1dSL1​\(sg​\[𝐡t\+i\],𝐡^t\+i\)\]\\displaystyle=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\tfrac\{1\}\{d\}\\textstyle\\sum\_\{i=1\}^\{d\}\\mathrm\{SL1\}\(\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\+i\}\],\\hat\{\\mathbf\{h\}\}\_\{t\+i\}\)\\right\]\(B\)ℒKL\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{KL\}\}=𝔼t\[1d∑i=1dDKL\(pθsg\(⋅∣sg\[𝐡t\+i\]\)∥\\displaystyle=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\tfrac\{1\}\{d\}\\textstyle\\sum\_\{i=1\}^\{d\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{\\theta\}^\{\\mathrm\{sg\}\}\(\\cdot\\mid\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\+i\}\]\)\\,\\middle\\\|\\right\.\\right\.pθsg\(⋅∣𝐡^t\+i\)\)\]\\displaystyle\\quad\\quad\\left\.\\left\.p\_\{\\theta\}^\{\\mathrm\{sg\}\}\(\\cdot\\mid\\hat\{\\mathbf\{h\}\}\_\{t\+i\}\)\\right\)\\right\]\(C\)ℒu\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{u\}\}=𝔼t​\[SmoothL1​\(sg​\[𝐮t\+k\],𝐮^t\+k\)\]\\displaystyle=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\mathrm\{SmoothL1\}\(\\mathrm\{sg\}\[\\mathbf\{u\}\_\{t\+k\}\],\\,\\hat\{\\mathbf\{u\}\}\_\{t\+k\}\)\\right\]\(D\)ℒcntp\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{cntp\}\}=𝔼t<T\[\\displaystyle=\\mathbb\{E\}\_\{t<T\}\\\!\\bigl\[−logpρ\(Xt\+1∣𝐡t,𝐮~t\)\]\\displaystyle\\quad\-\\log p\_\{\\rho\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\},\\tilde\{\\mathbf\{u\}\}\_\{t\}\)\\bigr\]\(E\)
WhereSL1​\(⋅,⋅\)\\mathrm\{SL1\}\(\\cdot,\\cdot\)denotes the SmoothL1 loss,sg​\[⋅\]\\mathrm\{sg\}\[\\cdot\]the stop\-gradient operator, and𝐮~t:=SWA​\(sg​\[𝐡t−k\+1:t\]\)\\tilde\{\\mathbf\{u\}\}\_\{t\}\\\!:=\\\!\\mathrm\{SWA\}\(\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\-k\+1:t\}\]\)the higher\-level latent withsg\\mathrm\{sg\}applied inside SWA so gradients do not flow back into𝐡\\mathbf\{h\}\. We use this stop\-gradient in the combined NTP path to keep the token\-level representation𝐡t\\mathbf\{h\}\_\{t\}governed by the standard NTP and lower\-level transition consistency losses, while allowing the combined head to train the SWA module and its higher\-level representation\. Without this separation, the combined\-head cross\-entropy would also update the lower latents through the SWA window, double\-counting token\-level supervision and creating a competing optimization signal for𝐡t\\mathbf\{h\}\_\{t\}\.

#### Consistency conditions\.

At optimality, the losses enforce:

pθ​\(Xt\+1∣𝐡t\)\\displaystyle p\_\{\\theta\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\}\)=ℙ​\(Xt\+1∣X1:t\)\\displaystyle=\\mathbb\{P\}\(X\_\{t\+1\}\\mid X\_\{1:t\}\)\(I\)pψ​\(𝐡t\+1∣𝐡t,Xt\+1\)\\displaystyle p\_\{\\psi\}\(\\mathbf\{h\}\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\},X\_\{t\+1\}\)=ℙ​\(𝐡t\+1∣X1:t\+1\)\\displaystyle=\\mathbb\{P\}\(\\mathbf\{h\}\_\{t\+1\}\\mid X\_\{1:t\+1\}\)\(II\)pϕ​\(𝐮t\+k∣𝐮t\)\\displaystyle p\_\{\\phi\}\(\\mathbf\{u\}\_\{t\+k\}\\mid\\mathbf\{u\}\_\{t\}\)=ℙ​\(𝐮t\+k∣X1:t\)\\displaystyle=\\mathbb\{P\}\(\\mathbf\{u\}\_\{t\+k\}\\mid X\_\{1:t\}\)\(III\)pρ​\(Xt\+1∣𝐡t,𝐮t\)\\displaystyle p\_\{\\rho\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\},\\mathbf\{u\}\_\{t\}\)=ℙ​\(Xt\+1∣X1:t\)\\displaystyle=\\mathbb\{P\}\(X\_\{t\+1\}\\mid X\_\{1:t\}\)\(IV\)where \(I\) and \(IV\) enforce next\-token consistency, \(II\) and \(III\) enforce transition consistency\.

![Refer to caption](https://arxiv.org/html/2608.05806v1/x1.png)Figure 1:Overview of HiLP with sliding\-window sizeW=3W=3\. The higher\-level latent is computed by sliding\-window attention over backbone latents, the higher next\-latent predictor mapsutu\_\{t\}tou^t\+W\\hat\{u\}\_\{t\+W\}without token input, and both the standard NTP head and combined NTP head are trained, but the combined NTP head is dropped during inference to ensure no slow down in inference speed\.

## 3Experiments

1B\-parameter models are trained on 100B tokens using 8×\\timesNVIDIA B200 GPUs\.

### 3\.1Coding and Multi\-step Reasoning Benchmarks

After pretraining, we use LM Evaluation Harness\(Gaoet al\.,[2024](https://arxiv.org/html/2608.05806#bib.bib32)\)to evaluate the zero\-shot performance of the models on HumanEval coding benchmarkChenet al\.\([2021](https://arxiv.org/html/2608.05806#bib.bib34)\), and DataComp for LLMs\(Liet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib33)\)to evaluate the models on a set of symbolic and multi\-step benchmarks\. Results in Tab\.[1](https://arxiv.org/html/2608.05806#S3.T1)and Fig\.[2](https://arxiv.org/html/2608.05806#S3.F2)show that HiLP is improve performance from the multi\-scale latent prediction\.

Table 1:Code\-generation benchmark accuracy on HumanEval\. HumanEval results are computed from 1000 samples per task\.![Refer to caption](https://arxiv.org/html/2608.05806v1/figures/dclm/dclm_symbolic_multistep_comparison_generated.png)Figure 2:DCLM symbolic/multi\-step eval results![Refer to caption](https://arxiv.org/html/2608.05806v1/figures/ce_curve/latent_ce_curves.png)Figure 3:Latent cross\-entropy curves for NextLat and HiLP on web and code splits\. HiLP has comparable near\-term latent prediction loss and lower long\-horizon future and excess cross\-entropy, indicating more stable multi\-step latent rollout\.
### 3\.2Speculative Decoding

We evaluate speculative decoding on held\-out validation splits of code and Nemotron\-Climbmix data\(Diaoet al\.,[2026](https://arxiv.org/html/2608.05806#bib.bib2)\)\. For each model, we report theaverage number of accepted tokensper drafting step, andAvg match, the per\-position draft\-verifier argmax agreement averaged overK=1​…​4K\{=\}1\{\\ldots\}4\. Note thatK=0K\{=\}0NTP position is always accepted and is omitted from the per\-KKcolumns;

Table 2:Speculative decoding results evaluated on a held\-out validation split of code data\.Table 3:Speculative decoding results evaluated on a held\-out validation split of Nemotron\-ClimbMix data\.

## 4Discussion

#### Longer horizon prediction\.

The latent prediction cross\-entropy losses across rollout horizons up to 8 steps ahead are shown in[3](https://arxiv.org/html/2608.05806#S3.F3)\. As the curves show, HiLP preserves the near\-term cross\-entropy of NextLat while producing lower future prediction error at longer horizons, suggesting that the hierarchical latent mitigates multi\-step error accumulation\.

## 5Related Work

Our work sits at the intersection of several active research threads: pretraining objectives beyond next\-token prediction, latent\-space language modeling, and hierarchical models\.

### 5\.1Pretraining Objectives Beyond Next\-Token Prediction

The next\-token prediction objective has largely limited the capacity of language models in downstream tasks that require longer\-horizon reasoning and planning\(Bachmann and Nagarajan,[2024](https://arxiv.org/html/2608.05806#bib.bib11); Nagarajanet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib35)\)\. Recent works have started to introduce auxiliary learning signals to mitigate this myopic gap through further future prediction\([Gloeckleet al\.,](https://arxiv.org/html/2608.05806#bib.bib16); Ahnet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib10); Teohet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib31); Mahajanet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib27)\)\. However, these approaches usually operate on a single granularity scale and have limited horizon capability due to compounding error\. HiLP differs in two ways: it introduces two latent prediction pathways at different temporal scales, so the coarser pathway supervises long\-horizon structure directly rather than through repeated single\-step unrolling; and all auxiliary heads are dropped at inference, so the longer\-horizon signal costs nothing at deployment time\.

### 5\.2Latent\-Space Language Modeling

Another line of work moves language generation itself into a continuous latent space\. Large Concept Models\(Barraultet al\.,[2024](https://arxiv.org/html/2608.05806#bib.bib1)\)perform autoregressive prediction over sentence\-level embeddings, treating each sentence as a “concept” in a shared representation space\. CALM\(Shaoet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib5)\)replaces next\-token prediction with next\-vector prediction, compressing a chunk ofKKtokens into a single continuous vector to raise the semantic bandwidth of each generative step\. CoconutShaoet al\.\([2025](https://arxiv.org/html/2608.05806#bib.bib5)\)lets the model reason in latent space by feeding its own hidden state back as the next input embedding instead of decoding to tokens\. All of these approaches change the inference\-time generation process to operate on latents, requiring bespoke decoding procedures or latent\-to\-text decoders\. HiLP is the opposite design point: latent prediction is used purely as an auxiliary training signal to shape representations, while inference remains standard token\-level autoregressive decoding with zero added latency\.

### 5\.3Hierarchical Models

Hierarchical model design has been researched in both language modeling area and other domains\.

MegaByte\(Yuet al\.,[2023](https://arxiv.org/html/2608.05806#bib.bib9)\)stacks a global patch\-level transformer over a local byte\-level one, the Byte Latent Transformer\(Pagnoniet al\.,[2025](https://arxiv.org/html/2608.05806#bib.bib7)\)dynamically segments bytes into entropy\-based patches that serve as the units of computation, and H\-NetHwanget al\.\([2025](https://arxiv.org/html/2608.05806#bib.bib8)\)learns content\-dependent chunking end\-to\-end within a hierarchical U\-Net\-like network\.

In model\-based reinforcement learning, Hierarchical Planning with Latent World Models\(Zhanget al\.,[2026](https://arxiv.org/html/2608.05806#bib.bib3)\)learns world models at multiple temporal scales within a shared latent space, using long\-horizon latent predictions as subgoals for short\-horizon control\. In representation learning theory, Learning Discrete Concepts in Latent Hierarchical Models\(Konget al\.,[2024](https://arxiv.org/html/2608.05806#bib.bib6)\)formalizes concepts as discrete latent variables organized in a hierarchical causal model and derives identifiability conditions for recovering such hierarchical concept structure from high\-dimensional unsupervised data\. In time series analysis\. HiTimeTaoet al\.\([2024](https://arxiv.org/html/2608.05806#bib.bib4)\)employs a hierarchical feature encoder together with a hybrid prompting strategy to align time series and text modalities, improving multivariate time series classification with large language models\. These works demonstrate the benefit of hierarchical latent structure across planning, representation identifiability, and time series domains\. HiLP differs in a way that the hierarchy complexity is not consumed at inference time: to our knowledge it is the first to introduce hierarchical latent prediction as a pretraining objective for autoregressive language models, where the hierarchy shapes the representation during training and is then removed entirely\.

## 6Conclusion

HiLP introduces temporal hierarchy into the latent\-space self\-prediction objective, shaping the representation of language model transformers to encode multi\-scale predictive structure during pretraining\. Experiments show that it improves both speculative decoding and language models on tasks that need longer\-horizon reasoning and planning\.

## 7Future Work

In current work, the abstract latent prediction horizon is a manually set hyperparameter, which limits the flexibility of lookahead planning\. Further designs including dynamically choosing the lookahead horizon could be beneficial\. Also, the NTP head is used during inference in the current implementation, but combined NTP can also be used if we trade speed for accuracy\.

## References

- Efficient joint prediction of multiple future tokens\.arXiv preprint arXiv:2503\.21801\.Cited by:[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- G\. Bachmann and V\. Nagarajan \(2024\)The pitfalls of next\-token prediction\.arXiv preprint arXiv:2403\.06963\.Cited by:[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- P\. Bacon, J\. Harb, and D\. Precup \(2017\)The option\-critic architecture\.InProceedings of the AAAI conference on artificial intelligence,Vol\.31\.Cited by:[§1](https://arxiv.org/html/2608.05806#S1.p3.1)\.
- L\. Barrault, P\. Duquenne, M\. Elbayad, A\. Kozhevnikov, B\. Alastruey, P\. Andrews, M\. Coria, G\. Couairon, M\. R\. Costa\-jussà, D\. Dale,et al\.\(2024\)Large concept models: language modeling in a sentence representation space\.arXiv preprint arXiv:2412\.08821\.Cited by:[§5\.2](https://arxiv.org/html/2608.05806#S5.SS2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374Cited by:[§3\.1](https://arxiv.org/html/2608.05806#S3.SS1.p1.1)\.
- S\. Diao, Y\. Yang, Y\. Fu, X\. Dong, D\. Su, M\. Kliegl, Z\. Chen, P\. Belcak, Y\. Suhara, H\. Yin,et al\.\(2026\)Nemotron\-climb: clustering\-based iterative data mixture bootstrapping for language model pre\-training\.Advances in Neural Information Processing Systems38\.Cited by:[§3\.2](https://arxiv.org/html/2608.05806#S3.SS2.p1.3)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§3\.1](https://arxiv.org/html/2608.05806#S3.SS1.p1.1)\.
- \[8\]F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. SynnaeveBetter & faster large language models via multi\-token prediction, 2024\.URL https://arxiv\. org/abs/2404\.19737\.Cited by:[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- S\. Hwang, B\. Wang, and A\. Gu \(2025\)Dynamic chunking for end\-to\-end hierarchical sequence modeling\.arXiv preprint arXiv:2507\.07955\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p2.1)\.
- L\. Kong, G\. Chen, B\. Huang, E\. Xing, Y\. Chi, and K\. Zhang \(2024\)Learning discrete concepts in latent hierarchical models\.Advances in Neural Information Processing Systems37,pp\. 36938–36975\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p3.1)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. Abbas, C\. Hsieh, D\. Ghosh, J\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. G\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. Shankar \(2025\)DataComp\-lm: in search of the next generation of training sets for language models\.External Links:2406\.11794,[Link](https://arxiv.org/abs/2406.11794)Cited by:[§3\.1](https://arxiv.org/html/2608.05806#S3.SS1.p1.1)\.
- D\. Mahajan, S\. Goyal, B\. Y\. Idrissi, M\. Pezeshki, I\. Mitliagkas, D\. Lopez\-Paz, and K\. Ahuja \(2025\)Beyond multi\-token prediction: pretraining llms with future summaries\.arXiv preprint arXiv:2510\.14751\.Cited by:[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- V\. Nagarajan, C\. H\. Wu, C\. Ding, and A\. Raghunathan \(2025\)Roll the dice & look before you leap: going beyond the creative limits of next\-token prediction\.External Links:2504\.15266,[Link](https://arxiv.org/abs/2504.15266)Cited by:[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- A\. Pagnoni, R\. Pasunuru, P\. Rodriguez, J\. Nguyen, B\. Muller, M\. Li, C\. Zhou, L\. Yu, J\. E\. Weston, L\. Zettlemoyer,et al\.\(2025\)Byte latent transformer: patches scale better than tokens\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9238–9258\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p2.1)\.
- C\. Shao, D\. Li, F\. Meng, and J\. Zhou \(2025\)Continuous autoregressive language models\.arXiv preprint arXiv:2510\.27688\.Cited by:[§5\.2](https://arxiv.org/html/2608.05806#S5.SS2.p1.1)\.
- R\. S\. Sutton, D\. Precup, and S\. Singh \(1999\)Between mdps and semi\-mdps: a framework for temporal abstraction in reinforcement learning\.Artificial intelligence112\(1\-2\),pp\. 181–211\.Cited by:[§1](https://arxiv.org/html/2608.05806#S1.p3.1)\.
- X\. Tao, T\. Pan, M\. Cheng, Y\. Luo, Q\. Liu, and E\. Chen \(2024\)Hierarchical multimodal llms with semantic space alignment for enhanced time series classification\.ACM Transactions on Intelligent Systems and Technology\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p3.1)\.
- J\. Teoh, M\. Tomar, K\. Ahn, E\. S\. Hu, T\. Pearce, P\. Sharma, A\. Krishnamurthy, R\. Islam, A\. Lamb, and J\. Langford \(2025\)Next\-latent prediction transformers learn compact world models\.arXiv preprint arXiv:2511\.05963\.Cited by:[§1](https://arxiv.org/html/2608.05806#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.05806#S5.SS1.p1.1)\.
- L\. Yu, D\. Simig, C\. Flaherty, A\. Aghajanyan, L\. Zettlemoyer, and M\. Lewis \(2023\)Megabyte: predicting million\-byte sequences with multiscale transformers\.Advances in Neural Information Processing Systems36,pp\. 78808–78823\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p2.1)\.
- W\. Zhang, B\. Terver, A\. Zholus, S\. Chitnis, H\. Sutaria, M\. Assran, R\. Balestriero, A\. Bar, A\. Bardes, Y\. LeCun,et al\.\(2026\)Hierarchical planning with latent world models\.arXiv preprint arXiv:2604\.03208\.Cited by:[§5\.3](https://arxiv.org/html/2608.05806#S5.SS3.p3.1)\.

## Appendix AHiLP Training Procedure

Algorithm[1](https://arxiv.org/html/2608.05806#alg1)summarizes one HiLP training step, combining the five objectives into a single backward pass\. The SWA module is a causal sliding\-window self\-attention over the lower latents, so every positionttcarries its own higher\-level latent𝐮t\\mathbf\{u\}\_\{t\}summarizing the lastkklower latents\. All latent\-space prediction targets are stop\-gradiented, and both distributions in the KL term are decoded through a frozen copy of the NTP head \(pθsgp\_\{\\theta\}^\{\\mathrm\{sg\}\}\), so each auxiliary loss shapes the representation only through its designated pathway: the rollout losses reach𝐡\\mathbf\{h\}throughpψp\_\{\\psi\}, the higher\-level consistency loss trainspϕp\_\{\\phi\}and the SWA module, and the combined cross\-entropy trainspρp\_\{\\rho\}and the SWA module alone via the stop\-gradient inside𝐮~t\\tilde\{\\mathbf\{u\}\}\_\{t\}\. At inference, onlyGθG\_\{\\theta\}andpθp\_\{\\theta\}are kept: the SWA module, both dynamics models, and the combined head are discarded\.

Algorithm 1One HiLP training step\.E​\(⋅\)E\(\\cdot\)denotes the backbone,kkthe SWA window size and lookahead offset, andddthe rollout depth\. Expectations overttaverage over all valid positions in the batch\.1:batch

X1:TX\_\{1:T\}; backbone

GθG\_\{\\theta\}with NTP head

pθp\_\{\\theta\}; lower dynamics

pψp\_\{\\psi\}; SWA module; higher dynamics

pϕp\_\{\\phi\}; combined head

pρp\_\{\\rho\}; loss weights

λntp,λh,λKL,λu,λcntp\\lambda\_\{\\mathrm\{ntp\}\},\\lambda\_\{\\mathrm\{h\}\},\\lambda\_\{\\mathrm\{KL\}\},\\lambda\_\{\\mathrm\{u\}\},\\lambda\_\{\\mathrm\{cntp\}\}
2:

𝐡1:T←Gθ​\(X1:T\)\\mathbf\{h\}\_\{1:T\}\\leftarrow G\_\{\\theta\}\(X\_\{1:T\}\)⊳\\trianglerightone backbone pass

3:

ℒntp←𝔼t​\[−log⁡pθ​\(Xt\+1∣𝐡t\)\]\\mathcal\{L\}\_\{\\mathrm\{ntp\}\}\\leftarrow\\mathbb\{E\}\_\{t\}\\big\[\{\-\\log p\_\{\\theta\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\}\)\}\\big\]⊳\\trianglerightEq\. \([A](https://arxiv.org/html/2608.05806#S2.Ex1)\)

4:Lower\-level rollout \(teacher\-forced tokens\):

5:

𝐡^t←𝐡t\\hat\{\\mathbf\{h\}\}\_\{t\}\\leftarrow\\mathbf\{h\}\_\{t\}for all

tt;

ℒh,ℒKL←0\\mathcal\{L\}\_\{\\mathrm\{h\}\},\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\\leftarrow 0
6:for

i=1,…,di=1,\\dots,ddo

7:

𝐡^t\+i←pψ​\(𝐡^t\+i−1,E​\(Xt\+i\)\)\\hat\{\\mathbf\{h\}\}\_\{t\+i\}\\leftarrow p\_\{\\psi\}\\big\(\\hat\{\\mathbf\{h\}\}\_\{t\+i\-1\},\\,E\(X\_\{t\+i\}\)\\big\)
8:

ℒh\+=1d𝔼t\[SL1\(sg\[𝐡t\+i\],𝐡^t\+i\)\]\\mathcal\{L\}\_\{\\mathrm\{h\}\}\\mathrel\{\+\}=\\tfrac\{1\}\{d\}\\,\\mathbb\{E\}\_\{t\}\\big\[\\mathrm\{SL1\}\\big\(\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\+i\}\],\\,\\hat\{\\mathbf\{h\}\}\_\{t\+i\}\\big\)\\big\]⊳\\trianglerightEq\. \([B](https://arxiv.org/html/2608.05806#S2.Ex2)\)

9:

ℒKL\+=1d𝔼t\[DKL\(pθsg\(⋅∣sg\[𝐡t\+i\]\)∥pθsg\(⋅∣𝐡^t\+i\)\)\]\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\\mathrel\{\+\}=\\tfrac\{1\}\{d\}\\,\\mathbb\{E\}\_\{t\}\\big\[D\_\{\\mathrm\{KL\}\}\\big\(p\_\{\\theta\}^\{\\mathrm\{sg\}\}\(\\cdot\\mid\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\+i\}\]\)\\,\\big\\\|\\,p\_\{\\theta\}^\{\\mathrm\{sg\}\}\(\\cdot\\mid\\hat\{\\mathbf\{h\}\}\_\{t\+i\}\)\\big\)\\big\]⊳\\trianglerightEq\. \([C](https://arxiv.org/html/2608.05806#S2.Ex4)\)

10:endfor

11:Higher\-level channel:

12:

𝐮t←SWA​\(𝐡t−k\+1:t\)\\mathbf\{u\}\_\{t\}\\leftarrow\\mathrm\{SWA\}\(\\mathbf\{h\}\_\{t\-k\+1:t\}\)⊳\\trianglerightabstract latent

13:

𝐮^t\+k←pϕ​\(𝐮t\)\\hat\{\\mathbf\{u\}\}\_\{t\+k\}\\leftarrow p\_\{\\phi\}\(\\mathbf\{u\}\_\{t\}\)⊳\\trianglerightkk\-step lookahead, no token input

14:

ℒu←𝔼t​\[SmoothL1​\(sg​\[𝐮t\+k\],𝐮^t\+k\)\]\\mathcal\{L\}\_\{\\mathrm\{u\}\}\\leftarrow\\mathbb\{E\}\_\{t\}\\big\[\\mathrm\{SmoothL1\}\\big\(\\mathrm\{sg\}\[\\mathbf\{u\}\_\{t\+k\}\],\\,\\hat\{\\mathbf\{u\}\}\_\{t\+k\}\\big\)\\big\]⊳\\trianglerightEq\. \([D](https://arxiv.org/html/2608.05806#S2.Ex5)\)

15:Combined head \(sg keeps𝐡\\mathbf\{h\}governed by NTP and rollout\):

16:

𝐮~t←SWA​\(sg​\[𝐡t−k\+1:t\]\)\\tilde\{\\mathbf\{u\}\}\_\{t\}\\leftarrow\\mathrm\{SWA\}\\big\(\\mathrm\{sg\}\[\\mathbf\{h\}\_\{t\-k\+1:t\}\]\\big\)
17:

ℒcntp←𝔼t​\[−log⁡pρ​\(Xt\+1∣𝐡t,𝐮~t\)\]\\mathcal\{L\}\_\{\\mathrm\{cntp\}\}\\leftarrow\\mathbb\{E\}\_\{t\}\\big\[\{\-\\log p\_\{\\rho\}\(X\_\{t\+1\}\\mid\\mathbf\{h\}\_\{t\},\\tilde\{\\mathbf\{u\}\}\_\{t\}\)\}\\big\]⊳\\trianglerightEq\. \([E](https://arxiv.org/html/2608.05806#S2.Ex7)\)

18:

ℒ←λntp​ℒntp\+λh​ℒh\+λKL​ℒKL\+λu​ℒu\+λcntp​ℒcntp\\mathcal\{L\}\\leftarrow\\lambda\_\{\\mathrm\{ntp\}\}\\mathcal\{L\}\_\{\\mathrm\{ntp\}\}\+\\lambda\_\{\\mathrm\{h\}\}\\mathcal\{L\}\_\{\\mathrm\{h\}\}\+\\lambda\_\{\\mathrm\{KL\}\}\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\+\\lambda\_\{\\mathrm\{u\}\}\\mathcal\{L\}\_\{\\mathrm\{u\}\}\+\\lambda\_\{\\mathrm\{cntp\}\}\\mathcal\{L\}\_\{\\mathrm\{cntp\}\}
19:One optimizer step on

∇ℒ\\nabla\\mathcal\{L\}w\.r\.t\.

θ\\theta,

ψ\\psi,

ϕ\\phi,

ρ\\rho, and the SWA module

## Appendix BExperiment hyperparameters

GroupHyperparameterValueBackbone modelVocabulary size100,352Training sequence length8,192Hidden layers24Attention heads16Latent input combinationGLU\-crossArchitectureHidden multiplier2Latent window size4Latent lookahead steps4NTP loss weight1\.0MSE loss weight10\.0Loss weightsKL loss weight0\.1Higher\-level MSE loss weight1\.0Combined NTP loss weight0\.5Input combinationGLU\-crossCombined headDownstream LM headNTPDraft LM headNTPOptimizationLatent learning rate1\.0×10−31\.0\\times 10^\{\-3\}Latent evaluationEvaluation horizon8Table 4:HiLP hyperparameter configuration\.
## Appendix CLatent overhead and efficiency\.

Table[5](https://arxiv.org/html/2608.05806#A3.T5)compares the four 100B\-token code\-data runs along three axes: total parameter count, the subset of parameters actually used during greedy NTP decoding, and training throughput\. All four models share the same 1\.06 B\-parameter trunk and tied\-free LM head, so the verifier path used at inference is identical and the additional parameters in MTP, NextLat, and HiLP are auxiliary draft components that are not required to produce the next\-token distribution\. NextLat adds a small \(≈\\approx19 M\) next\-latent predictor on top of the trunk, while HiLP and MTP each add a≈\\approx190–200 M draft module\. Training throughput scales inversely with the size of the auxiliary loss graph: the NTP baseline reaches≈\\approx126 K tokens/s/GPU, NextLat retains≈\\approx83% of that throughput, HiLP retains≈\\approx65%, and MTP drops to≈\\approx49% because its four future\-token heads must each be evaluated against the LM logits at every step\.

Model\# ParamsInf\. ParamsTraining Step Breakdown \(ms\)Training ThroughputStepFwdBwdOptimTok/s/GPU↑\\uparrowTF/GPU↑\\uparrowMFU↑\\uparrowSamp/s↑\\uparrowNTP1\.06 B1\.06 B44911929337126,2788980\.399123\.3MTP1\.25 B1\.06 B9166312434361,7895090\.22660\.3NextLat1\.08 B1\.06 B54220529542104,5058840\.393102\.1HiLP1\.27 B1\.06 B6903293154681,6537800\.34779\.7Table 5:Parameter counts and training efficiency for the 100B\-token runs on8×8\\\!\\times\\\!B200 GPUs\.\# Params: total trained parameters\.Inf\. Params: verifier path \(trunk \+ LM head\) used during greedy NTP decoding, identical across all models\.Step breakdown:timing/step\_time\_secand its components \(forward, backward, optimizer\), in milliseconds\.Throughput: tokens/s/GPU, TFLOP/s/GPU, model FLOPs utilization, and samples/s \(all GPUs\)\.
## Appendix DInput\-Combination Modes

We include an implementation\-level ablation comparing the historicalconcatinput\-combination mode against the currentglu\_crossmode used by latent prediction modules\. LetDDdenote the hidden size,mmthe latent\-head hidden multiplier, andVVthe vocabulary size\. In the lower next\-latent predictor,concatforms\[𝐡t;et\+1\]∈ℝ2​D\[\\mathbf\{h\}\_\{t\};e\_\{t\+1\}\]\\in\\mathbb\{R\}^\{2D\}and feeds it directly to the SwiGLU predictor\. This gives an intermediate width of2​m​D2mDand costs10​m​D210mD^\{2\}parameters and multiply\-adds per token for the gate, up, and down projections, ignoring biases\. By contrast,glu\_crosscomputes

Wh​𝐡t⊙σ​\(We​et\+1\)∈ℝD,W\_\{h\}\\mathbf\{h\}\_\{t\}\\odot\\sigma\(W\_\{e\}e\_\{t\+1\}\)\\in\\mathbb\{R\}^\{D\},adding twoD×DD\\times Dprojections but reducing the following SwiGLU width tom​DmD\. Its corresponding cost is therefore\(3​m\+2\)​D2\(3m\+2\)D^\{2\}\. With the defaultm=4m=4, this is14​D214D^\{2\}versus40​D240D^\{2\}, or about35%35\\%of theconcatpredictor cost\.

The same distinction appears in the combined NTP head\. Forconcat, the combined head maps\[𝐡t;𝐮t\]∈ℝ2​D\[\\mathbf\{h\}\_\{t\};\\mathbf\{u\}\_\{t\}\]\\in\\mathbb\{R\}^\{2D\}to logits, costing2​D​V2DV\. Forglu\_cross, the gated fusion costs2​D22D^\{2\}and the logits are produced from aDD\-dimensional vector, costingD​VDV, for a total ofD​V\+2​D2DV\+2D^\{2\}\. Thusglu\_crossis especially attractive whenV≫DV\\gg D: it approximately halves the combined\-head logit computation while also reducing the latent predictor from a2​D2D\-wide to aDD\-wide SwiGLU input\.

Table 6:Input\-combination ablation comparing next\-token prediction loss and training step time\.![Refer to caption](https://arxiv.org/html/2608.05806v1/x2.png)Figure 4:concatinput\-combination mode used by the latent and combined NTP heads\.![Refer to caption](https://arxiv.org/html/2608.05806v1/x3.png)Figure 5:glu\_crossinput\-combination mode, which gates a projected lower latent with the conditioning latent while preserving hidden widthDD\.