Microsoft's Full-bandwidth Transformers (26 minute read)

# Full-bandwidth transformer
Source: [https://arxiv.org/html/2608.08888](https://arxiv.org/html/2608.08888)
Ziyang CaiZheng ZhanHarry DongYing Fan Gustavo de RosaTim PearceJohn Langford\[0\.5em\] Johns Hopkins University Princeton University Microsoft

###### Abstract

Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth\. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top\-layer hidden state is discarded\. We introduce the*full\-bandwidth transformer*, which widens this channel with*latent feedback*: at each decoding step, the previous top\-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input\. Latent feedback lets non\-verbalized computation re\-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language\-modeling objective\. To train full\-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi\-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability\. We train 1B\-parameter full\-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5\-shot language\-model evaluation, math and coding generation, and instruction\-tuned performance\. With negligible per\-token decoding overhead, full\-bandwidth transformers match or approach standard transformers trained with roughly1\.5×1\.5\\timesmore tokens, and manage to produce shorter reasoning traces at equal or better accuracy\.

††footnotetext:\*Correspondence to Xi Wang <[xwang457@cs\.jhu\.edu](mailto:[email protected])\>, John Langford <[jcl@microsoft\.com](mailto:[email protected])\>\.†\\daggerWork done during an internship at Microsoft AI Frontiers\.## 1Introduction

Scaling large language models has largely meant increasing model parameters and training on more tokens\([22](https://arxiv.org/html/2608.08888#bib.bib39)\)\. As pre\-training continues to scale, however, the availability of high\-quality unique data becomes an increasingly constraint\. This motivates revisiting the scaling axes themselves: rather than relying solely on more data, can we extract more useful learning signal from each token by allocating more computation to it? Recurrent, iterative, and feedback\-based computation offer a natural way to pursue this direction, but additional FLOPs matter only if they translate into richer representations during training or more effective computation at inference time\.

Autoregressive transformers expose a particularly underused opportunity for such computation\. They already contain a feedback loop: the token sampled at stept−1t\-1becomes the input at steptt\(Fig\.[1](https://arxiv.org/html/2608.08888#S1.F1), left\)\. This loop is what lets chain\-of\-thought decoding\([33](https://arxiv.org/html/2608.08888#bib.bib37)\)perform computation whose depth grows with the number of generated tokens\([24](https://arxiv.org/html/2608.08888#bib.bib12)\)\. But measured as a communication channel, the loop is extremely narrow: Decoding compresses the model’s entire top\-layer state, aDD\-dimensional vector, down to a single symbol carrying at mostlog2⁡\|V\|\\log\_\{2\}\|V\|bits\. Non\-verbalized computation is not erased—intermediate activations persist in the KV cache and remain accessible—but it is*depth\-frozen*: a state produced at layerℓ\\ellis readable only by layers aboveℓ\\ell, so it can never return to the bottom of the stack for further processing, and the deepest state of all, the top layer’s output, is never cached\. Verbalization is thus the only channel by which information re\-enters the bottom layer and receives fresh computation, at the cost of being squeezed through a single token\. The model must either spend tokens narrating its intermediate state or recompute that state from scratch at every position\.

In this work, we propose*full\-bandwidth transformer*where we widen this channel to its full width\. In particular, we introduce latent feedback decoding, which fuses the previous top\-layer hidden state with the sampled token’s embedding during decoding, through a gated linear unit using the state on the value pathway, the token acting as the gate, and feeds the result back as the next input \(Fig\.[1](https://arxiv.org/html/2608.08888#S1.F1)right, Sec\.[3\.1](https://arxiv.org/html/2608.08888#S3.SS1)\)\. We call a transformer capable of decoding this way a full\-bandwidth transformer, since its inter\-step feedback now carries the entire hidden state rather than a thin token\. The sampled token is retained, so the model still produces ordinary text and can be flexibly trained with standard supervised language modeling losses; what changes is that the feedback is no longer limited to the token’s identity\. By design, this affords two things standard decoding lacks: \(i\) non\-verbalized state—uncertainty, partial results, plans—can re\-enter the bottom of the stack with a renewed depth budget and be processed further across steps, rather than staying frozen in the cache at the level where it was produced; \(ii\) every layer, including the shallowest, sees the past as processed by the*full*stack, not only by the layers beneath it; Crucially, these come with almost no architecture changes and extra serving cost: the fusion adds two matrix multiplications per generated token, attention and the KV cache are untouched, and prefill is run either once or, optionally, twice for better performance\.

The obstacle is training\. A pretrained model has never seen hidden states in its input, so latent feedback cannot simply be switched on at inference; and the recurrence it defines is sequential over positions, so training on it directly would forfeit the parallel teacher forcing that makes transformers efficient to train\. We resolve this with a*multi\-pass*regime \(Sec\.[3\.3](https://arxiv.org/html/2608.08888#S3.SS3)\): each pass shifts the previous pass’s hidden states one position rightward, fuses them with the token embeddings, and re\-runs the stack in parallel across all positions, so sequentiality is paid across a handful of passes rather than across the sequence\. Two ingredients make this practical\. A*progressive schedule*spends the bulk of training on the ordinary single\-pass objective such that the run can start from a standard pretraining checkpoint and introduces extra feedback passes only late; and a*prefix mixin*randomizes where fused inputs begin within a sequence, matching the prompt\-then\-generate structure of inference\. Empirically, we find the schedule’s composition matters in an unexpected way: training with two feedback passes alone produces a recurrence that*diverges*once rolled past its trained depth, whereas mixing in as little as 3% three\-pass batches turns the learned map into a*contraction*toward a fixed point that stays stable beyond the trained depth \(Fig\.[3](https://arxiv.org/html/2608.08888#S3.F3)\)\.

Empirically, full\-bandwidth transformers convert negligible extra inference compute into gains equivalent to substantially more training data\. Utilizing multiple forward pass for prefill, the recurrence\-trained model matches no\-recurrence baselines trained on twice the tokens in both validation loss and multiple\-choice accuracy \(Fig\.[4](https://arxiv.org/html/2608.08888#S4.F4)\)\. On free\-form generation \(Fig\.[5](https://arxiv.org/html/2608.08888#S4.F5)\)—GSM8K, Math500, HumanEval, MBPP—latent feedback improves over standard decoding of the*same*weights on every task, matches the2×2\\times\-token baselines, and on some tasks approaches baselines trained with up to5×5\\timesthe tokens; the gains carry over through long\-context extension and instruction tuning \(Table[1](https://arxiv.org/html/2608.08888#S4.T1)\)\. On base models, latent feedback often yields markedly shorter reasoning traces at equal or better accuracy \(Fig\.[6](https://arxiv.org/html/2608.08888#S4.F6)and[8](https://arxiv.org/html/2608.08888#S4.F8)\)—the behavior the widened channel predicts, with computation riding the hidden state instead of being verbalized token by token\.

![Refer to caption](https://arxiv.org/html/2608.08888v1/figures/fig1.png)Figure 1:Standard decoding vs\. latent feedback decoding\.Left: In a standard transformer, the current state can access only lower\-layer past states \(blue\); deeper past states \(white\) are unreachable, and the only inter\-step feedback is the sampled token embedding \(green\)\.Right: A*full\-bandwidth transformer*uses*latent feedback*, fusing the previous top\-layer hidden state with the sampled token embedding through a dimension\-preserving gate \(⊗\\otimes, Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\)\) and feeding it back as the next input\. This returns full hidden\-state information to the bottom of the stack, making the past as processed by all layers accessible to subsequent computation\.
## 2Background

Given a vocabulary of size\|V\|\|V\|and aDD\-dimensional residual stream, a decoder\-only LLM maps an input sequence ofTTtokens, with embeddings\{𝒆t\}t=1T∈ℝT×D\\\{\\bm\{e\}\_\{t\}\\\}\_\{t=1\}^\{T\}\\in\\mathbb\{R\}^\{T\\times D\}, throughLLattention–MLP blocks\. The final\-layer hidden states\{𝒉tL\}t=1T\\\{\\bm\{h\}\_\{t\}^\{L\}\\\}\_\{t=1\}^\{T\}are projected by the language\-model headWhead∈ℝ\|V\|×DW\_\{\\text\{head\}\}\\in\\mathbb\{R\}^\{\|V\|\\times D\}to next\-token distributions:

𝒉tL=fθ\(𝒆t;C\),𝒆t←Decode\(𝒉t−1L\),C=𝒆0,𝒆1,…,𝒆t−1\.\\bm\{h\}\_\{t\}^\{L\}=f\_\{\\theta\}\(\\bm\{e\}\_\{t\};\\,C\),\\qquad\\bm\{e\}\_\{t\}\\leftarrow\\mathrm\{Decode\}\\\!\\left\(\\bm\{h\}\_\{t\-1\}^\{L\}\\right\),\\quad C=\\bm\{e\}\_\{0\},\\bm\{e\}\_\{1\},\\ldots,\\bm\{e\}\_\{t\-1\}\.\(1\)
#### KV cache\.

During autoregressive decoding with transformers, previously computed keys and values are cached and reused, avoiding repeated computation over the prefix\. Unlike RNNs and state\-space models, which compress history into a fixed\-size recurrent state, dense\-attention transformers retain explicit representations of all past tokens, so each new hidden state can attend directly to the full cached history\.

#### Bandwidths of a model’s horizonal axis vs\. veritical axis\.

It is useful to separate the horizontal axis \(across positions\) from the vertical axis \(across depth\), because the two carry information at different rates\.*Horizontally*, dense attention is effectively full\-bandwidth: when generating tokentt, the layer\-ℓ\\ellstate𝒉tℓ\\bm\{h\}\_\{t\}^\{\\ell\}can read the cached representations of every earlier position\.*Vertically*, access is restricted:𝒉tℓ\\bm\{h\}\_\{t\}^\{\\ell\}cannot read any deeper past state𝒉t′ℓ′\\bm\{h\}\_\{t^\{\\prime\}\}^\{\\ell^\{\\prime\}\}witht′<tt^\{\\prime\}<tandℓ′≥ℓ\\ell^\{\\prime\}\\geq\\ell\(Fig\.[1](https://arxiv.org/html/2608.08888#S1.F1), left\)\. Formally, the states reachable when computing positionttat layerℓ\\ellare

ℛstd\(t,ℓ\)=\{\(t′,ℓ′\):t′<t,ℓ′<ℓ\},\|ℛstd\|=Θ\(Tℓ\),\\mathcal\{R\}\_\{\\text\{std\}\}\(t,\\ell\)=\\big\\\{\(t^\{\\prime\},\\ell^\{\\prime\}\):t^\{\\prime\}<t,\\;\\ell^\{\\prime\}<\\ell\\big\\\},\\qquad\\bigl\\lvert\\mathcal\{R\}\_\{\\mathrm\{std\}\}\\bigr\\rvert=\\Theta\(T\\ell\),\(2\)so a shallow layer of a new token sees only a*partially processed*view of the past, even though the deeper, more fully processed states of those same positions have already been computed and sit in the cache\. Past computation therefore persists but is*depth\-frozen*in that the representations produced at layerℓ\\ellis readable only to layers aboveℓ\\elland can never be routed back down for further processing\. This is the narrow vertical channel that sec\.[3\.1](https://arxiv.org/html/2608.08888#S3.SS1)widens\.

Importantly, this depth\-wise dependency constraint is also what lets transformers train in parallel across positions: sequential computation is required only across layers, not across tokens\. At decoding time, however, generation is already sequential over tokens, so the constraint buys nothing—opening the door to richer dependencies on past hidden states, which we develop next\.

## 3Widening the bandwidth with latent feedback decoding

### 3\.1Latent feedback decoding

The central innovation in full\-bandwidth transformer is latent feedback decoding, which feeds the previous top\-layer hidden state back into the input\. At steptt,

𝒉tL=fθ\(𝒆t⊗𝒉t−1L;C\),where𝒆t←Decode\(𝑾head𝒉t−1L\),C=𝒆0,𝒆1⊗𝒉0L,…,𝒆t−1⊗𝒉t−2L\\bm\{h\}\_\{t\}^\{L\}=f\_\{\\theta\}\\\!\\left\(\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{L\};\\;C\\right\),\\qquad\\textrm\{where\}~\\bm\{e\}\_\{t\}\\leftarrow\\mathrm\{Decode\}\\\!\\left\(\\bm\{W\}^\{\\textrm\{head\}\}\\bm\{h\}\_\{t\-1\}^\{L\}\\right\),\\;C=\\bm\{e\}\_\{0\},\\bm\{e\}\_\{1\}\\otimes\\bm\{h\}\_\{0\}^\{L\},\\ldots,\\bm\{e\}\_\{t\-1\}\\otimes\\bm\{h\}\_\{t\-2\}^\{L\}\(3\)wherefθf\_\{\\theta\}is theLL\-layer transformer stack,⋅⊗⋅\\cdot\\otimes\\cdotfuses the sampled token’s embedding with the previous latent state, andCCis the past context \(the KV cache of all earlier positions\)\. Standard decoding \(Eq\. \([1](https://arxiv.org/html/2608.08888#S2.E1)\)\) is the special case in which only the sampled token crosses between steps\.

The fusion⊗\\otimesis a gated linear unit:

𝒆t⊗𝒉t−1=𝑾U​𝒉t−1⊙σ⁡\(𝑾G​𝒆t\),\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}=\\bm\{W\}^\{U\}\\bm\{h\}\_\{t\-1\}\\odot\\sigma\(\\bm\{W\}^\{G\}\\bm\{e\}\_\{t\}\),\(4\)with𝑾U,𝑾G∈ℝD×D\\bm\{W\}^\{U\},\\bm\{W\}^\{G\}\\in\\mathbb\{R\}^\{D\\times D\}\. The asymmetry is deliberate: the hidden state occupies the value pathway, while the token embedding enters only as a multiplicative gate\. A symmetric fusion such as𝒆t\+𝑾​𝒉t−1\\bm\{e\}\_\{t\}\+\\bm\{W\}\\bm\{h\}\_\{t\-1\}would leave a shortcut open: the model could suppress the state pathway, recover the plain token input, and reach ordinary pretraining loss, leaving the wide channel unused\. That shortcut is especially tempting when training starts from a standard checkpoint whose low loss the additive path can reproduce\. Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\) closes it, since discarding𝒉t−1\\bm\{h\}\_\{t\-1\}discards the input itself, and the token’s identity survives only in theDD\-dimensional gating pattern it imposes on the state\. Reading the state is thereby mandatory rather than optional\.

#### Latent feedback is free to serve\.

The added inference cost is independent of context\-length and model\-depth and under1%1\\%per token\. The state𝒉t−1L\\bm\{h\}\_\{t\-1\}^\{L\}is already computed during standard decoding, so the only extra work is the fusion: twoD×DD\\times Dmatrix multiplications, negligible against a forward pass throughLLblocks\. Because fusion preserves the input dimensionDD, the architecture, KV\-cache layout, and serving stack are untouched, and the decoding loop changes by two lines \(Fig\.[2](https://arxiv.org/html/2608.08888#S3.F2), right\)\. The scheme is also vLLM\-compatible: we store top\-layer states in a dedicated buffer, adapting the mechanism used by multi\-token\-prediction implementations \(Appendix[D](https://arxiv.org/html/2608.08888#A4)\)\.

### 3\.2Latent feedback decoding vs\. standard CoT

Standard CoT performs serial computation through a single feedback channel: each token is appended to the context and becomes the next input\. The state is the token sequence,

st\+1=st∥at,at∼πθ\(⋅∣st\)∈𝒱,st=x1:t,s\_\{t\+1\}=s\_\{t\}\\\|a\_\{t\},\\qquad a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\in\\mathcal\{V\},\\qquad s\_\{t\}=x\_\{1:t\},\(5\)so the only thing crossing between steps is the discrete action sequence\. The underlying problem\-solving state may in principle be a deterministic function of the past actions, but recovering it from the token history is itself a state\-tracking problem, and a fixed\-depth transformer has only bounded serial computation per forward pass\. CoT sidesteps this by externalizing intermediate state into language: the model writes out partial results, subgoals, and bookkeeping, then conditions future computation on the written trace\.

Let𝒖i=𝒆⁡\(ai−1\)⊗𝒛i−1\\bm\{u\}\_\{i\}=\\bm\{e\}\(a\_\{i\-1\}\)\\otimes\\bm\{z\}\_\{i\-1\}be the fused input at positionii\(with𝒖1=𝒆0\\bm\{u\}\_\{1\}=\\bm\{e\}\_\{0\}\), so the attended context isCt=𝒖1:t−1C\_\{t\}=\\bm\{u\}\_\{1:t\-1\}\. The state isst=\(a1:t,𝒛t\)s\_\{t\}=\(a\_\{1:t\},\\,\\bm\{z\}\_\{t\}\): the token trace and the most recent latent\. One step for latent feedback decoding is

at∼πθ\(⋅∣st\)∈𝒱,𝒛t\+1=fθ\(𝒆\(at\)⊗𝒛t;𝒖1:t\),a1:t\+1=a1:t∥at,a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\in\\mathcal\{V\},\\qquad\\bm\{z\}\_\{t\+1\}=f\_\{\\theta\}\\\!\\left\(\\bm\{e\}\(a\_\{t\}\)\\otimes\\bm\{z\}\_\{t\};\\;\\bm\{u\}\_\{1:t\}\\right\),\\qquad a\_\{1:t\+1\}=a\_\{1:t\}\\\|a\_\{t\},\(6\)where⋅⊗⋅\\cdot\\otimes\\cdotis the gate of Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\) andfθf\_\{\\theta\}the full stack\. The past latents𝒛1:t−1\\bm\{z\}\_\{1:t\-1\}are not carried explicitly: each is already folded into𝒖1:t\\bm\{u\}\_\{1:t\}and hence into the KV cache, so only𝒛t\\bm\{z\}\_\{t\}, which the cache never stores, propagates as a recurrence variable\.

#### Latent feedback improves computational accessibility\.

Sincezt\+1z\_\{t\+1\}is a deterministic function ofx1:t\+1x\_\{1:t\+1\}, it carries no information the context does not already determine; the gain is computational, not informational\. Concretely, re\-injection lifts the depth restriction of Eq\. \([2](https://arxiv.org/html/2608.08888#S2.E2)\), whose reachable set requiresℓ′<ℓ\\ell^\{\\prime\}<\\ell, so that every layer, including the lowest, reads the full history,

ℛlf\(t,ℓ\)=\{\(t′,ℓ′\):t′<t,0≤ℓ′≤L\},\|ℛlf\|=Θ\(TL\),\\mathcal\{R\}\_\{\\mathrm\{lf\}\}\(t,\\ell\)\\;=\\;\\bigl\\\{\\,\(t^\{\\prime\},\\ell^\{\\prime\}\)\\;:\\;t^\{\\prime\}<t,\\;0\\leq\\ell^\{\\prime\}\\leq L\\,\\bigr\\\},\\qquad\\bigl\\lvert\\mathcal\{R\}\_\{\\mathrm\{lf\}\}\\bigr\\rvert=\\Theta\(TL\),\(7\)shown in Fig\.[1](https://arxiv.org/html/2608.08888#S1.F1)\(right\)\. In standard CoT each new token instead accesses only a partially processed view of the context\. The improved accessibility is also empirically verified in Sec\.[4\.4](https://arxiv.org/html/2608.08888#S4.SS4)\.

#### Latent feedback adds draft space\.

Latent feedback also supplies an implicit scratchpad, relieving the pressure to verbalize intermediate state\. State maintenance moves from the sequence axis alone to the depth axis as well: intermediate results can be updated throughzzalong the stack rather than only by extending the token sequence\. This predicts shorter rollouts on reasoning tasks, which Sec\.[4\.3](https://arxiv.org/html/2608.08888#S4.SS3)confirms\.

#### What latent feedback does not provide\.

We provide two important clarifications:

- •No mutable register\.RNNs and state\-space models overwrite a compressed state at each step\. Latent feedback is recurrent in form, but past states persist in the KV cache rather than being overwritten, so every earlier state stays directly readable by the current token\.
- •No added asymptotic depth at decoding time\.Latent feedback does not change the serial depth of decoding: with or without it, each step has a depth\-𝒪⁡\(L\)\\mathcal\{O\}\(L\)graph, soTTtokens cost𝒪⁡\(T​L\)\\mathcal\{O\}\(TL\)\. What changes is the*bandwidth*of the path, with a verbal channel and a continuous channel now evolving in parallel\. Note that a full\-bandwidth transformer can further increase the depth at prefilling time through a multipass prefill, which we will introduce in the following section\.

### 3\.3Parallel training for latent feedback decoding

Listing 1:Training: one step withkkpasses\.1defglu\_cross\(h,e\):

2return\(h@W\_u\)\*sigmoid\(e@W\_g\)

3

4e=embed\(tokens\)

5h=model\(e\)

6loss=ntp\_loss\(h\)

7for\_inrange\(k\-1\):

8x=glu\_cross\(shift\_right\(h\),e\)

9x=prefix\_mixin\(x,e\)

10h=model\(x\)

11loss\+=ntp\_loss\(h\)

Listing 2:Inference \(Soft\); uncommenting line 2 givesFused; line 7 shows theStandard\-decoding input\)\.1h=model\(embed\(prompt\)\)

2

3tok=sample\(lm\_head\(h\[\-1\]\)\)

4h\_prev=h\[\-1\]

5whilenotdone:

6x=glu\_cross\(h\_prev,embed\(tok\)\)

7

8h\_prev=model\_step\(x,kv\_cache\)

9tok=sample\(lm\_head\(h\_prev\)\)

Figure 2:Latent feedback in pseudo\-code\. Training \(left\) pays sequentiality acrosskkpasses, each parallel over positions\. Inference \(right\) differs from standard decoding by a single line \(line 6 vs\. the commented line 7\): the input is the fused state rather than the token embedding alone, reusing the state previously used for decoding\.Figure 3:A small fraction of three\-pass batches stabilizes long\-horizon latent feedback\.We test whether the learned feedback map extrapolates beyond the number of passes seen during training by repeatedly applying fused prefill passes\. A model trained with only single\- and two\-pass batches fails beyond its trained horizon: validation loss increases and the hidden\-state update size oscillates\. Adding a small fraction of three\-pass batches makes the iterates stable:\(left\)validation loss remains flat across many feedback steps, and\(right\)the hidden\-state change‖𝒉\(k\)−𝒉\(k−1\)‖\\\|\\bm\{h\}^\{\(k\)\}\-\\bm\{h\}^\{\(k\-1\)\}\\\|decays toward a small plateau\. This diagnostic uses repeated feedback passes as a proxy for the long\-horizon self\-composition encountered during latent\-feedback decoding\.At decoding time, latent feedback unrolls over generated positions\. Let𝒖t\\bm\{u\}\_\{t\}be the input actually fed to the transformer stack at positiontt\. The first position receives a plain token embedding, while each later position receives a fusion of the current token embedding and the previous top\-layer state:

𝒖1\\displaystyle\\bm\{u\}\_\{1\}=𝒆1,\\displaystyle=\\bm\{e\}\_\{1\},𝒉1\\displaystyle\\bm\{h\}\_\{1\}=fθ​\(𝒖1,C1\),\\displaystyle=f\_\{\\theta\}\(\\bm\{u\}\_\{1\};C\_\{1\}\),\(8\)𝒖t\\displaystyle\\bm\{u\}\_\{t\}=𝒆t⊗𝒉t−1,\\displaystyle=\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\},𝒉t\\displaystyle\\bm\{h\}\_\{t\}=fθ\(𝒖t;Ct\),t≥2\.\\displaystyle=f\_\{\\theta\}\(\\bm\{u\}\_\{t\};C\_\{t\}\),\\qquad t\\geq 2\.Here⋅⊗⋅\\cdot\\otimes\\cdotis the gated fusion of Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\), andCtC\_\{t\}is the KV cache over the previous inputs𝒖1:t−1\\bm\{u\}\_\{1:t\-1\}\. Thus the stack sees the input sequence

𝒆1,𝒆2⊗𝒉1,𝒆3⊗𝒉2,𝒆4⊗𝒉3,…\\bm\{e\}\_\{1\},\\ \\bm\{e\}\_\{2\}\\otimes\\bm\{h\}\_\{1\},\\ \\bm\{e\}\_\{3\}\\otimes\\bm\{h\}\_\{2\},\\ \\bm\{e\}\_\{4\}\\otimes\\bm\{h\}\_\{3\},\\ldotsrather than plain embeddings alone\. Since a standard next\-token\-prediction model is trained only on plain token embeddings in this slot, full\-bandwidth transformers must be trained on these latent\-feedback inputs as well\.

The exact recurrence of Eq\. \([8](https://arxiv.org/html/2608.08888#S3.E8)\) is sequential intt: the input at positionttdepends on the completed forward pass at positiont−1t\-1, so training on it directly would sacrifice the parallel teacher forcing that makes transformers efficient to pre\-train\. We instead adopt a multi\-forward\-pass approximation\. For each position in the sequence, we compute the top\-layer state several times, writing𝒉t\(k\)\\bm\{h\}\_\{t\}^\{\(k\)\}for the state at positiontton passkk\(the layer superscriptLLis omitted throughout this section\):

𝒉t\(1\)\\displaystyle\\bm\{h\}\_\{t\}^\{\(1\)\}=fθ​\(𝒆t,C\(1\)\),\\displaystyle=f\_\{\\theta\}\(\\bm\{e\}\_\{t\};\\,C^\{\(1\)\}\),C\(1\)\\displaystyle C^\{\(1\)\}=𝒆1,…,𝒆t−1,\\displaystyle=\\bm\{e\}\_\{1\},\\ldots,\\bm\{e\}\_\{t\-1\},\(9\)𝒉t\(2\)\\displaystyle\\bm\{h\}\_\{t\}^\{\(2\)\}=fθ​\(𝒆t⊗𝒉t−1\(1\),C\(2\)\),\\displaystyle=f\_\{\\theta\}\\\!\\big\(\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{\(1\)\};\\,C^\{\(2\)\}\\big\),C\(2\)\\displaystyle C^\{\(2\)\}=𝒆1,𝒆2⊗𝒉1\(1\),…,𝒆t−1⊗𝒉t−2\(1\),\\displaystyle=\\bm\{e\}\_\{1\},\\,\\bm\{e\}\_\{2\}\\otimes\\bm\{h\}\_\{1\}^\{\(1\)\},\\,\\ldots,\\,\\bm\{e\}\_\{t\-1\}\\otimes\\bm\{h\}\_\{t\-2\}^\{\(1\)\},\(10\)…\\displaystyle\\qquad\\ldots𝒉t\(k\)\\displaystyle\\bm\{h\}\_\{t\}^\{\(k\)\}=fθ​\(𝒆t⊗𝒉t−1\(k−1\),C\(k\)\),\\displaystyle=f\_\{\\theta\}\\\!\\big\(\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{\(k\-1\)\};\\,C^\{\(k\)\}\\big\),C\(k\)\\displaystyle C^\{\(k\)\}=𝒆1,𝒆2⊗𝒉1\(k−1\),…,𝒆t−1⊗𝒉t−2\(k−1\)\.\\displaystyle=\\bm\{e\}\_\{1\},\\,\\bm\{e\}\_\{2\}\\otimes\\bm\{h\}\_\{1\}^\{\(k\-1\)\},\\,\\ldots,\\,\\bm\{e\}\_\{t\-1\}\\otimes\\bm\{h\}\_\{t\-2\}^\{\(k\-1\)\}\.\(11\)The first pass is the ordinary no\-feedback forward pass \(𝒉t\(1\)≡𝒉t\\bm\{h\}\_\{t\}^\{\(1\)\}\\equiv\\bm\{h\}\_\{t\}\); each subsequent pass shifts the previous pass’s states one position rightward, fuses them with the token embeddings, and re\-runs the full stack in parallel across all positions, since every state it requires was completed in the previous pass\.

We then apply the standard teacher\-forced next\-token\-prediction loss111Other supervision on the output states, such as MTP\([16](https://arxiv.org/html/2608.08888#bib.bib10)\)/ JTP\([2](https://arxiv.org/html/2608.08888#bib.bib28)\)/ next\-latent predictions\([32](https://arxiv.org/html/2608.08888#bib.bib27)\), are compatible with this scheme and left to future work\.to the outputs of every pass\. Retaining the first\-pass loss preserves the model’s no\-feedback mode of operation, which is what processes the prompt at inference time\. We do not detach the gradient, so the loss from later passes backpropagates into earlier passes’ latent states, acting as an auxiliary objective; this does increase the memory footprint\. The overall objective is

ℒK\(θ\)=∑t=1T−logpθ\(xt\+1∣𝒆1:t\)⏟standard NTP objective\+λ1K−1∑k=2K∑t=1T−logpθ\(xt\+1∣𝒆1:t\(k\)\),\\mathcal\{L\}^\{K\}\(\\theta\)=\\underbrace\{\\sum\_\{t=1\}^\{T\}\-\\log p\_\{\\theta\}\\\!\\left\(x\_\{t\+1\}\\mid\\bm\{e\}\_\{1:t\}\\right\)\}\_\{\\text\{standard NTP objective\}\}\\;\+\\;\\lambda\\,\\frac\{1\}\{K\-1\}\\sum\_\{k=2\}^\{K\}\\sum\_\{t=1\}^\{T\}\-\\log p\_\{\\theta\}\\\!\\left\(x\_\{t\+1\}\\mid\\bm\{e\}\_\{1:t\}^\{\(k\)\}\\right\),\(12\)where𝒆1:t\(k\)=𝒆1,𝒆2⊗𝒉1\(k−1\),…,𝒆t⊗𝒉t−1\(k−1\)\\bm\{e\}\_\{1:t\}^\{\(k\)\}=\\bm\{e\}\_\{1\},\\,\\bm\{e\}\_\{2\}\\otimes\\bm\{h\}\_\{1\}^\{\(k\-1\)\},\\,\\ldots,\\,\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{\(k\-1\)\}are the pass\-kkfused inputs of Eqs\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\)–\([11](https://arxiv.org/html/2608.08888#S3.E11)\)\. In all experiments we setλ=1\\lambda=1without any tuning\.

A pseudo code is shown in Fig\.[2](https://arxiv.org/html/2608.08888#S3.F2)left\. We refer to this training scheme as*temporal parallelism*, following a common strategy for parallelizing recurrent computation during training\([37](https://arxiv.org/html/2608.08888#bib.bib16);[4](https://arxiv.org/html/2608.08888#bib.bib13);[21](https://arxiv.org/html/2608.08888#bib.bib14)\)\. Each pass is a Jacobi\-style update of the latent\-feedback recurrence: the hidden states from the previous pass are shifted one position to the right, fused with the token embeddings, and used to update all positions in parallel\. Each additional pass therefore advances latent feedback by one token\. Afterkkpasses, a top\-layer state from positionttcan affect the input at positions up tot\+k−1t\+k\-1, sokkpasses train the feedback transition over a horizon ofk−1k\-1token steps\. Training thus pays sequentiality across passes rather than across positions, reducing a length\-TTrecurrent unroll tokkparallel transformer evaluations, at roughlyk×k\\timesthe compute of standard teacher forcing\. The learned local transition is nevertheless the same one used during decoding, where latent feedback is applied causally once per generated token\.

#### Feedback\-pass scheduling\.

At decoding time the feedback loop unrolls indefinitely, so the trained map must remain stable under many more self\-compositions than any training budget can simulate; yet running many passes throughout training is prohibitively expensive, since each pass multiplies the cost of the run\. Scheduling the number of forward passes—how many, and when—is therefore central to making latent\-feedback training practical\.

*How many passes\.*We choose the number of passes by checking whether the iterated feedback map reaches a stable fixed point: a depth beyond which additional passes neither change the hidden states substantially nor improve the loss\. This stability is easier to obtain than in architectures that repeatedly recompute the entire input \(e\.g\. a loop transformer\), because each feedback pass keeps the token embedding fixed and updates only the hidden\-state pathway through the gate\. In practice, this means the goal is not to train at the full inference horizon, but to train the feedback map until it becomes stable under further self\-composition\.

*When to introduce feedback passes\.*Because feedback passes are expensive, most of training uses the standard single\-pass objective\. We introduce latent feedback progressively in the middle of training: first with two\-pass batches, and later with a small fraction of batches with more passes\. This lets the run begin from an ordinary pretrained checkpoint, spend the bulk of its compute on standard teacher forcing, and pay the extra feedback\-pass cost only mid\-training, when it is needed to stabilize the feedback map\.

Fig\.[3](https://arxiv.org/html/2608.08888#S3.F3)illustrates the feasibility of the scheduling\. We studied a 1B model trained on 200B tokens\. A model trained with only single\- and two\-pass batches \(75% single\-pass, 25% two\-pass; green\) performs well at the trained depth but fails to extrapolate: beyond that depth, validation loss rises sharply and the hidden\-state change‖𝒉\(k\)−𝒉\(k−1\)‖\\\|\\bm\{h\}^\{\(k\)\}\-\\bm\{h\}^\{\(k\-1\)\}\\\|oscillates rather than decays, indicating that the iterates have left the trained state distribution\. Adding only 3% three\-pass batches \(75% single\-pass, 22% two\-pass, 3% three\-pass; blue\) qualitatively changes the behavior: validation loss remains flat through3030feedback steps, and the hidden\-state change decays to a small plateau\. This suggests that the learned feedback map behaves like a contraction toward a fixed point, making feedback depths far beyond those seen in training stable in our tests\. The same extrapolation behavior carries over to inference: hundred\-token rollouts show no sign of breakdown \(Fig\.[5](https://arxiv.org/html/2608.08888#S4.F5), solid green line\), and we observe similar stability underk=1000k=1000feedback passes \(Fig\.[10](https://arxiv.org/html/2608.08888#A5.F10)in the appendix\)\.

#### Prefix mixin\.

A distribution mismatch remains between multi\-pass training and inference\. At decoding time a sequence is heterogeneous: prompt positions carry plain token embeddings \(processed by a single prefill pass\), while generated positions carry fused inputs\. In the passes of Eqs\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\)–\([11](https://arxiv.org/html/2608.08888#S3.E11)\), by contrast,*every*position beyond the first is fused\. A model trained only on fully\-fused passes therefore encounters an out\-of\-distribution boundary at inference, precisely where the prompt ends and generation begins\. To close this gap we apply a*prefix mixin*: in each pass beyond the first, we sample a random prefix lengthppand revert positionst≤pt\\leq pto plain embeddings, fusing only the suffix\. Training thus covers sequences that switch from plain to fused inputs at an arbitrary point, i\.e\. the structure of single\-prefill inference\. Alternatively, the prompt itself can be run through a second, fused prefill pass so that all positions match the fused distribution; the mixin removes the need for this, but we support both, corresponding to the “identical or doubled prefill” overhead stated in the abstract\.

#### Stability recipes for long feedback horizons\.

At inference time, latent feedback may be applied for hundreds or thousands of generated tokens, far beyond the few feedback passes used during training\. We therefore use several lightweight stabilization techniques to keep the feedback map well behaved under long self\-composition\.

- •Stationary hidden\-state scale\.We keep the magnitude of carried state𝒉tL\\bm\{h\}\_\{t\}^\{L\}bounded as feedback is repeatedly applied\. To prevent the top\-layer state norm from growing with depth, we use depth scaling\([34](https://arxiv.org/html/2608.08888#bib.bib29);[30](https://arxiv.org/html/2608.08888#bib.bib30)\)so that‖𝒉tL‖∼𝒪⁡\(1\)\\left\\lVert\\bm\{h\}\_\{t\}^\{L\}\\right\\rVert\\sim\\mathcal\{O\}\(1\)rather than𝒪⁡\(L\)\\mathcal\{O\}\(L\), as can occur in a standard pre\-norm model\. We also apply RMSNorm to the fused input𝒆t⊗𝒉t−1L\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{L\}before feeding it into the model\.
- •Shared input basis with weight tying\.The model processes two types of inputs: plain token embeddings during standard prefill, and fused hidden\-state/token inputs during latent\-feedback decoding\. We therefore encourage the embedding space and top\-layer hidden\-state space to remain in a compatible basis by tying the weights of the embedding layer and readout layer, reducing the burden on the fusion weights to learn a large corrective rotation between the two input distributions\.
- •Noise regularization\.During training, we add small jitter noise to the carried hidden state before fusion, 𝒉tL=fθ​\(𝒆t⊗𝒉t−1L\+ϵ,C\),ϵ∼Uniform​\[−σ,σ\]D\.\\bm\{h\}\_\{t\}^\{L\}=f\_\{\\theta\}\\\!\\left\(\\bm\{e\}\_\{t\}\\otimes\\bm\{h\}\_\{t\-1\}^\{L\}\+\\epsilon;C\\right\),\\qquad\\epsilon\\sim\\mathrm\{Uniform\}\[\-\\sigma,\\sigma\]^\{D\}\.\(13\)This exposes the feedback map to a local neighborhood around each training state, making it less sensitive to small deviations that can accumulate over long feedback horizons\.

The complete pseudo code for training where the tricks are adopted is presented in Fig\.[9](https://arxiv.org/html/2608.08888#A3.F9)in the appendix\.

### 3\.4Latent\-feedback training improves pre\-training data efficiency

Beyond enabling latent feedback at decoding time, the feedback passes also act as an auxiliary training signal on the hidden states\. In the standard next\-token\-prediction loss, the top\-layer state𝒉tL\\bm\{h\}\_\{t\}^\{L\}is supervised only through the prediction of the next token\. In later feedback passes, however,𝒉tL\\bm\{h\}\_\{t\}^\{L\}is shifted, fused into the input of subsequent positions, and can influence losses at multiple future positions through causal attention\. Thus gradients from later predictions backpropagate into earlier hidden states, encouraging them to be reusable as inputs rather than merely predictive at the output layer\.

Empirically, this improves pre\-training data efficiency even when latent feedback is not used at decoding time\. When evaluated with standard decoding, models trained with the latent\-feedback objective improve on LM Eval and free\-form generation tasks relative to comparable models trained only with the ordinary next\-token objective\. We can therefore view latent\-feedback training as a way to spend additional training\-time compute on the same token stream, improving the representations without changing the serving\-time decoding pipeline\.

Latent\-feedback training also enables a simple form of prefill\-time test\-time scaling\. At evaluation, we can applykkadditional fused passes over the prompt using Eqs\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\)–\([11](https://arxiv.org/html/2608.08888#S3.E11)\)\. These passes refine the prompt states before generation begins, improving perplexity and downstream accuracy at the cost ofkkextra parallel prefill forward passes\. See Sec\.[4\.1](https://arxiv.org/html/2608.08888#S4.SS1)\.

## 4Experiments

To evaluate full\-bandwidth transformers, we pretrain 1B\-parameter models \(Appendix[A](https://arxiv.org/html/2608.08888#A1)\) using the latent\-feedback training recipe from Sec\.[3\.3](https://arxiv.org/html/2608.08888#S3.SS3)\. We use NorMuon\([25](https://arxiv.org/html/2608.08888#bib.bib11)\)for matrix parameters with learning rate1×10−21\\times 10^\{\-2\}and weight decay0\.010\.01, and Adam for all other parameters with learning rate5×10−45\\times 10^\{\-4\}and no weight decay\. All runs use a WSD learning\-rate schedule\([18](https://arxiv.org/html/2608.08888#bib.bib32);[20](https://arxiv.org/html/2608.08888#bib.bib33)\)with 200 warmup steps and a 25% cooldown phase decaying to zero\. During cooldown, we add a z\-loss\([6](https://arxiv.org/html/2608.08888#bib.bib34)\)with coefficient1×10−51\\times 10^\{\-5\}and decay weight decay together with the learning rate following AdamC\([10](https://arxiv.org/html/2608.08888#bib.bib31)\), which helps prevent weight and gradient norms from becoming unstable\. For all experiments we use a jitter noise withσ=0\.02\\sigma=0\.02\(Eq\. \([13](https://arxiv.org/html/2608.08888#S3.E13)\)\) during training\.

Models are trained on the same data mixture as Phi\-4\([1](https://arxiv.org/html/2608.08888#bib.bib35)\), with context length 8192\. Unless otherwise stated, we use a global batch size of 300K tokens; the 1T\-token no\-feedback baseline uses a larger global batch size of 1\.2M tokens\. For latent\-feedback runs, we report both the number of training tokens and the*token\-equivalent compute*, defined as training tokens multiplied by the average number of forward passes per batch\. Under this accounting, a two\-pass batch costs2×2\\timesstandard teacher forcing and a three\-pass batch costs3×3\\times\.

RunFeedback\-pass mixtureTokensToken\-equivalent compute10B100% three\-pass10B40B100B75% one\-pass, 25% three\-pass100B150B200B75% one\-pass, 22% two\-pass, 3% three\-pass200B256B400B75% one\-pass, 22% two\-pass, 3% three\-pass400B512B
### 4\.1Fused prefilling improves non\-generative performance

Figure 4:Feedback passes during prefilling improve non\-generative performance\.Re\-running prefill with additional fused passes \(Eqs\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\)–\([11](https://arxiv.org/html/2608.08888#S3.E11)\)\) improves both validation perplexity \(left\) and 5\-shot LM Eval accuracy across 10 tasks \(right\) across training scales; most of the gain arrives at the first recurrence step\. Error bars denote one standard error of the mean, obtained by propagating individual per\-task standard errors \(summing in quadrature and dividing by the number of tasks\)\.Fig\.[4](https://arxiv.org/html/2608.08888#S4.F4)plots validation loss and average 5\-shot LM Eval accuracy across RTE, TruthfulQA\-MC2, ARC\-Easy, ARC\-Challenge, BoolQ, PIQA, WinoGrande, OpenBookQA, COPA, and MMLU, as a function of the number of feedback passes applied during prefill\. Step 0 is ordinary prefill with no latent feedback, corresponding to Eq\. \([9](https://arxiv.org/html/2608.08888#S3.E9)\)\. Each additional step re\-runs the stack on fused inputs from Eqs\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\)–\([11](https://arxiv.org/html/2608.08888#S3.E11)\), feeding the previous pass’s top\-layer states back through the gate\. Three findings stand out\.

First,*the gain is front\-loaded*\. Most of the improvement appears after the first fused prefill pass, the first pass in which top\-layer hidden states are made available at the input\. Further passes continue to help, but with diminishing returns\. This is consistent with latent feedback acting as added effective depth for the prompt, with the largest gain arriving once the full\-stack state is exposed to layer 0\.

Second,*latent\-feedback training costs little when unused*\. At step 0, where the model is evaluated as an ordinary transformer with no feedback, the latent\-feedback model gives up only a small amount of validation loss relative to the standard baseline, while already improving average LM Eval accuracy\. Thus, the training recipe is useful even for deployments that do not apply fused prefill passes at inference time\.

Third,*a small amount of prefill\-time compute matches substantially larger standard baselines*\. With two feedback passes, the 100B\-token full\-bandwidth transformer reaches the 200B\-token standard baseline, and the 200B\-token full\-bandwidth transformer reaches the 400B\-token standard baseline\. In this regime, fused prefilling converts modest inference\-time compute into roughly2×2\\timespretraining data efficiency\.

Lastly, we compare our model with other models of similar parameter scale on 0\-shot LM Eval performance are shown in Table[2](https://arxiv.org/html/2608.08888#A2.T2)in Appendix[B](https://arxiv.org/html/2608.08888#A2), where we found our model performs on\-par or better than models trained under similar or more budget\. These results imply that full feedback transformers improve on a strong baseline\.

### 4\.2Latent feedback decoding improves decoding performance

Figure 5:We compare the three decoding regimes defined at the start of Sec\.[4\.2](https://arxiv.org/html/2608.08888#S4.SS2):Standard,Soft, andFused\. on free\-form generation tasks; On math tasks,Softtypically gives the largest gains, suggesting that carrying hidden state through generation helps reasoning\. On coding tasks,Fusedis often strongest, suggesting that refining the prompt representation before generation is especially useful\. Coding results report Pass@3 from 10 rollouts per problem, with temperature selected from\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}separately for each method\.We now evaluate whether latent feedback improves open\-ended generation\. We compare three decoding regimes:

- •Standard: single\-pass prefill; generation uses token embeddings only\. This evaluates the full\-bandwidth model as an ordinary transformer, and measures the cost of latent\-feedback training when the feedback channel is not used at inference\.
- •Soft: single\-pass prefill; generation uses latent feedback as in Eq\. \([3](https://arxiv.org/html/2608.08888#S3.E3)\)\. Prompt positions carry plain embeddings, while generated positions carry fused inputs, matching the heterogeneous prompt\-then\-generation regime induced by prefix mixin in Sec\.[3\.3](https://arxiv.org/html/2608.08888#S3.SS3)\. The only per\-token overhead is twoD×DD\\times Dmatrix multiplications\.
- •Fused: the prompt is first processed by an additional fused prefill pass, as in Eq\. \([10](https://arxiv.org/html/2608.08888#S3.E10)\); generation then proceeds as inSoft\. This gives the prompt states one round of latent\-feedback refinement before decoding begins, at the cost of one extra prefill pass that is parallel over prompt tokens\.

ThusStandardandSofthave identical prefill cost, whileFuseddoubles prefill cost while keeping the same per\-token decoding cost asSoftand effectivelyStandard\.

#### Evaluation setting\.

We evaluate on GSM8K\([7](https://arxiv.org/html/2608.08888#bib.bib23)\), MATH\-500\([26](https://arxiv.org/html/2608.08888#bib.bib24)\), HumanEval\([5](https://arxiv.org/html/2608.08888#bib.bib25)\), and MBPP\([3](https://arxiv.org/html/2608.08888#bib.bib26)\)\. We report Pass@1 for math and Pass@3 for coding\. For coding, Pass@3 is estimated from 10 rollouts per problem, with temperature grid\-searched over\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}separately for each decoding regime\. We do not use top\-kkor top\-ppsampling\.

#### Latent feedback decoding improves the base model

Fig\.[5](https://arxiv.org/html/2608.08888#S4.F5)evaluates the three decoding regimes on base models at two recurrence\-training scales \(100B\-400B tokens, solid lines\), against no\-recurrence baselines trained on 100B–1T tokens \(dashed lines\)\. Four observations\. First,Softimproves overStandardon every task at both scales; the gains come from decoding alone, with model weights held fixed\. Second, the preferred regime is task\-dependent:Softyields the largest gains on math \(on Math500 the 200B model improves from0\.270\.27to0\.370\.37, surpassing even the 1T no\-recurrence baseline\), whileFusedis strongest on coding \(HumanEval0\.31→0\.340\.31\\to 0\.34; MBPP0\.38→0\.400\.38\\to 0\.40at 200B\), consistent with coding rewarding a deeper representation of the prompt and math rewarding state carried through generation\. Third, under latent feedback the 200B recurrent model approaches or exceeds no\-recurrence baselines trained with22–5×5\\timesthe tokens \(e\.g\., near the 1T baseline on GSM8K and HumanEval\)\. Fourth, Pass@3 improves alongside Pass@1, indicating that conditioning generation on hidden states does not collapse sampling diversity or hurt exploration\.

#### The improvement carries over through instruction tuning\.

We further apply long\-context extension \(12B tokens\) from 8K to 32K and instruction tuning \(6B tokens\) for the 200B and 400B model \(green and purple lines in Fig\.[4](https://arxiv.org/html/2608.08888#S4.F4)and[5](https://arxiv.org/html/2608.08888#S4.F5)\), then evaluate without few\-shot examples\. Because these stages are much shorter than pretraining, we train them with*three*forward passes throughout rather than using the pretraining feedback\-pass schedule\. Results are shown in Table[1](https://arxiv.org/html/2608.08888#S4.T1)\. BothSoftandFusedcontinue to improve overStandardacross all four tasks; for example, GSM8K improves from64\.564\.5to67\.967\.9, and HumanEval from42\.542\.5to45\.945\.9\. They also outperform the matched 200B\-token standard baseline on every task\. On MBPP,Fusedcloses most of the remaining gap to the 1T\-token standard baseline \(41\.241\.2vs\.41\.941\.9\)

Full\-bandwidth, 200BFull\-bandwidth, 400BStandard transformerTaskStandardSoftFusedStandardSoftFused200B400B1TGSM8K \(Pass@1\)64\.5267\.9367\.5567\.9071\.0071\.8062\.9368\.3970\.13MATH\-500 \(Pass@1\)43\.8045\.6045\.6046\.0045\.4048\.4042\.4046\.4047\.40HumanEval \(Pass@3\)42\.5445\.0645\.9246\.5047\.2047\.6037\.1644\.8550\.01MBPP \(Pass@3\)38\.3939\.8041\.2240\.5040\.6041\.7038\.6140\.2841\.93Table 1:Latent\-feedback gains carry over through instruction tuning\.We evaluate full\-bandwidth transformers after long\-context extension and instruction tuning, using no few\-shot examples\. Scores are percentages\. For math tasks, we report Pass@1; for coding tasks, we report Pass@3 estimated from 10 rollouts per problem, selecting the best temperature from\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}for each setting\. Bold indicates the best decoding regime within each full\-bandwidth training scale\.Figure 6:Reasoning length and accuracy on Math500 from the 200B run \(green line in Fig\.[5](https://arxiv.org/html/2608.08888#S4.F5)\)\. Base model without any few shot examples or instruction tuning generates shorter solution \(measured by median rather than mean to prevent outliers\) while giving better accuracy, a concrete example is provided in Fig\.[8](https://arxiv.org/html/2608.08888#S4.F8)\.\(\(a\)\)State retrieval\.Each sequence specifies either a binary relation between two counters \(“Completion”\) or a stored absolute binary value \(“Memory”\), followed by varying number \(denoted by line color\) of label\-independent distraction tokens\. One recurrent step makes the target state nearly perfectly decodable at layer00across input lengths, whereas standard prefilling requires multiple layers to reconstruct it from the perfix\.\(\(b\)\)Multi\-register latest\-write tracking\.Each sequence performs 2,4, or 8 writes to each of eight binary registers and then queries one register’s latest value\. We probe its value at each residual depth\. Recurrent prefilling improves shallow standard accessibility over standard prefilling; One recurrent step’s gain diminishes in deeper layers and in inputs with more overwriting, where full recurrent performs the best, showing the benefit of maintaining state throughout the sequence\.
Figure 7:Full\-bandwidth transformer exposes global state to shallow layers\.On three synthetic tasks, we linearly probe the final input token’s residual stream across depth \(0 denotes input\) to predict a binary state of the input \(See Appendix[F](https://arxiv.org/html/2608.08888#A6)for details\); We compared standard prefilling, which uses token embedding as inputs, with recurrent prefilling, where the preceding token’s top\-layer state is fused into the current token’s input, similar to Eq\. \([8](https://arxiv.org/html/2608.08888#S3.E8)\) but uses input token rather than sampled token;kk\-step recurrence applies this fusion over the finalkktokens \(at the cost ofk\+1k\+1forward pass\), while full recurrence applies it throughout the task sequence \(at the cost of fully sequential prefill\)\.

### 4\.3Latent feedback enables more concise reasoning

On the base model,Softdecoding often produces markedly shorter reasoning traces thanStandardat equal or better accuracy; Fig\.[8](https://arxiv.org/html/2608.08888#S4.F8)shows examples \(other examples are shown in Appendix[G](https://arxiv.org/html/2608.08888#A7)\)\. This is the behavior the widened channel predicts: intermediate computation thatStandardmust verbalize—token by token, atlog2⁡\|V\|\\log\_\{2\}\|V\|bits per step—can instead ride the hidden state, so fewer tokens are needed to reach the answer\. Notably, the effect disappears after instruction tuning\. We attribute this to the tuning data being off\-policy with respect to latent\-feedback decoding: the target traces were produced by \(and imitate the verbosity of\) standard token\-by\-token reasoning, so fitting them re\-imposes the fully verbalized style regardless of what the state can carry\. On\-policy post\-training under latent feedback may preserve the conciseness, which we leave to future work

### 4\.4Full\-bandwidth transformer carries richer information in shallow\-layer residuals

Lastly, to verify the added bandwidth directly, we run controlled state\-tracking experiments in which the target is fixed but the intervening context varies \(full construction in App\.[F](https://arxiv.org/html/2608.08888#A6)\)\. Two tasks isolate the effect\.*Completion tracking*asks whether a completed counter has reached a required one after a run of no\-op updates;*delayed memory*asks the model to recover an initial binary state after a sequence of label\-independent scratch operations\. Both end at a shared colon, and the label is determined entirely by information before it, so a probe at that colon measures how much of the global state each layer has already reconstructed\.

We compare two prefilling regimes\. Under*standard prefilling*, the final token enters as its plain embedding; under*one\-step recurrent prefilling*, that embedding is fused with the preceding token’s top\-layer state \(Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\)\), exactly the layer\-0 input latent feedback supplies at decode time\. We then fit a linear probe for the target \(done/moreorzero/one\) at each residual\-stream depth\.

The two regimes differ sharply at the bottom of the stack\. Under standard prefilling, a shallow residual can read only the layer\-matched, partially processed prefix \(the reachability constraint of Eq\. \([2](https://arxiv.org/html/2608.08888#S2.E2)\)\), so reconstructing the global state takes several layers of further computation; the layer\-0 probe is near chance\. Recurrent prefilling instead exposes a fully processed prefix summary at the layer\-0 input, and layer\-0 probe accuracy rises to99\.6%99\.6\\%for completion tracking and100%100\\%for delayed memory\. Recurrence thus provides a high\-bandwidth shortcut that transports globally aggregated information into shallow computation, the mechanism the full\-bandwidth view predicts\.

One caveat bears emphasis: improved*decodability*does not by itself imply improved*output*\. That a target is linearly recoverable at layer 0 shows the information is present, not that the model uses it to decide the next token; making state available and causally exploiting it are distinct, and only the downstream task results \(Sec\.[4](https://arxiv.org/html/2608.08888#S4)\) speak to the latter\.

ProblemWhat is the last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}?

Standard decoding\(200B model\)To determine the last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}, we start by performing the division of 137 by 500\.First, we set up the division:137÷500137\\div 500Since 137 is less than 500, the decimal expansion will start with a zero\.\[…326 tokens omitted…\]Since the remainder is now00, the decimal expansion of137500\\frac\{137\}\{500\}is:0\.2740\.274The last nonzero digit to the right of the decimal point is4\\boxed\{4\}\.

Soft decoding\(200B model\)To find the last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}, we can perform the division and observe the pattern of the decimal expansion\.137500=0\.274\\frac\{137\}\{500\}=0\.274The last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}is 4\.

Figure 8:Qualitative comparison of outputs from the 200B model under standard and soft decoding \(greedy decoding\)\. Both reach the correct answer; soft decoding is substantially more concise\. Truncated text is indicated by \[…\]\. This is no longer observable on instruction\-tuned version\.

## 5Related work

#### Alleviating the depth bottleneck at decoding time\.

One central idea behind full bandwidth transformer is to introduce extra compute that overlaps with the sequential decoding process\. There are other works that consider similar ideas\.*Feedback Transformer*\([12](https://arxiv.org/html/2608.08888#bib.bib4)\)is the pioneering work along this line; At each position, they generate a mixture of each layer’s representation and let attention in future positions attend to aggregated representation rather than the same\-layer key values as in standard transformers\. However, their training is sequential over input tokens, limiting their scalability, whereas our training is parallelized over all positions\. Additionally, our approach does not involve modifying the structure but only the input\. Note that their ablations also support our choice of feedback layer: a memory built from the topmost layer alone nearly matches the full\-layer mixture, while one built from the first layer performs no better than a standard transformer\. There are also very recent works exploring a similar direction\.*T2T^\{2\}MLR*\([4](https://arxiv.org/html/2608.08888#bib.bib13)\)injects the representations at a late middle layer in the last position with the representation at an early middle layer in the current position\.*Latent Recurrent Transformer*\([21](https://arxiv.org/html/2608.08888#bib.bib14)\)stores a hidden state from a fixed source layer at the previous position and injects it into the current position through the attention via an extra key/value projections and directly into the residual stream\. Methodology\-wise, our approach is similar in the training approach and the motivation\. Our approach mainly differs in the point of reinjection; specifically, our injection happens “externally” to the model and therefore introduces no architecture changes since we only modify the construction of the input\. We also introduce the least amount of extra parameters\. For aLL\-layer transformer withDD\-dimension residual, we introduce only two linear projection \(each of sizeD×DD\\times D\), in contrast toT2T^\{2\}MLR’s extra MLPs \(5​D25D^\{2\}parameters\) and LRT’s layerwise projection which introducesL​D2LD^\{2\}parameters\. The bigger and major difference lies in the scope of empirical evaluation: our work performs much larger scale pre\-training \(up to 400B tokens\), with recurrence scheduling; therefore, we manage to empirically verify the actual inference time improvement on different workloads, whereas LRT only considers non freeform eval \(similar to our setting in Fig\.[4](https://arxiv.org/html/2608.08888#S4.F4)right\), andT2T^\{2\}MLR considers synthetic state tracking tasks and gsm8k only after fine\-tuning the model on the math corpus\. However, considering the similarity in spirit, we do not foresee a reason why the performance one approach would differ significantly from the others and exactly which method \(and more broadly, which form of past hidden state injection\) gives the best performance at large scale remains unclear since we do not have the resources for verification\.

#### Latent and continuous reasoning\.

Our approach feeds top layer latent into the context, similar to the central idea of latent reasoning approaches such as Coconut\([19](https://arxiv.org/html/2608.08888#bib.bib20)\)and Soft Thinking\([39](https://arxiv.org/html/2608.08888#bib.bib22)\)\. The biggest differences are : \(a\) We focus on pre\-training; \(b\) We use the hidden state to “augment” the generation rather than replacing the discrete tokens, therefore our approach is easier to supervise \(but we may be less token efficient\)\. Hybrid Latent Reasoning via Reinforcement Learning\([36](https://arxiv.org/html/2608.08888#bib.bib21)\)proposes to use both the top layer hidden state and the generated tokens’ embedding at post\-training time during rollout, however they did not utilize top layer hidden state but instead they use it to generate a weighted mixture of vocabulary embedding so it is unclear whether it improves the reachability as the full bandwidth transformer does\. There are also works studying latent reasoning at pre\-training time, in particular, PonderLM\-2\([37](https://arxiv.org/html/2608.08888#bib.bib16)\)considers an interleaved embedding / hidden state as the input\. Notably, their training approach is similar to us in that they use multiple forward passes to replace sequential rollout, however their approach doubles the input length \(as well as KV cache size\) so they introduce more training and inference overhead than the full bandwidth transformer\.

#### Parallel training of recurrent networks\.

Another related direction is parallel training of recurrent networks\. Most applications of this consider the linear special case like Mamba\([17](https://arxiv.org/html/2608.08888#bib.bib1)\)or Gated Deltanet\([35](https://arxiv.org/html/2608.08888#bib.bib2)\)\. These are clearly powerful techniques with use in various architectures yet in all such uses they are hybridized with standard transformer layers which can compensate for the missing representational capacity inherited from the linear constraint\. ParaRNN\([9](https://arxiv.org/html/2608.08888#bib.bib3)\)goes further by parallelizing training of nonlinear recurrent neural networks via decoupling the optimizations at each point in the process and using newton’s iterations to achieve convergence with results comparable to transformers for language modeling\. This approach here goes the other way, constructing recurrence on transformers with results that improve over baseline transformers, and it appears that the approach here is significantly more efficient\.

#### Data\-efficient pre\-training\.

Lastly, our work falls into the broad category of improving LLM pre\-training’s data efficiency, i\.e\., given the same model size and fixed data, how can we use more flops to build a more powerful model under fixed or more inference overhead\. Existing approaches consider additional objectives \(beyond NTP\) on the representation\([27](https://arxiv.org/html/2608.08888#bib.bib5);[38](https://arxiv.org/html/2608.08888#bib.bib6);[8](https://arxiv.org/html/2608.08888#bib.bib8);[32](https://arxiv.org/html/2608.08888#bib.bib27)\)that encourage the hidden state to contain richer information\. There has also been a recent NanoGPT slow run competition222[https://qlabs\.sh/slowrun/](https://qlabs.sh/slowrun/)that studies this setting, where the official solution\([28](https://arxiv.org/html/2608.08888#bib.bib7)\)trains a deep ensemble of LLMs and distills them into a single one for better performance\. Compared with these approaches, our framework uses additional training flops for unlocking a new type of decoding regime that gives a free performance boost at inference time\. Additionally, we believe techniques can flow between literature, for example, the depth scaling we used has also been shown to be important for the stability of training loop transformers\([29](https://arxiv.org/html/2608.08888#bib.bib15)\)\. Our empirical verification of recurrence scheduling also suggests the feasibility of introducing computationally intensive auxiliary objectives only later on in the training\.

*Loop transformers*\([13](https://arxiv.org/html/2608.08888#bib.bib17);[11](https://arxiv.org/html/2608.08888#bib.bib18);[15](https://arxiv.org/html/2608.08888#bib.bib19);[14](https://arxiv.org/html/2608.08888#bib.bib38)\)also fall into this category, where additional training FLOPs enable test\-time scaling through repeated computation at inference\. Our approach is similar to loop transformers during training in that the model’s outputs are repeatedly fed back as inputs across multiple forward passes\. At inference time, however, the two approaches differ in where the additional computation is paid\. Loop transformers obtain additional effective depth by explicitly reapplying the transformer stack, thereby increasing inference compute with the number of recurrent steps\. In contrast, latent feedback is integrated into the autoregressive decoding loop: it reuses the top\-layer state already produced at the previous token and requires only a lightweight fusion operation, without additional transformer\-block evaluations per generated token\. Thus, full\-bandwidth transformers retain much of the benefit of recurrent computation while incurring negligible per\-token decoding overhead, with additional compute required only when optional multi\-pass prefilling is used\.

More broadly, these methods point to a shift in the relevant scaling axes for pre\-training\. Conventional scaling primarily varies model parameters and training tokens\. However, in large\-scale training, the feasible design space is also constrained by pod size of GPUs, wall\-clock budget, and the availability of high\-quality unique tokens\. Once the token\-per\-parameter ratio and the accessible pool of high\-quality data become binding, simply increasing the number of unique training tokens is no longer the only, or even the most direct, path to improvement\. A promising axis is to spend more computation per unique token through recurrent, iterative, or feedback\-based mechanisms\.

## 6Limitation

There are two major limitations of the current work\. Firstly our experiment scale is limited to 1B parameter models, and we did not verify the approach on models of larger scale\. However we believe latent feedback decoding can potentially introduce more benefit for a deeper model where the top layer hidden state includes even richer information\. Secondly, the feedback pass scheduling is based on a heuristic; future work can consider more rigorous ablation on the length of the recurrence training phase as well as more principled approach to determine the number of recurrence steps, e\.g\. via the Jacobi iteration convergence diagnostics from[37](https://arxiv.org/html/2608.08888#bib.bib16)\.

## References

- Abdinet al\.\(2024\)M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p2.1)\.
- Ahnet al\.\(2025\)K\. Ahn, A\. Lamb, and J\. LangfordEfficient joint prediction of multiple future tokens\.arXiv preprint arXiv:2503\.21801\.Cited by:[footnote 1](https://arxiv.org/html/2608.08888#footnote1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.2](https://arxiv.org/html/2608.08888#S4.SS2.SSS0.Px1.p1.1)\.
- Caiet al\.\(2026\)Z\. Cai, X\. Zhu, Y\. Dong, Y\. He, and S\. AroraTˆ 2mlr: transformer with temporal middle\-layer recurrence\.arXiv preprint arXiv:2607\.15178\.Cited by:[§3\.3](https://arxiv.org/html/2608.08888#S3.SS3.p4.1),[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374Cited by:[§4\.2](https://arxiv.org/html/2608.08888#S4.SS2.SSS0.Px1.p1.1)\.
- Chowdheryet al\.\(2023\)A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.Palm: scaling language modeling with pathways\.Journal of machine learning research24\(240\),pp\. 1–113\.Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.2](https://arxiv.org/html/2608.08888#S4.SS2.SSS0.Px1.p1.1)\.
- Daiet al\.\(2025\)B\. Dai, Y\. Liu, D\. Xue, Y\. Song, Q\. Guo, K\. Chen, X\. Wang, B\. Zhou, and Z\. LinContext\-level language modeling by learning predictive context embeddings\.arXiv preprint arXiv:2510\.20280\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1)\.
- Danieliet al\.\(2025\)F\. Danieli, P\. Rodriguez, M\. Sarabia, X\. Suau, and L\. ZappellaPararnn: unlocking parallel training of nonlinear rnns for large language models\.arXiv preprint arXiv:2510\.21450\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px3.p1.1)\.
- Defazio \(2025\)A\. DefazioWhy gradients rapidly increase near the end of training\.arXiv preprint arXiv:2506\.02285\.Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p1.1)\.
- Dehghaniet al\.\(2018\)M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and Ł\. KaiserUniversal transformers\.arXiv preprint arXiv:1807\.03819\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p2.1)\.
- Fanet al\.\(2020\)A\. Fan, T\. Lavril, E\. Grave, A\. Joulin, and S\. SukhbaatarAddressing some limitations of transformers with feedback memory\.arXiv preprint arXiv:2002\.09402\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px1.p1.1)\.
- Fanet al\.\(2026\)Y\. Fan, A\. Svete, and K\. LeeBridging the gap between latent and explicit reasoning with looped transformers\.arXiv preprint arXiv:2606\.31779\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p2.1)\.
- Geipinget al\.\(2025\)J\. Geiping, S\. M\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. GoldsteinScaling up test\-time compute with latent reasoning: a recurrent depth approach\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=S3GhJooWIC)Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p2.1)\.
- Giannouet al\.\(2023\)A\. Giannou, S\. Rajput, J\. Sohn, K\. Lee, J\. D\. Lee, and D\. PapailiopoulosLooped transformers as programmable computers\.InInternational Conference on Machine Learning,pp\. 11398–11442\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p2.1)\.
- Gloeckleet al\.\(2024\)F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. SynnaeveBetter & faster large language models via multi\-token prediction\.arXiv preprint arXiv:2404\.19737\.Cited by:[Appendix D](https://arxiv.org/html/2608.08888#A4.p1.1),[footnote 1](https://arxiv.org/html/2608.08888#footnote1)\.
- Gu and Dao \(2024\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.InFirst conference on language modeling,Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px3.p1.1)\.
- Hägeleet al\.\(2024\)A\. Hägele, E\. Bakouch, A\. Kosson, L\. B\. Allal, L\. Von Werra, and M\. JaggiScaling laws and compute\-optimal training beyond fixed training durations\.Advances in Neural Information Processing Systems37,pp\. 76232–76264\.Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p1.1)\.
- Haoet al\.\(2024\)S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. TianTraining large language models to reason in a continuous latent space\.arXiv preprint arXiv:2412\.06769\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2024\)S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao,et al\.Minicpm: unveiling the potential of small language models with scalable training strategies\.arXiv preprint arXiv:2404\.06395\.Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p1.1)\.
- Huanget al\.\(2026\)Z\. Huang, X\. He, L\. Ren, Y\. Wang, B\. Peng, H\. Cheng, S\. Wang, P\. He, J\. Gao, Y\. J\. Lee,et al\.Latent recurrent transformer: architecture exploration, training strategies, and scaling behavior\.arXiv preprint arXiv:2605\.26797\.Cited by:[§3\.3](https://arxiv.org/html/2608.08888#S3.SS3.p4.1),[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2608.08888#S1.p1.1)\.
- Liet al\.\(2024a\)Y\. Li, F\. Wei, C\. Zhang, and H\. ZhangEagle: speculative sampling requires rethinking feature uncertainty\.arXiv preprint arXiv:2401\.15077\.Cited by:[Appendix D](https://arxiv.org/html/2608.08888#A4.p1.1)\.
- Liet al\.\(2024b\)Z\. Li, H\. Liu, D\. Zhou, and T\. MaChain of thought empowers transformers to solve inherently serial problems\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 11911–11943\.Cited by:[§1](https://arxiv.org/html/2608.08888#S1.p2.1)\.
- Liet al\.\(2026\)Z\. Li, L\. Liu, C\. Liang, W\. Chen, and T\. ZhaoNorMuon: making muon more efficient and scalable\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=m1IRWFAMsa)Cited by:[§4](https://arxiv.org/html/2608.08888#S4.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§4\.2](https://arxiv.org/html/2608.08888#S4.SS2.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, Y\. Song, Y\. Wang, K\. Ge, A\. Lamb, Q\. Guo, K\. Chen, B\. Zhou, and Z\. LinNext concept prediction in discrete latent space leads to stronger language models\.arXiv preprint arXiv:2602\.08984\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1)\.
- Mandalet al\.\(2026\)B\. Mandal, S\. Berman, A\. Vegesna, and S\. DahalQ0: primitives for hyper\-epoch pretraining\.arXiv preprint arXiv:2606\.03938\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1)\.
- Movahediet al\.\(2026\)S\. Movahedi, V\. Milovanović, S\. L\. Feigin, A\. Theus, T\. Hofmann, V\. Boeva, T\. K\. Rusch, and A\. OrvietoFixed\-point reasoners: stable and adaptive deep looped transformers\.arXiv preprint arXiv:2606\.18206\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1)\.
- Nociet al\.\(2022\)L\. Noci, S\. Anagnostidis, L\. Biggio, A\. Orvieto, S\. P\. Singh, and A\. LucchiSignal propagation in transformers: theoretical perspectives and the role of rank collapse\.Advances in Neural Information Processing Systems35,pp\. 27198–27211\.Cited by:[1st item](https://arxiv.org/html/2608.08888#S3.I2.i1.p1.1)\.
- Qiet al\.\(2025\)Z\. Qi, F\. Nie, A\. Alahi, J\. Zou, H\. Lakkaraju, Y\. Du, E\. Xing, S\. Kakade, and H\. ZhangEvoLM: in search of lost language model training dynamics\.arXiv preprint arXiv:2506\.16029\.Cited by:[Table 2](https://arxiv.org/html/2608.08888#A2.T2),[Table 2](https://arxiv.org/html/2608.08888#A2.T2.3.8.1.1.2.1.2.1),[Table 2](https://arxiv.org/html/2608.08888#A2.T2.5)\.
- Teohet al\.\(2025\)J\. Teoh, M\. Tomar, K\. Ahn, E\. S\. Hu, T\. Pearce, P\. Sharma, A\. Krishnamurthy, R\. Islam, A\. Lamb, and J\. LangfordNext\-latent prediction transformers learn compact world models\.arXiv preprint arXiv:2511\.05963\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1),[footnote 1](https://arxiv.org/html/2608.08888#footnote1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2608.08888#S1.p2.1)\.
- Yanget al\.\(2024\)G\. Yang, D\. Yu, C\. Zhu, and S\. HayouTensor programs VI: feature learning in infinite depth neural networks\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=17pVDnpwwl)Cited by:[1st item](https://arxiv.org/html/2608.08888#S3.I2.i1.p1.1)\.
- Yanget al\.\(2025\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving mamba2 with delta rule\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 29687–29707\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px3.p1.1)\.
- Yueet al\.\(2026\)Z\. Yue, B\. Jin, H\. Zeng, H\. Zhuang, Z\. Qin, J\. Yoon, L\. Shang, J\. Han, and D\. WangHybrid latent reasoning via reinforcement learning\.Advances in Neural Information Processing Systems38,pp\. 5501–5530\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px2.p1.1)\.
- Zenget al\.\(2025\)B\. Zeng, H\. Li, S\. Song, Y\. Wang, Z\. Wang, Z\. He, X\. Wang, and Z\. LinPonderlm\-2: pretraining llm with latent thoughts in continuous space\.arXiv preprint arXiv:2509\.23184\.Cited by:[§3\.3](https://arxiv.org/html/2608.08888#S3.SS3.p4.1),[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2608.08888#S6.p1.1)\.
- Zhanget al\.\(2026a\)X\. Zhang, D\. Zhang, S\. Zhang, X\. Qin, Y\. Cheng, and J\. YanNITP: next implicit token prediction for llm pre\-training\.InForty\-third International Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2026b\)Z\. Zhang, X\. He, W\. Yan, A\. Shen, C\. Zhao, and X\. WangSoft thinking: unlocking the reasoning potential of llms in continuous concept space\.Advances in Neural Information Processing Systems38,pp\. 168990–169012\.Cited by:[§5](https://arxiv.org/html/2608.08888#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AModel architecture

The model is a decoder\-only causal language model with a tied 100,352\-token embedding and output head, 24 transformer layers, a 1,536\-dimensional hidden state, and 6,656\-dimensional SiLU GLU feed\-forward blocks\. Its gated grouped\-query attention uses 16 query heads, 8 shared key/value heads, headwise gates, QK RMS normalization, and rotary positions over an 8,192\-token context; most layers use a 2,048\-token sliding window, while every sixth layer uses full attention\. RMS normalization is applied around each residual block and at the final output\.

## Appendix BComparison of LM eval performance with other models of similar scale

Model NameTokensW/GPIQAOBQAARC\-EARC\-CAvg\.OPT 1\.3B300B59\.5972\.3633\.4050\.8029\.4449\.87Pythia 1B300B53\.4369\.2131\.4048\.9927\.0546\.21Pythia 1\.4B300B57\.3870\.9533\.2054\.0028\.5049\.34TinyLlama 1B2T59\.4373\.5636\.8055\.4732\.6853\.23Llama3\.2 1B9T60\.4674\.5437\.0060\.4835\.7555\.31Qwen3 1\.7B36T61\.0172\.3636\.8069\.9143\.2657\.30EvoLM 1B\([31](https://arxiv.org/html/2608.08888#bib.bib36)\)20B51\.3067\.8532\.8054\.8029\.6146\.4440B54\.6269\.5936\.2058\.0830\.2949\.3880B53\.5970\.7837\.2062\.7135\.9251\.88160B53\.9971\.7136\.6063\.0936\.0952\.30320B53\.5171\.9337\.2062\.2936\.1852\.49Full\-bandwithtransformer 1B200B \(0 feedback pass\)60\.4671\.1134\.6062\.4234\.7352\.66200B \(1 feedback pass\)62\.5971\.4935\.0063\.4335\.4153\.58Table 2:0\-shot LM Eval performance comparision, numbers for EvoLM and other open\-sourced models are adopted from Table 4 in the appendix of[31](https://arxiv.org/html/2608.08888#bib.bib36)\.
## Appendix CFull pseudo code for training

Listing 3:Training: one step withkkpasses\.1defglu\_cross\(h,e\):

2return\(h@W\_u\)\*sigmoid\(e@W\_g\)

3

4e=embed\(tokens\)

5h=model\(e\)

6loss=ntp\_loss\(h\)

7for\_inrange\(k\-1\):

8h=h\+uniform\(\-delta,delta\)

9x=glu\_cross\(shift\_right\(h\),input\_rmsnorm\_1\(e\)\)

10x=prefix\_mixin\(x,e\)

11h=model\(input\_rmsnorm\_1\(x\)\)

12loss\+=ntp\_loss\(h\)

Figure 9:Full training code for full\-bandwidth transformer, with normalization layer and regularization noise included\.
## Appendix DvLLM compatibility

The implementation on vLLM follows the same design pattern as EAGLE[23](https://arxiv.org/html/2608.08888#bib.bib9)/ MTP[16](https://arxiv.org/html/2608.08888#bib.bib10): it retains each request’s latest trunk hidden state and copies it in place into a persistent, fixed\-address model buffer before the next decode step, allowing CUDA graphs to capture the glu cross gate \(Eq\. \([4](https://arxiv.org/html/2608.08888#S3.E4)\)\) insideforward\. A patchedGPUModelRunner\.\_model\_forwardstores detached hidden states in a dictionary keyed by request ID, usesquery\_start\_locto map packed rows to requests, and removes completed requests\. Our forward function than fuses the saved state with the next token embedding through the learned glu cross gate, then recycles the resulting hidden state\. Unlike EAGLE/MTP, which send target hidden states to a separate speculative draft model, our model feeds its own state back into the same model to define the actual next\-token distribution\.

## Appendix EExtended extrapolation results

Figure 10:Similar to Fig\.[3](https://arxiv.org/html/2608.08888#S3.F3), but extending number of feedback pass to 1,000\. The extrapolation remains stable far beyond the 3 passes used in training time\.
## Appendix FExplanation on state tracking tasks

We construct paired synthetic examples whose label is determined by information appearing before a shared final colon\. The target token itself is never included in the input\. We append0,8,32,128,0,8,32,128,or256256semantically null scratch updates, allowing us to vary sequence length without changing the target\. At the final colon, we record the layer\-0 input and the output of every Transformer block\.

#### Completion tracking\.

Each input specifies a required countaaand a completed countbb\. The target isdoneifa=ba=bandmoreotherwise\. For each unordered numeral pair\{a,b\}\\\{a,b\\\}, we include all four assignments\(a,a\),\(a,b\),\(b,a\),\(b,b\)\(a,a\),\(a,b\),\(b,a\),\(b,b\), balancing every numeral across fields and labels\. A representative matched pair, abbreviated to show eight repeated distractors, is

```
required = 4                 required = 4
completed = 9                completed = 4
scratch = 7                  scratch = 7
scratch += 0                 scratch += 0
  ... (8 updates)              ... (8 updates)
Status:                      Status:
```

The left target ismore, whereas the right target isdone\. The two examples share the required count, scratch context, distractor sequence, and final token; only the relation between the two counters changes\.

#### Delayed memory\.

Each input first assigns a binary state and then presents label\-independent scratch operations\. The target iszerooroneaccording to the initial state\. For example,

```
state = 0                    state = 1
scratch = 0                  scratch = 0
scratch ˆ= 0                 scratch ˆ= 0
scratch ˆ= 1                 scratch ˆ= 1
scratch ˆ= 1                 scratch ˆ= 1
scratch ˆ= 0                 scratch ˆ= 0
scratch ˆ= 1                 scratch ˆ= 1
scratch += 0                 scratch += 0
  ... (8 updates)              ... (8 updates)
# final state:               # final state:
```

The corresponding targets arezeroandone\. Thus the model must retain the initial bit while processing an identical intervening context\. Completion tracking tests a relational state computed from multiple fields, whereas delayed memory tests persistent transport of an already specified state\.

#### Multi\-register latest\-write tracking\.

We additionally test whether recurrent prefilling can expose several independently updated variables\. An input assigns binary values to registersr0,…,rm−1r\_\{0\},\\ldots,r\_\{m\-1\}, performs eight label\-independent scratch updates, and then queries one register\. The target iszerooroneaccording to that register’s most recent assignment\. For example, the following matched inputs share the complete update history and differ only in the queried register:

```
r4 = 0                      r4 = 0
r4 = 1                      r4 = 1
r0 = 1                      r0 = 1
r7 = 0                      r7 = 0
  ... (10 assignments)        ... (10 assignments)
r7 = 1                      r7 = 1
r1 = 0                      r1 = 0
scratch = 7                 scratch = 7
scratch += 0                scratch += 0
  ... (7 updates)             ... (7 updates)
query = r0                   query = r1
Value:                       Value:
```

Here the latest values arer0=1r\_\{0\}=1andr1=0r\_\{1\}=0, so the left target isoneand the right target iszero\. The model must therefore preserve the latest value of every register and bind the final query to the appropriate component of that state\.

#### Probe construction\.

We train anL2L\_\{2\}\-regularized linear classifier at each residual\-stream depth using four\-fold grouped cross\-validation\. Completion splits hold out entire unordered numeral\-pair groups, and memory splits hold out complete scratch\-context groups\. The enlarged experiment contains1,6001\{,\}600completion examples from 80 groups and1,2801\{,\}280memory examples from 128 groups\. Because every example ends at the same colon token, the standard layer\-0 representation contains no label information beyond the shared token embedding; any above\-chance accessibility must be introduced by processing the prefix or by recurrent fusion\.

#### Register\-count and overwrite sweeps\.

In the register\-count sweep, every input contains 16 assignments and eight null updates and is padded to exactly 180 tokens; only the number of registers varies overm∈\{1,2,4,8\}m\\in\\\{1,2,4,8\\\}\. This separates the effect of maintaining more variables from input length and total update count\. We use 128 structural groups per register count\. Each group contains a random register\-update schedule, its bitwise value complement, and queries for every register, and grouped cross\-validation holds out the entire schedule and all associated queries\. The resulting sweep contains3,8403\{,\}840examples per prefill condition\. To vary overwrite interference directly, we then fixm=8m=8and use2,4,2,4,or88writes per register\. Each setting contains2,0482\{,\}048examples from 128 groups and produces inputs of 180, 276, and 468 tokens, respectively\.

#### Recurrent\-suffix controls\.

Besides standard and full recurrent prefilling, we recurrently prefill only the finalk∈\{1,2,4\}k\\in\\\{1,2,4\\\}input tokens while standard\-prefilling the preceding prefix\. One step fuses state only at the shared final colon, two steps recurrently processValue:, and four steps additionally include the queried\-register digit and newline\. We probe the residual stream at the final colon at layers0,1,2,0,1,2,and44, as well as at every remaining depth, using the same groupedL2L\_\{2\}\-regularized classifiers\. This sweep distinguishes information accumulated throughout the update sequence from information made accessible locally while processing the final query\.

## Appendix GModel outpout

Soft decoding\*\*\*\* 200b model, standard decoding \*\*\*\*To determine the last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}, we start by performing the division of 137 by 500\.First, we set up the division:137÷500137\\div 500Since 137 is less than 500, the decimal expansion will start with a zero\. We can rewrite the division as:137\.0÷500137\.0\\div 500Next, we perform the division step\-by\-step: 1\.500500goes into13401340\(the first two digits of137\.0137\.0\) twice:500×2=1000500\\times 2=1000Subtract10001000from13401340:1340−1000=3401340\-1000=3402\. Bring down the next digit \(0\), making it340\.0340\.0:500×0=0500\\times 0=0Subtract00from340340:340−0=340340\-0=3403\. Bring down the next digit \(0\), making it34003400:500×6=3000500\\times 6=3000Subtract30003000from34003400:3400−3000=4003400\-3000=4004\. Bring down the next digit \(0\), making it40004000:500×8=4000500\\times 8=4000Subtract40004000from40004000:4000−4000=04000\-4000=0Since the remainder is now00, the decimal expansion of137500\\frac\{137\}\{500\}is:0\.2740\.274The last nonzero digit to the right of the decimal point is:4\\boxed\{4\}——————————\*\*\*\* 200b model, soft \*\*\*\*To find the last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}, we can perform the division and observe the pattern of the decimal expansion\.137500=0\.274\\frac\{137\}\{500\}=0\.274The last nonzero digit to the right of the decimal point in the decimal expansion of137500\\frac\{137\}\{500\}is 4\.

Soft decoding\*\*\*\* 200b model, standard decoding \*\*\*\*To simplify the expression\(10​r3\)​\(4​r6\)8​r4\\frac\{\(10r^\{3\}\)\(4r^\{6\}\)\}\{8r^\{4\}\}, we will follow these steps:1\. \*\*Multiply the numerator\*\*:\(10​r3\)​\(4​r6\)\(10r^\{3\}\)\(4r^\{6\}\)Using the properties of exponents, we can combine the terms:10⋅4⋅r3\+6=40​r910\\cdot 4\\cdot r^\{3\+6\}=40r^\{9\}2\. \*\*Rewrite the expression with the simplified numerator\*\*:40​r98​r4\\frac\{40r^\{9\}\}\{8r^\{4\}\}3\. \*\*Simplify the fraction\*\*: \- First, simplify the coefficients:408=5\\frac\{40\}\{8\}=5\- Next, simplify the exponents ofrr:r9r4=r9−4=r5\\frac\{r^\{9\}\}\{r^\{4\}\}=r^\{9\-4\}=r^\{5\}4\. \*\*Combine the simplified parts\*\*:5​r55r^\{5\}Thus, the simplified form of the expression\(10​r3\)​\(4​r6\)8​r4\\frac\{\(10r^\{3\}\)\(4r^\{6\}\)\}\{8r^\{4\}\}is:5​r5\\boxed\{5r^\{5\}\}——————————\*\*\*\* 200b model, soft \*\*\*\*To simplify the expression, we can first simplify the numerator and then divide by the denominator\.Numerator:\(10​r3\)​\(4​r6\)=40​r9\(10r^\{3\}\)\(4r^\{6\}\)=40r^\{9\}Denominator:8​r48r^\{4\}Now, we divide the numerator by the denominator:40​r98​r4=5​r9−4=5​r5\\frac\{40r^\{9\}\}\{8r^\{4\}\}=5r^\{9\-4\}=5r^\{5\}So, the simplified expression is5​r55r^\{5\}\.