Full-bandwidth transformer
Summary
A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.
View Cached Full Text
Cached at: 08/14/26, 03:26 AM
Paper page - Full-bandwidth transformer
Source: https://huggingface.co/papers/2608.08888
Abstract
Full-bandwidth transformers use latent feedback of top-layer hidden states to improve reasoning and efficiency without altering the core architecture.
Autoregressive transformerscompute along two axes: horizontally across generated tokens, and vertically through model depth.Dense attentiongives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce thefull-bandwidth transformer, which widens this channel withlatent feedback: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through agated linear unitand fed back as the next input.Latent feedbacklets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture,KV cache, and language-modeling objective. To trainfull-bandwidth transformers without losing parallelteacher forcing, we use ascheduled multi-pass objectivethat introduceslatent feedbacklate in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameterfull-bandwidth transformers up to 400B tokens and find thatlatent feedbackimproves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead,full-bandwidth transformers match or approach standard transformers trained with roughly 1.5times more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.08888
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.08888 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.08888 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.08888 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Variable-Width Transformers
Proposes a nonuniform width allocation transformer (hourglass shape) that outperforms uniform baselines in language modeling, reducing FLOPs and KV cache size.
ResBM: a new transformer-based architecture for low-bandwidth pipeline-parallel training, achieving 128× activation compression [R]
ResBM introduces a transformer-based architecture with residual encoder-decoder bottlenecks for pipeline-parallel training, achieving 128× activation compression while maintaining convergence. The work advances decentralized, internet-grade distributed training by reducing inter-stage communication overhead.
Transformer co-author validates post-transformer cost efficiency breakthrough
A 150M-parameter non-transformer architecture achieves state-of-the-art cost-efficiency on ARC-AGI-1, validated by Transformer co-author Łukasz Kaiser, suggesting that recurrent latent reasoning can replace brute-force scaling.
@yingfan_bot: New paper on Looped Transformers! Latent reasoning is fast, but struggles to match CoT-level accuracy at scale. Can loo…
A new paper on Looped Transformers finds that a looped padded backbone provides a parallel workspace for latent reasoning, enabling supervision similar to explicit chain-of-thought (CoT) and achieving both speed and accuracy.
@ZhihuFrontier: Half a year ago, a Zhihu contributor predicted that the next Transformer would absorb loops, recurrent state, sparse ro…
A Zhihu contributor's half-year-old prediction that the next Transformer would absorb loops, recurrent state, sparse routing, and latent reasoning is gaining relevance as Loop Engineering advances. The article explores how future Transformer architectures may evolve into hybrid models blending linear-complexity layers for background context with attention for precise reasoning, plus finer-grained sparsity and native System 2 reasoning.