Variable-Width Transformers
Summary
Proposes a nonuniform width allocation transformer (hourglass shape) that outperforms uniform baselines in language modeling, reducing FLOPs and KV cache size.
View Cached Full Text
Cached at: 06/17/26, 03:35 AM
Paper page - Variable-Width Transformers
Source: https://huggingface.co/papers/2606.18246
Abstract
A novel transformer architecture with nonuniform width allocation across layers achieves better performance and efficiency compared to uniform designs by utilizing a parameter-free residual resizing mechanism.
Scaling model size, specificallydepthandwidth, has driven significant progress intransformer-based language models. However, most architectures maintain a constantwidthacross all layers, allocating a fixed parameter and computation budget evenly despite different layers potentially playing distinct computational roles. In this work, we empirically investigate nonuniform capacity allocation across networkdepthby proposing a times-shaped > <former architecture. This design maintains wider early and late layers while narrowing the middle layers, utilizing a parameter-freeresidual resizing mechanism. Acrossdecoder-only language modelsranging from 200M to 2B parameters (dense) and 3B parameters (MoE), our > <former consistently outperforms parameter-matched uniform baselines on language modeling loss. By reducing the average layerwidth, this architecture also requires fewer overallFLOPs(22% reduction under fitted loss-matched scaling curves) and smallerKV cache memoryand I/O cost (15% reduction). In analysis, we show that this bottleneck structure results in qualitatively different representations in residual streams. Overall, our results demonstrate that nonuniformwidthallocation can result in more resource-optimal scaling of language models.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.18246
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.18246 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.18246 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.18246 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Microsoft's Full-bandwidth Transformers (26 minute read)
The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.
Multi-Branch Transformer
The Multi-Branch Transformer (MBT) splits Transformer blocks into parallel parameterized branches with shared initial/final layers, and its Balanced variant with a shared Joint-KV cache outperforms sequential Transformers at matched compute while reducing KV-cache storage by 34% and boosting decoding throughput by up to 51.4% on an H200.
Full-bandwidth transformer
A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.
Improving Test-Time Scaling with Adaptive Looped Transformers
This paper introduces TaH2, an adaptive looped transformer that improves test-time scaling by dynamically allocating extra iterations to beneficial tokens, achieving a 53% improvement in accuracy-compute slope over baselines on benchmarks like AIME.
Loopback Transformer: Towards a Unified View of Residual Looped Transformers
The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.