Loopback Transformer: Towards a Unified View of Residual Looped Transformers
Summary
The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.
View Cached Full Text
Cached at: 11/16/26, 08:50 AM
Paper page - Loopback Transformer: Towards a Unified View of Residual Looped Transformers
Source: https://huggingface.co/papers/2611.07226 In this paper, we study how residual connections should retain and reuse information when the same Transformer blocks run repeatedly. Inspired by recurrent memory, we develop aunified view of residual connections as memory operating along network depth. The write–transport–read formulation lets us compare different residual designs through what they store, how they carry information forward, and how they retrieve it for subsequent computation.
Guided by this view, we introduceLoopback Transformer. Loopback stores individual block updates, uses geometric forgetting to progressively discount earlier passes, and learns content-dependent attention over the stored updates. This allows the model to retrieve useful information from earlier computation.
At matched training tokens and executed depth, Loopback achieveslower validation loss and higher mean accuracy across 13 zero-shot tasksthan Standard Transformers at both 1.4B and 3B, retaining only68% and 75% of their Transformer parameters, respectively. At 3B, validation loss improves from2.0005 to 1.9849, mean accuracy rises from48.56% to 49.45%, and Loopback scores higher on11 of 13 tasks. In our main comparisons at 191M, 374M, and 778M, it also achieves the lowest validation loss and highest thirteen-task mean among Standard and seven looped residual baselines. Parameter counts exclude token embeddings and the output head.
Our results highlight residual design as an important part of parameter-efficient modeling: sharing computation works better when the model also learns how to retain and reuse its outputs. We welcome feedback and discussion!
Similar Articles
What Are Looped Transformers? Explained Clearly (8 minute read)
Looped transformers reuse the same layers across multiple passes to trade parameter count for compute, achieving better reasoning with fewer weights. The article traces the idea back to the Universal Transformer (2018) and explains why it initially failed due to scaling laws and timing.
DeepLoop: Depth Scaling for Looped Transformers
DeepLoop introduces a residual scaling method for looped Transformers that adjusts for parameter visits, improving stability and performance when physical blocks are reused across multiple rounds.
On the Residual Scaling of Looped Transformers: Stability and Transferability
This paper analyzes residual scaling in looped (weight-tied) transformers, showing that weight sharing requires stronger scaling (1/N) than standard residual networks, and derives a factored parameterization that enables hyperparameter transfer across loop counts without retuning.
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.
LSTM-Style Looped Transformers with Shared KV Caches
This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.
