Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
Summary
Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.
View Cached Full Text
Cached at: 05/12/26, 10:53 AM
Paper page - Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
Source: https://huggingface.co/papers/2605.07721
Abstract
A novel recurrent LLM architecture called MELT is proposed that decouples reasoning depth from memory consumption by sharing a single KV cache across reasoning loops and using chunk-wise training with interpolated transition and attention-aligned distillation.
Recurrent LLM architectureshave emerged as a promising approach for improving reasoning, as they enablemulti-step computationin theembedding spacewithout generating intermediate tokens. Models such as Ouro perform reasoning by iteratively updating internal representations while retaining a standard Key-Value (KV) cache across iterations, causingmemory consumptionto grow linearly withreasoning depth. Consequently, increasing the number of reasoning iterations can lead to prohibitive memory usage, limiting the practical scalability of such architectures. In this work, we propose Memory-Efficient Looped Transformer (MELT), a novel architecture that decouplesreasoning depthfrommemory consumption. Instead of using a standard KV cache per layer and loop, MELT maintains a single KV cache per layer that is shared across reasoning loops. This cache is updated over time via alearnable gating mechanism. To enable stable and efficient training under this architecture, we propose to train MELT usingchunk-wise trainingin a two phase procedure:interpolated transition, followed byattention-aligned distillation, both from theLoopLMstarting model to MELT. Empirically, we show that MELT models fine-tuned from pretrained Ouro parameters outperform standard LLMs of comparable size, while maintaining a memory footprint comparable to those models and dramatically smaller than Ouro’s. Overall, MELT achieves constant-memoryiterative reasoningwithout sacrificingLoopLMperformance, using only a lightweight post-training procedure.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.07721
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.07721 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.07721 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.07721 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Surprising Effectiveness of Shared Memory in Looped Transformers
An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.
Loopback Transformer: Towards a Unified View of Residual Looped Transformers
The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.
LSTM-Style Looped Transformers with Shared KV Caches
This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Introduces Looped Latent Attention (LLA), a post-training codec that compresses KV cache in looped transformers by exploiting low-rank structure across recurrence steps, achieving significant compression ratios while maintaining performance.
Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory
This paper introduces CoMem, a method that exploits the depth-wise division of labor in LLMs to cache intermediate residual tensors and recompute only upper layers for retrieval, enabling bounded read compute and memory independent of stored-context length. Evaluated on Qwen3-8B, CoMem achieves strong long-context performance with significant memory savings and prefill speedups.