Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

Hugging Face Daily Papers Papers
memory-efficient looped-transformer recurrent-llm kv-cache reasoning chunk-wise-training

Summary

Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.

Recurrent LLM architectures have emerged as a promising approach for improving reasoning, as they enable multi-step computation in the embedding space without generating intermediate tokens. Models such as Ouro perform reasoning by iteratively updating internal representations while retaining a standard Key-Value (KV) cache across iterations, causing memory consumption to grow linearly with reasoning depth. Consequently, increasing the number of reasoning iterations can lead to prohibitive memory usage, limiting the practical scalability of such architectures. In this work, we propose Memory-Efficient Looped Transformer (MELT), a novel architecture that decouples reasoning depth from memory consumption. Instead of using a standard KV cache per layer and loop, MELT maintains a single KV cache per layer that is shared across reasoning loops. This cache is updated over time via a learnable gating mechanism. To enable stable and efficient training under this architecture, we propose to train MELT using chunk-wise training in a two phase procedure: interpolated transition, followed by attention-aligned distillation, both from the LoopLM starting model to MELT. Empirically, we show that MELT models fine-tuned from pretrained Ouro parameters outperform standard LLMs of comparable size, while maintaining a memory footprint comparable to those models and dramatically smaller than Ouro's. Overall, MELT achieves constant-memory iterative reasoning without sacrificing LoopLM performance, using only a lightweight post-training procedure.
Original Article
View Cached Full Text

Cached at: 05/12/26, 10:53 AM

Paper page - Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models

Source: https://huggingface.co/papers/2605.07721

Abstract

A novel recurrent LLM architecture called MELT is proposed that decouples reasoning depth from memory consumption by sharing a single KV cache across reasoning loops and using chunk-wise training with interpolated transition and attention-aligned distillation.

Recurrent LLM architectureshave emerged as a promising approach for improving reasoning, as they enablemulti-step computationin theembedding spacewithout generating intermediate tokens. Models such as Ouro perform reasoning by iteratively updating internal representations while retaining a standard Key-Value (KV) cache across iterations, causingmemory consumptionto grow linearly withreasoning depth. Consequently, increasing the number of reasoning iterations can lead to prohibitive memory usage, limiting the practical scalability of such architectures. In this work, we propose Memory-Efficient Looped Transformer (MELT), a novel architecture that decouplesreasoning depthfrommemory consumption. Instead of using a standard KV cache per layer and loop, MELT maintains a single KV cache per layer that is shared across reasoning loops. This cache is updated over time via alearnable gating mechanism. To enable stable and efficient training under this architecture, we propose to train MELT usingchunk-wise trainingin a two phase procedure:interpolated transition, followed byattention-aligned distillation, both from theLoopLMstarting model to MELT. Empirically, we show that MELT models fine-tuned from pretrained Ouro parameters outperform standard LLMs of comparable size, while maintaining a memory footprint comparable to those models and dramatically smaller than Ouro’s. Overall, MELT achieves constant-memoryiterative reasoningwithout sacrificingLoopLMperformance, using only a lightweight post-training procedure.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.07721

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.07721 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.07721 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.07721 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

arXiv cs.CL

This paper introduces CoMem, a method that exploits the depth-wise division of labor in LLMs to cache intermediate residual tensors and recompute only upper layers for retrieval, enabling bounded read compute and memory independent of stored-context length. Evaluated on Qwen3-8B, CoMem achieves strong long-context performance with significant memory savings and prefill speedups.

Language Models Need Sleep

Hacker News Top

This paper introduces a sleep-like consolidation mechanism for Transformer-based LLMs that periodically converts recent context into persistent fast weights in SSM blocks, clearing the KV cache to improve long-horizon reasoning without increasing inference latency.

Δ-Mem: Efficient Online Memory for Large Language Models

Hacker News Top

Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.