Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

Hugging Face Daily Papers Papers
looped-transformers test-time-compute attention reasoning architecture

Summary

该论文提出 InfiLoop，一种为循环 Transformer 设计的原生残差连接机制，通过内容加权与学习式时间衰减维持循环状态的运行摘要，使模型在数万次测试时迭代中持续提升推理性能，在 Sudoku-Extreme 上达到 97.9% 精度、ARC-AGI-2 上 13.6% pass@2。

In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions. This leaves subsequent iterations to recover lost information from an already degraded representation: once an error arises in an earlier loop, often as a result of long-range propagation through the recurrence, later loops find it difficult to correct. In this paper, we introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update. InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps its persistent aggregation memory constant as the loop count grows. The resulting adaptive update suppresses unreliable proposals and preserves useful intermediate states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2. Notably, on Sudoku-Extreme, InfiLoop continues to improve with test-time looping beyond 20,000 effective steps, showing that added depth translates directly into stronger reasoning. Our code is available at https://github.com/pixeli99/InfiLoop.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:26 AM

Paper page - Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

Source: https://huggingface.co/papers/2610.11570

Abstract

Inthispaper,wearguethatloopedTransformersneedtheirownresidualconnectionstopreventperformancedegradationasthenumberofiterationsgrows.Weobservethatincreasingloopiterationscanreducereasoningaccuracy:noisystateupdatesoverwritecorrectintermediatedeductionsandevenundocompletedsolutions.Thisleavessubsequentiterationstorecoverlostinformationfromanalreadydegradedrepresentation:onceanerrorarisesinanearlierloop,oftenasaresultoflong-rangepropagationthroughtherecurrence,laterloopsfinditdifficulttocorrect.Inthispaper,weintroduceInfiLoop,aloop-nativeresidualconnectionthatlearnswhichpastcomputationstoretainandhowmuchtoacceptfromeachnewupdate.InfiLoopcombinescontent-basedweightingwithlearnedtemporaldecaytomaintainarunningsummaryofrecurrentstates.Anexactstreamingrecurrencekeepsitspersistentaggregationmemoryconstantastheloopcountgrows.Theresultingadaptiveupdatesuppressesunreliableproposalsandpreservesusefulintermediatestates.Acrossextensivereasoningtasks,a7M-parameterInfiLoopmodeloutperformsexistingrecursivearchitectures,reaching97.9%exactaccuracyonSudoku-Extreme,and13.6%[email protected],onSudoku-Extreme,InfiLoopcontinuestoimprovewithtest-timeloopingbeyond20,000effectivesteps,showingthataddeddepthtranslatesdirectlyintostrongerreasoning.Ourcodeisavailableathttps://github.com/pixeli99/InfiLoop.

View arXiv pageView PDFGitHub1Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.11570 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.11570 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.11570 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Simply Stabilizing the Loop via Fully Looped Transformer

arXiv cs.LG

This paper identifies gradient oscillation and residual explosion as causes of training instability in Looped Transformers, and proposes Fully Looped Transformer with two parameter-free modifications (Fully Looped Architecture and Attention Injection) to stabilize training up to 12 loop iterations, achieving up to 13.2% improvement in downstream performance.

Improving Test-Time Scaling with Adaptive Looped Transformers

Hugging Face Daily Papers

This paper introduces TaH2, an adaptive looped transformer that improves test-time scaling by dynamically allocating extra iterations to beneficial tokens, achieving a 53% improvement in accuracy-compute slope over baselines on benchmarks like AIME.

DeepLoop: Depth Scaling for Looped Transformers

arXiv cs.LG

DeepLoop introduces a residual scaling method for looped Transformers that adjusts for parameter visits, improving stability and performance when physical blocks are reused across multiple rounds.

@askalphaxiv: Another cool research on Looped Transformers They ask the question: "Can we loop a frozen, off-the-shelf checkpoint dir…

X AI KOLs Timeline

This research introduces a technique to loop frozen, off-the-shelf transformer checkpoints at inference time by using damped Runge-Kutta substeps, treating transformer layers as Euler steps in a residual ODE. This allows extra latent compute without fine-tuning, architecture changes, or new weights, showing gains on knowledge tasks like MMLU-Pro, GPQA, and ARC.

Decoding Looped Transformers Better for (Almost) Free

Hugging Face Daily Papers

The paper introduces LoopCD, a training-free contrastive decoding framework for looped transformers that uses intermediate recurrent states as weak/strong prediction pairs, delivering substantial gains (e.g., Ouro-2.6B-Thinking's AIME 2024 pass@1 rising from 61.88% to 73.33%) while enabling halving of recurrent loops and 22.5%-48.2% reductions in forward FLOPs.