Decoding Looped Transformers Better for (Almost) Free

Hugging Face Daily Papers Papers
looped-transformers contrastive-decoding inference-efficiency recurrent-transformers language-models training-free

Summary

The paper introduces LoopCD, a training-free contrastive decoding framework for looped transformers that uses intermediate recurrent states as weak/strong prediction pairs, delivering substantial gains (e.g., Ouro-2.6B-Thinking's AIME 2024 pass@1 rising from 61.88% to 73.33%) while enabling halving of recurrent loops and 22.5%-48.2% reductions in forward FLOPs.

Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:30 PM

Paper page - Decoding Looped Transformers Better for (Almost) Free

Source: https://huggingface.co/papers/2610.02185

Abstract

LoopedTransformersachieveparameterefficiencybyrepeatedlyexecutingasharedblockacrossrecurrentloops.Eachloopyieldsanintermediaterepresentationdecodableforthesamenexttoken,yetstandarddecodingdiscardsearlierstates.Becauseearlierloopsembodylesscomputation,recurrenceinherentlysuppliesalignedweak-and-strongpredictionpairswithoutauxiliarymodelsorexternaltraining.WeintroduceLoopCD,atraining-freecontrastivedecodingframeworkthatguidestokenselectionbycontrastingthefinalpredictionwithanearlierrecurrentpass,operatingeitherinlogitspacewithoneextraoutputpass(LoopCD-Logits)orinhidden-statespacewithzerooutputoverhead(LoopCD-Hidden).AcrossfourloopedTransformerfamilies,LoopCDdeliverssubstantial,consistentgainsatfullrecurrentdepth:LoopCD-LogitsraisesOuro-2.6B-Thinking’[email protected]%to73.33%,whileLoopCD-HiddenliftsHuginn’[email protected]%to31.71%.Crucially,theseperformancegainsenablehalvingthenumberofrecurrentloopswhilestillmatchingorexceedingfull-depthunguidedbaselines,reducingforwardFLOPsby22.5%to48.2%.Bytransformingintermediaterecurrentstatesintoeffectiveguidancesignals,LoopCDachievessuperiordecodingqualitywhilesubstantiallyreducinginferencecompute.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02185 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02185 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02185 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

@askalphaxiv: Another cool research on Looped Transformers They ask the question: "Can we loop a frozen, off-the-shelf checkpoint dir…

X AI KOLs Timeline

This research introduces a technique to loop frozen, off-the-shelf transformer checkpoints at inference time by using damped Runge-Kutta substeps, treating transformer layers as Euler steps in a residual ODE. This allows extra latent compute without fine-tuning, architecture changes, or new weights, showing gains on knowledge tasks like MMLU-Pro, GPQA, and ARC.

What Are Looped Transformers? Explained Clearly (8 minute read)

TLDR AI

Looped transformers reuse the same layers across multiple passes to trade parameter count for compute, achieving better reasoning with fewer weights. The article traces the idea back to the Universal Transformer (2018) and explains why it initially failed due to scaling laws and timing.