Improving Test-Time Scaling with Adaptive Looped Transformers

Hugging Face Daily Papers Papers

Summary

This paper introduces TaH2, an adaptive looped transformer that improves test-time scaling by dynamically allocating extra iterations to beneficial tokens, achieving a 53% improvement in accuracy-compute slope over baselines on benchmarks like AIME.

Looped transformers have demonstrated promising parameter efficiency by reusing layers for latent computation. Prior studies compare looped and non-looped models at matched parameters or per-token FLOPs. However, to the best of our knowledge, whether looping improves test-time scaling as outputs grow longer remains underexplored. Through post-training looped transformers, we study the accuracy-compute slope, measured as the accuracy gain per doubling of test-time decoding FLOPs. We find that existing looped transformers often yield steeper slopes than their non-looped baseline, yet underperform it at matched compute. While fixed-depth looping spends extra iterations on every token, our analysis shows that many tokens do not benefit from extra iterations. We therefore propose TaH2, which enables the model to focus extra iterations on the tokens that benefit from looping. It jointly post-trains the backbone and an iteration decider through lookahead depth supervision, which uses online labels indicating whether further iteration improves the prediction. TaH2 improves both the efficiency and attainable accuracy of test-time scaling. On challenging AIME benchmarks, TaH2 improves the accuracy-compute slope by 53% (2.74 vs. 1.79) over the non-looped baseline, exceeding the baseline's peak accuracy by about 3.4 points at matched test-time compute. As the maximum iteration depth increases, existing looped models largely plateau, while TaH2's gain over the non-looped baseline continues to grow from +2.8 points at depth 2 to +3.9 points at depth 8. Our code is available at https://github.com/thu-nics/TaH.
Original Article
View Cached Full Text

Cached at: 09/29/26, 08:19 AM

Paper page - Improving Test-Time Scaling with Adaptive Looped Transformers

Source: https://huggingface.co/papers/2609.35748

Abstract

Loopedtransformershavedemonstratedpromisingparameterefficiencybyreusinglayersforlatentcomputation.Priorstudiescompareloopedandnon-loopedmodelsatmatchedparametersorper-tokenFLOPs.However,tothebestofourknowledge,whetherloopingimprovestest-timescalingasoutputsgrowlongerremainsunderexplored.Throughpost-trainingloopedtransformers,westudytheaccuracy-computeslope,measuredastheaccuracygainperdoublingoftest-timedecodingFLOPs.Wefindthatexistingloopedtransformersoftenyieldsteeperslopesthantheirnon-loopedbaseline,yetunderperformitatmatchedcompute.Whilefixed-depthloopingspendsextraiterationsoneverytoken,ouranalysisshowsthatmanytokensdonotbenefitfromextraiterations.WethereforeproposeTaH2,whichenablesthemodeltofocusextraiterationsonthetokensthatbenefitfromlooping.Itjointlypost-trainsthebackboneandaniterationdeciderthroughlookaheaddepthsupervision,whichusesonlinelabelsindicatingwhetherfurtheriterationimprovestheprediction.TaH2improvesboththeefficiencyandattainableaccuracyoftest-timescaling.OnchallengingAIMEbenchmarks,TaH2improvestheaccuracy-computeslopeby53%(2.74vs.1.79)overthenon-loopedbaseline,exceedingthebaseline’speakaccuracybyabout3.4pointsatmatchedtest-timecompute.Asthemaximumiterationdepthincreases,existingloopedmodelslargelyplateau,whileTaH2’sgainoverthenon-loopedbaselinecontinuestogrowfrom+2.8pointsatdepth2to+3.9pointsatdepth8.Ourcodeisavailableathttps://github.com/thu-nics/TaH.

View arXiv pageView PDFProject pageGitHub85Add to collection

Get this paper in your agent:

hf papers read 2609\.35748

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.35748 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.35748 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.35748 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Scheduling Recursive Reasoning in Looped Transformers

Hugging Face Daily Papers

The paper proposes TAPS (Trajectory Adaptive Progress–Fluctuation Scheduler), which adaptively adjusts the step size of recurrent updates in looped transformers by decomposing terminal loss sensitivity into persistent progress and centered fluctuation, improving accuracy without retraining and delivering up to 1.56× wall-clock speedup when applied during training.

Agentic Test-Time Scaling (GitHub Repo)

TLDR AI

AutoTTS is an open-source tool that uses agentic discovery to automatically find optimal test-time scaling strategies for LLMs, significantly reducing token usage and cost through replay-based evaluation.

Decoding Looped Transformers Better for (Almost) Free

Hugging Face Daily Papers

The paper introduces LoopCD, a training-free contrastive decoding framework for looped transformers that uses intermediate recurrent states as weak/strong prediction pairs, delivering substantial gains (e.g., Ouro-2.6B-Thinking's AIME 2024 pass@1 rising from 61.88% to 73.33%) while enabling halving of recurrent loops and 22.5%-48.2% reductions in forward FLOPs.