Improving Test-Time Scaling with Adaptive Looped Transformers
Summary
This paper introduces TaH2, an adaptive looped transformer that improves test-time scaling by dynamically allocating extra iterations to beneficial tokens, achieving a 53% improvement in accuracy-compute slope over baselines on benchmarks like AIME.
View Cached Full Text
Cached at: 09/29/26, 08:19 AM
Paper page - Improving Test-Time Scaling with Adaptive Looped Transformers
Source: https://huggingface.co/papers/2609.35748
Abstract
Loopedtransformershavedemonstratedpromisingparameterefficiencybyreusinglayersforlatentcomputation.Priorstudiescompareloopedandnon-loopedmodelsatmatchedparametersorper-tokenFLOPs.However,tothebestofourknowledge,whetherloopingimprovestest-timescalingasoutputsgrowlongerremainsunderexplored.Throughpost-trainingloopedtransformers,westudytheaccuracy-computeslope,measuredastheaccuracygainperdoublingoftest-timedecodingFLOPs.Wefindthatexistingloopedtransformersoftenyieldsteeperslopesthantheirnon-loopedbaseline,yetunderperformitatmatchedcompute.Whilefixed-depthloopingspendsextraiterationsoneverytoken,ouranalysisshowsthatmanytokensdonotbenefitfromextraiterations.WethereforeproposeTaH2,whichenablesthemodeltofocusextraiterationsonthetokensthatbenefitfromlooping.Itjointlypost-trainsthebackboneandaniterationdeciderthroughlookaheaddepthsupervision,whichusesonlinelabelsindicatingwhetherfurtheriterationimprovestheprediction.TaH2improvesboththeefficiencyandattainableaccuracyoftest-timescaling.OnchallengingAIMEbenchmarks,TaH2improvestheaccuracy-computeslopeby53%(2.74vs.1.79)overthenon-loopedbaseline,exceedingthebaseline’speakaccuracybyabout3.4pointsatmatchedtest-timecompute.Asthemaximumiterationdepthincreases,existingloopedmodelslargelyplateau,whileTaH2’sgainoverthenon-loopedbaselinecontinuestogrowfrom+2.8pointsatdepth2to+3.9pointsatdepth8.Ourcodeisavailableathttps://github.com/thu-nics/TaH.
View arXiv pageView PDFProject pageGitHub85Add to collection
Get this paper in your agent:
hf papers read 2609\.35748
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.35748 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.35748 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.35748 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Scheduling Recursive Reasoning in Looped Transformers
The paper proposes TAPS (Trajectory Adaptive Progress–Fluctuation Scheduler), which adaptively adjusts the step size of recurrent updates in looped transformers by decomposing terminal loss sensitivity into persistent progress and centered fluctuation, improving accuracy without retraining and delivering up to 1.56× wall-clock speedup when applied during training.
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
该论文提出 InfiLoop,一种为循环 Transformer 设计的原生残差连接机制,通过内容加权与学习式时间衰减维持循环状态的运行摘要,使模型在数万次测试时迭代中持续提升推理性能,在 Sudoku-Extreme 上达到 97.9% 精度、ARC-AGI-2 上 13.6% pass@2。
LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling
LoopCoder-v2 proposes Parallel Loop Transformers (PLT) for efficient test-time computation scaling in code generation, showing that two loops yield significant gains while more loops cause diminishing returns and positional mismatch costs.
Agentic Test-Time Scaling (GitHub Repo)
AutoTTS is an open-source tool that uses agentic discovery to automatically find optimal test-time scaling strategies for LLMs, significantly reducing token usage and cost through replay-based evaluation.
Decoding Looped Transformers Better for (Almost) Free
The paper introduces LoopCD, a training-free contrastive decoding framework for looped transformers that uses intermediate recurrent states as weak/strong prediction pairs, delivering substantial gains (e.g., Ouro-2.6B-Thinking's AIME 2024 pass@1 rising from 61.88% to 73.33%) while enabling halving of recurrent loops and 22.5%-48.2% reductions in forward FLOPs.