Loop the Loopies!
Summary
Loopie introduces looped Mixture-of-Experts Transformers that outperform vanilla transformers under the same compute budget, achieving gold-medal performance at the 2025 IMO and IPhO without tools.
View Cached Full Text
Cached at: 07/20/26, 09:40 AM
Paper page - Loop the Loopies!
Source: https://huggingface.co/papers/2607.16051
Abstract
WepresentLoopie,themostpowerfulloopedTransformertodate.TheLoopieseriesconsistsoftwoMixture-of-Experts(MoE)models:a20B-parametermodelwith2Bactiveparametersanda6Bparametermodelwith0.6Bactiveparameters.LoopedTransformershavelongfacedachallenge:givenanN-foldincreaseinpre-trainingcompute,increasingtheparametercountbyafactorofNusuallyoutperformsloopingamodelNtimes.Loopieaddressesthischallenge.Extensiveablationstudies,includingcomparisonswithavanilla30B-A3Bmodel,showthatLoopiesubstantiallyoutperformsvanillaTransformerbaselinestrainedwiththesamecomputebudget.Ournovelpost-trainingpipelineequipsLoopiewithstrongreasoningabilities.Atthe2025IMOandIPhO,Loopieachievesgold-medalperformancewithouttools.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.16051 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.16051 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.16051 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Loop the Loopies!
Loopie is a new looped Transformer model that achieves gold-medal performance at the 2025 IMO and IPhO without external tools, using a novel post-training pipeline. It outperforms vanilla Transformers trained with the same compute budget.
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
SMELT is a method that loops middle layers in Mixture-of-Experts Transformers to improve training efficiency and downstream performance while matching compute, parameter, and cache budgets, leading to faster loss reduction and practical gains.
@DorothyDDU: LoopCoder-v2 is out Loop Transformers reuse the same block for recurrent hidden-state refinement — letting models “thin…
This paper introduces LoopCoder-v2, a family of 7B parameter parallel loop transformers for code generation, and studies the optimal number of loops, finding that two loops yield significant gains while more loops cause degradation.
@askalphaxiv: Another cool research on Looped Transformers They ask the question: "Can we loop a frozen, off-the-shelf checkpoint dir…
This research introduces a technique to loop frozen, off-the-shelf transformer checkpoints at inference time by using damped Runge-Kutta substeps, treating transformer layers as Euler steps in a residual ODE. This allows extra latent compute without fine-tuning, architecture changes, or new weights, showing gains on knowledge tasks like MMLU-Pro, GPQA, and ARC.
20B Looping model (paper) matches or beats Qwen3 Coder 30B at 10% of pre-training tokens
Loopie models use a looped transformer architecture to match or exceed Qwen3 Coder 30B performance with only 10% of the pre-training tokens, demonstrating strong reasoning abilities and efficient scaling.