@askalphaxiv: Another cool research on Looped Transformers They ask the question: "Can we loop a frozen, off-the-shelf checkpoint dir…

X AI KOLs Timeline Papers
looped-transformers inference-time-compute frozen-models runge-kutta residual-ode knowledge-tasks

Summary

This research introduces a technique to loop frozen, off-the-shelf transformer checkpoints at inference time by using damped Runge-Kutta substeps, treating transformer layers as Euler steps in a residual ODE. This allows extra latent compute without fine-tuning, architecture changes, or new weights, showing gains on knowledge tasks like MMLU-Pro, GPQA, and ARC.

Another cool research on Looped Transformers They ask the question: "Can we loop a frozen, off-the-shelf checkpoint directly at inference time without any modifications?" So naive repetition pushes hidden states outside the distribution later layers expect, so performance drops. But if you treat transformer layers as Euler steps in a residual ODE and replaces naive loops with damped Runge–Kutta substeps, it is possible. This lets the frozen models get extra latent compute at test time with no fine-tuning, no new weights, and no architecture changes. And the best gains show up on hard knowledge MC tasks like MMLU-Pro, GPQA, and ARC.
Original Article
View Cached Full Text

Cached at: 05/27/26, 03:18 AM

Another cool research on Looped Transformers

They ask the question: “Can we loop a frozen, off-the-shelf checkpoint directly at inference time without any modifications?”

So naive repetition pushes hidden states outside the distribution later layers expect, so performance drops.

But if you treat transformer layers as Euler steps in a residual ODE and replaces naive loops with damped Runge–Kutta substeps, it is possible.

This lets the frozen models get extra latent compute at test time with no fine-tuning, no new weights, and no architecture changes.

And the best gains show up on hard knowledge MC tasks like MMLU-Pro, GPQA, and ARC.

Similar Articles

What Are Looped Transformers? Explained Clearly (8 minute read)

TLDR AI

Looped transformers reuse the same layers across multiple passes to trade parameter count for compute, achieving better reasoning with fewer weights. The article traces the idea back to the Universal Transformer (2018) and explains why it initially failed due to scaling laws and timing.

DeepLoop: Depth Scaling for Looped Transformers

arXiv cs.LG

DeepLoop introduces a residual scaling method for looped Transformers that adjusts for parameter visits, improving stability and performance when physical blocks are reused across multiple rounds.

Simply Stabilizing the Loop via Fully Looped Transformer

arXiv cs.LG

This paper identifies gradient oscillation and residual explosion as causes of training instability in Looped Transformers, and proposes Fully Looped Transformer with two parameter-free modifications (Fully Looped Architecture and Attention Injection) to stabilize training up to 12 loop iterations, achieving up to 13.2% improvement in downstream performance.