Tag
This paper presents the first systematic evaluation of chain-of-thought monitorability in looped language models (LoopLMs), finding that deeper loop depths can reduce CoT monitorability on some tasks under stress tests, while looped architectures are not systematically less monitorable than size/depth-matched non-looped models.
LiFT introduces a family of looped generative models that scale computation by repeatedly applying a shared Diffusion Transformer (DiT) core, training each loop step on a continuous-depth flow-matching regression target that allows inference far beyond training depth. On ImageNet 256x256, LiFT-L/2 beats a dense DiT-XL/2 baseline by 3.34 FID points while using ~60% fewer parameters and 52% fewer inference FLOPs.
The paper proposes LOOM, a scalable recipe for Looped Mixture-of-Experts models that stabilizes recurrence via residual scaling and input re-injection while diversifying expert usage with per-loop routers, achieving stable scaling to 9-12 loops and improved perplexity and zero-shot accuracy on 100M-1.7B models.
The paper introduces LoopCD, a training-free contrastive decoding framework for looped transformers that uses intermediate recurrent states as weak/strong prediction pairs, delivering substantial gains (e.g., Ouro-2.6B-Thinking's AIME 2024 pass@1 rising from 61.88% to 73.33%) while enabling halving of recurrent loops and 22.5%-48.2% reductions in forward FLOPs.
Introduces Loop Scaling Laws, the first scaling law jointly modeling recurrence (looped transformers) and sparsity (MoE) alongside model size and data, showing looped MoEs match models ~2x larger on reasoning benchmarks at matched compute.
The paper proposes TAPS (Trajectory Adaptive Progress–Fluctuation Scheduler), which adaptively adjusts the step size of recurrent updates in looped transformers by decomposing terminal loss sensitivity into persistent progress and centered fluctuation, improving accuracy without retraining and delivering up to 1.56× wall-clock speedup when applied during training.
A preprint introducing Foil, which flattens MoE experts into fewer layers with more experts per layer and more recurrent passes, plus untied attention per pass, achieving lower pretraining loss at matched parameters and compute.
This paper introduces TaH2, an adaptive looped transformer that improves test-time scaling by dynamically allocating extra iterations to beneficial tokens, achieving a 53% improvement in accuracy-compute slope over baselines on benchmarks like AIME.
The article discusses OpenAI's GPT-6 Astra model release, its exceptional performance in benchmarks like ARC-AGI-3, and explores looped transformers and hidden reasoning concepts.
A comprehensive write-up explaining looped transformers and recurrent depth in AI, including cost-tradeoffs and whether they hide reasoning traces, with figures and a review of recent research.
This paper investigates whether the global workspace functionality in transformers persists under recurrence by applying Jacobian lens analysis to looped and recurrent models, finding that recurrence alters workspace accessibility and content transport.
The article debunks the hype around OpenAI's Astra model, explaining that the 'looped transformer' concept is a minor architectural tweak involving layer reuse for increased capacity without added parameters, as seen in models like Nanbeige 4.2.
OpenAI's upcoming Astra model reportedly uses looped transformers without an interpretable chain of thought, indicating faster AI progress than predicted by AI 2027 forecasts.
The paper proposes Source-Centered State Evolution (SCSE), a method for looped Transformers that reconciles input conditioning with reference-preserving shared recurrence, improving recurrent quality frontiers across multiple benchmarks.
Looped transformers reuse the same layers across multiple passes to trade parameter count for compute, achieving better reasoning with fewer weights. The article traces the idea back to the Universal Transformer (2018) and explains why it initially failed due to scaling laws and timing.
This paper investigates when weight-tied looped transformers implement actual algorithms, introducing the budget law and showing mechanisms are portable across training budgets, with implications for adaptive computation and interpretability.
This paper studies adaptive depth in looped Transformers by separating trajectory formation from exit readout, showing that fixed-prior depth supervision often outperforms jointly trained gates and that poor adaptive-compute performance stems more from the induced trajectory than from gate expressivity.
DeepLoop introduces a residual scaling method for looped Transformers that adjusts for parameter visits, improving stability and performance when physical blocks are reused across multiple rounds.
A new paper on Looped Transformers finds that a looped padded backbone provides a parallel workspace for latent reasoning, enabling supervision similar to explicit chain-of-thought (CoT) and achieving both speed and accuracy.
This paper introduces G-RRM, a neuro-symbolic approach that uses recurrent reasoning models to guide symbolic solvers for constraint satisfaction problems, showing significant speedups in certain conditions.