Tag
Loopie is a new looped Transformer model that achieves gold-medal performance at the 2025 IMO and IPhO without external tools, using a novel post-training pipeline. It outperforms vanilla Transformers trained with the same compute budget.
Introduces Looped Latent Attention (LLA), a post-training codec that compresses KV cache in looped transformers by exploiting low-rank structure across recurrence steps, achieving significant compression ratios while maintaining performance.
Loopie introduces looped Mixture-of-Experts Transformers that outperform vanilla transformers under the same compute budget, achieving gold-medal performance at the 2025 IMO and IPhO without tools.
This paper identifies gradient oscillation and residual explosion as causes of training instability in Looped Transformers, and proposes Fully Looped Transformer with two parameter-free modifications (Fully Looped Architecture and Attention Injection) to stabilize training up to 12 loop iterations, achieving up to 13.2% improvement in downstream performance.
Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.