Tag
A discussion arguing how to define "active parameters" for looped (weight-recycled) transformer models, proposing that active parameters should scale with loop iterations to preserve their meaning as estimates of GB/token and FLOPs/token, illustrated with examples like Inkling 2 Mini, SenseNova-Loop-9B, and Nanbeige4.2-3B.
The paper proposes LoopCache, which accelerates long-context self-speculative decoding for Loop Transformers through confidence-guided dynamic draft updating and loop KV cache scheduling, achieving average speedups of 10.25× and 14.56× over WFD and LoopSpec respectively.
This paper studies whether transformers trained with next-token prediction can compose two-hop facts whose intermediate entity never appears in training, introduces the Baton family of looped transformers that pass intermediate entities as token embeddings, and analyzes why accuracy declines despite adequate fact recall.
The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.
This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.
论文研究了循环语言模型中深度可复用性的来源,发现保持循环步数可外推的关键不是循环边界归一化,而是训练与推理时循环拓扑的一致性(agreement);移除 Ouro 模型的边界归一化会在训练预算内将准确率降至零,但从头训练的小模型显示不加归一化反而外推更好。
This paper measures exactly how much weight gradient is lost when training looped transformers via truncated backpropagation across nine models, finding that the lost gradient is largely set near the output, diminishes during fine-tuning, and does not clearly outperform measurement-free cost heuristics like Parcae's rule.
A scaling study over ~600 training runs finds that full BPTT on looped transformers outperforms truncated alternatives at matched compute, with two fully trained loops emerging as the compute-optimal setting that beats non-looped models with ~27% fewer parameters.
PAR (Progress-Aligned Recurrence) aligns each loop iteration of a looped language model with a step of the task and rescales hidden states to prevent drift, enabling 16× length extrapolation on swap-S5 state tracking with a 0.5B model reaching 95.5% accuracy, far surpassing RLT and CoT baselines.
该论文提出因果潜在工作区 Transformer(causal latent workspace Transformer),通过在共享深度的循环块之间加入可读写的前缀专属潜在槽位,提升了循环 Transformer 的语言建模困惑度,并在三个字节级语料上的多数设置中优于五个基线对照。
LEV 是一种通过循环重跑层而非生成文本来实现可变推理深度的决策模型,在相同权重下随深度提升准确率,并通过按需深度机制节省 15%-24% 计算量,在低延迟下显著优于同等规模的稠密模型。
论文提出 LatentLoop,通过非对称计算分配(压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度)来扩展循环 MoE 模型,在 1.3B–7B 参数范围内一致降低验证 NLL,相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。
NVIDIA 提出 Elastic Latent Recurrence (ELR) 训练配方,使循环(looped)语言模型在单次循环时达到与同规模非循环模型相当的精度,同时每次额外循环都能以更多推理计算换取精度提升,最终产出可在 1-4 循环间弹性的单一检查点 Koi-4B。
该论文提出 InfiLoop,一种为循环 Transformer 设计的原生残差连接机制,通过内容加权与学习式时间衰减维持循环状态的运行摘要,使模型在数万次测试时迭代中持续提升推理性能,在 Sudoku-Extreme 上达到 97.9% 精度、ARC-AGI-2 上 13.6% pass@2。
Microsoft confirmed on a public page that OpenAI uses Looped Transformers in the GPT-6 series, with GPT-6.1 Sol running 2 inference passes instead of three, though Microsoft later removed the page.
Apple researchers' paper 'Decoding Looped Transformers Better for (Almost) Free' shows that reusing discarded intermediate inference states via Contrastive Decoding cuts compute consumption by 47% and boosts math pass@1 from 61% to 73%, with zero training cost — a shift toward algorithmic fine-tuning over massive hardware scaling.
SanSi 将预训练的循环语言模型转化为类型化决策模型,通过在单次读出前递归复用相同层数层(即 System 1.5 思考)提升决策准确率,在 10,027 个测试决策上达到 72.0% 准确率,并可作为 RL 策略优化的判别器提升生成器 F1。
An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.
This paper presents the first systematic evaluation of chain-of-thought monitorability in looped language models (LoopLMs), finding that deeper loop depths can reduce CoT monitorability on some tasks under stress tests, while looped architectures are not systematically less monitorable than size/depth-matched non-looped models.
LiFT introduces a family of looped generative models that scale computation by repeatedly applying a shared Diffusion Transformer (DiT) core, training each loop step on a continuous-depth flow-matching regression target that allows inference far beyond training depth. On ImageNet 256x256, LiFT-L/2 beats a dense DiT-XL/2 baseline by 3.34 FID points while using ~60% fewer parameters and 52% fewer inference FLOPs.