looped-transformers

Tag

Cards List
#looped-transformers

[Discussion] In the age of loops, what do "active parameters" mean?

Reddit r/LocalLLaMA ↗ · 21h ago

A discussion arguing how to define "active parameters" for looped (weight-recycled) transformer models, proposing that active parameters should scale with loop iterations to preserve their meaning as estimates of GB/token and FLOPs/token, illustrated with examples like Inkling 2 Mini, SenseNova-Loop-9B, and Nanbeige4.2-3B.

0 favorites 0 likes
#looped-transformers

@qingke_ai: Speculative decoding for Loop Transformers — 10–14× speedup. The reason isn't a stronger model, it's a change in how the KV cache is moved. Loop Transformers iterate R times with a shared weight block, maintaining a separate KV cache at each step. WaveF…

X AI KOLs Timeline ↗ · yesterday Cached

The paper proposes LoopCache, which accelerates long-context self-speculative decoding for Loop Transformers through confidence-guided dynamic draft updating and loop KV cache scheduling, achieving average speedups of 10.25× and 14.56× over WFD and LoopSpec respectively.

0 favorites 0 likes
#looped-transformers

Latent Fact Composition under Next-Token Training

arXiv cs.CL ↗ · 3d ago Cached

This paper studies whether transformers trained with next-token prediction can compose two-hop facts whose intermediate entity never appears in training, introduces the Baton family of looped transformers that pass intermediate entities as token embeddings, and analyzes why accuracy declines despite adequate fact recall.

0 favorites 0 likes
#looped-transformers

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Hugging Face Daily Papers ↗ · 2026-11-04 Cached

The paper proposes Loopback Transformer, which reinterprets residual connections as memory along network depth via a write–transport–read formulation, storing block updates with geometric forgetting and content-dependent attention. At matched tokens and depth it achieves lower validation loss and higher 13-task zero-shot accuracy than Standard Transformers at 1.4B and 3B while retaining only 68% and 75% of their parameters.

0 favorites 0 likes
#looped-transformers

LSTM-Style Looped Transformers with Shared KV Caches

arXiv cs.LG ↗ · 2026-10-28 Cached

This paper studies an LSTM-style looped transformer that shares both weights and a single key-value cache across its recurrent core, cutting parameters by 40.4% at width 1024 and reducing persistent cache sets from fifteen to four. A consistency loss aligning parallel and recursive training passes removes 81-89% of the continuation-loss penalty, with weight tying proving cheapest at lower token budgets.

0 favorites 0 likes
#looped-transformers

Agreement rather than normalisation keeps depth reusable in looped language models

arXiv cs.LG ↗ · 2026-10-27 Cached

论文研究了循环语言模型中深度可复用性的来源,发现保持循环步数可外推的关键不是循环边界归一化,而是训练与推理时循环拓扑的一致性(agreement);移除 Ouro 模型的边界归一化会在训练预算内将准确率降至零,但从头训练的小模型显示不加归一化反而外推更好。

0 favorites 0 likes
#looped-transformers

How Much Gradient Does Truncated Backprop Lose in Looped Transformers?

arXiv cs.LG ↗ · 2026-10-22 Cached

This paper measures exactly how much weight gradient is lost when training looped transformers via truncated backpropagation across nine models, finding that the lost gradient is largely set near the output, diminishes during fine-tuning, and does not clearly outperform measurement-free cost heuristics like Parcae's rule.

0 favorites 0 likes
#looped-transformers

Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

Hugging Face Daily Papers ↗ · 2026-10-20 Cached

A scaling study over ~600 training runs finds that full BPTT on looped transformers outperforms truncated alternatives at matched compute, with two fully trained loops emerging as the compute-optimal setting that beats non-looped models with ~27% fewer parameters.

0 favorites 0 likes
#looped-transformers

PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking

Hugging Face Daily Papers ↗ · 2026-10-20 Cached

PAR (Progress-Aligned Recurrence) aligns each loop iteration of a looped language model with a step of the task and rescales hidden states to prevent drift, enabling 16× length extrapolation on swap-S5 state tracking with a 0.5B model reaching 95.5% accuracy, far surpassing RLT and CoT baselines.

0 favorites 0 likes
#looped-transformers

Causal Latent Workspaces for Looped Transformers

arXiv cs.CL ↗ · 2026-10-19 Cached

该论文提出因果潜在工作区 Transformer(causal latent workspace Transformer),通过在共享深度的循环块之间加入可读写的前缀专属潜在槽位,提升了循环 Transformer 的语言建模困惑度,并在三个字节级语料上的多数设置中优于五个基线对照。

0 favorites 0 likes
#looped-transformers

LEV: Thinking Fast and Deep with Looped Decision Models

arXiv cs.AI ↗ · 2026-10-16 Cached

LEV 是一种通过循环重跑层而非生成文本来实现可变推理深度的决策模型,在相同权重下随深度提升准确率,并通过按需深度机制节省 15%-24% 计算量,在低延迟下显著优于同等规模的稠密模型。

0 favorites 0 likes
#looped-transformers

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Hugging Face Daily Papers ↗ · 2026-10-14 Cached

论文提出 LatentLoop,通过非对称计算分配(压缩第一次循环访问的序列分辨率、保留第二次全分辨率并将节省的算力用于增大隐藏维度)来扩展循环 MoE 模型,在 1.3B–7B 参数范围内一致降低验证 NLL,相比原始 MoE 和循环基线分别节省约 19.0% 和 7.9% 的计算量。

0 favorites 0 likes
#looped-transformers

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

arXiv cs.CL ↗ · 2026-10-12 Cached

NVIDIA 提出 Elastic Latent Recurrence (ELR) 训练配方,使循环(looped)语言模型在单次循环时达到与同规模非循环模型相当的精度,同时每次额外循环都能以更多推理计算换取精度提升,最终产出可在 1-4 循环间弹性的单一检查点 Koi-4B。

0 favorites 0 likes
#looped-transformers

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

该论文提出 InfiLoop,一种为循环 Transformer 设计的原生残差连接机制,通过内容加权与学习式时间衰减维持循环状态的运行摘要,使模型在数万次测试时迭代中持续提升推理性能,在 Sudoku-Extreme 上达到 97.9% 精度、ARC-AGI-2 上 13.6% pass@2。

0 favorites 0 likes
#looped-transformers

Microsoft confirms OpenAI has been using Looped Transformers in the GPT-6 series

Reddit r/LocalLLaMA ↗ · 2026-10-06

Microsoft confirmed on a public page that OpenAI uses Looped Transformers in the GPT-6 series, with GPT-6.1 Sol running 2 inference passes instead of three, though Microsoft later removed the page.

0 favorites 0 likes
#looped-transformers

@mylifcc: The so-called large model upgrade you imagine: Buy more H100s, spend millions of dollars retraining, pile parameters fr…

X AI KOLs Timeline ↗ · 2026-10-06 Cached

Apple researchers' paper 'Decoding Looped Transformers Better for (Almost) Free' shows that reusing discarded intermediate inference states via Contrastive Decoding cuts compute consumption by 47% and boosts math pass@1 from 61% to 73%, with zero training cost — a shift toward algorithmic fine-tuning over massive hardware scaling.

0 favorites 0 likes
#looped-transformers

SanSi: A Looped Typed Decision Model for System 1.5 Thinking

Hugging Face Daily Papers ↗ · 2026-10-06 Cached

SanSi 将预训练的循环语言模型转化为类型化决策模型,通过在单次读出前递归复用相同层数层(即 System 1.5 思考)提升决策准确率,在 10,027 个测试决策上达到 72.0% 准确率,并可作为 RL 策略优化的判别器提升生成器 F1。

0 favorites 0 likes
#looped-transformers

The Surprising Effectiveness of Shared Memory in Looped Transformers

arXiv cs.LG ↗ · 2026-10-05 Cached

An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.

0 favorites 0 likes
#looped-transformers

On the Chain-of-Thought Monitorability of Looped Language Models

arXiv cs.AI ↗ · 2026-10-05 Cached

This paper presents the first systematic evaluation of chain-of-thought monitorability in looped language models (LoopLMs), finding that deeper loop depths can reduce CoT monitorability on some tasks under stress tests, while looped architectures are not systematically less monitorable than size/depth-matched non-looped models.

0 favorites 0 likes
#looped-transformers

LiFT: Loop Flow Transformers

Hugging Face Daily Papers ↗ · 2026-10-04 Cached

LiFT introduces a family of looped generative models that scale computation by repeatedly applying a shared Diffusion Transformer (DiT) core, training each loop step on a continuous-depth flow-matching regression target that allows inference far beyond training depth. On ImageNet 256x256, LiFT-L/2 beats a dense DiT-XL/2 baseline by 3.34 FID points while using ~60% fewer parameters and 52% fewer inference FLOPs.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback