Tag
This paper systematically isolates recurrence design axes—topology, input injection, and recurrent-state organization—in looped language models, finding that a prelude-recurrent block-coda sandwich architecture (Huginn-style) outperforms Ouro on reasoning and knowledge benchmarks and transfers its advantages to parameter-efficient MoE models with 75% fewer resident parameters.
The paper introduces triadic linear attention, which generalizes linear attention by writing a triadic outer product of two keys and a value into a third-order tensor state, enabling an E-fold state-size increase with only two extra projections. Applied to Gated DeltaNet and scalar-gated linear attention, it improves long-context language modeling and recall, outperforming alternative state-enlargement methods.
The paper establishes a theoretical connection between probabilistic Joint-Embedding Predictive Learning (JEPA) and Hidden Markov Models (HMMs), providing a state-space interpretation and introducing Markov-Chain JEPA for enhanced consistency.
This paper develops a local theory of gradient descent near bifurcations in dynamical models, showing that the state-space neural tangent kernel collapses to a rank-one operator that dominates learning dynamics, making optimization effectively low-dimensional and predictable from normal forms.