Tag
This paper analyzes how the parameterization of query-key and output-value circuits in self-attention models affects attention sharpness during training. Through gradient-flow analysis, they show that faster query-key learning relative to output-value learning produces sharper attention, improving interpretability without sacrificing predictive performance.
This paper introduces GradCuit, a method for test-time latent reasoning that inserts optimizable latent states at a selected Transformer layer. It achieves 64.5% average accuracy across five backbones and three reasoning benchmarks, outperforming chain-of-thought prompting and showing improved robustness and interpretability.
This paper studies the gradient flow dynamics of diagonal linear networks under infinitesimal initialization, generalizing previous results to deep networks and a broader class. It shows that the implicit bias corresponds to a modified ℓ1 norm and identifies the Structural Invariant Manifold as a key geometric structure.
GPT-5.6 significantly outperforms published state-of-the-art on a fundamental mathematical problem about gradient flow length, achieving exponential improvements. This marks a major advance in AI's ability to reason about complex mathematical questions.
This paper systematically identifies all qualitatively different extreme learning regimes for large weight-tied linear autoencoders, deriving explicit loss evolutions for five regimes associated with the faces of a triangular prism.
Value Gradient Flow (VGF) presents a scalable approach to behavior-regularized reinforcement learning by formulating it as an optimal transport problem solved through discrete gradient flow, achieving state-of-the-art results on offline RL and LLM RL benchmarks. The method eliminates explicit policy parameterization while enabling adaptive test-time scaling by controlling transport budget.