convergence

This paper sharpens the convergence analysis of LoRA, improving deterministic oracle complexity from exponential to O(epsilon^{-4}), and proposes stochastic variants LoRA-NSGDM and LoRA-STORM with improved oracle complexities of O(epsilon^{-8}) and O(epsilon^{-6}) respectively.

This paper investigates when weight-tied looped transformers implement actual algorithms, introducing the budget law and showing mechanisms are portable across training budgets, with implications for adaptive computation and interpretability.

This paper introduces the One-Word Census, a minimal instrument to measure answer-choice convergence across 44 language models, finding extreme conformity (e.g., 94% choose 'oak' for tree) and variation in divergence across model families and generations.

A treatise exploring the convergence of intelligence and the economy, available in multiple formats including text, audio, and video.

This paper provides worst-case convergence analyses for Schedule-Free gradient descent and stochastic gradient descent in nonconvex optimization, establishing optimal rates and strict-saddle avoidance, thus theoretically justifying their empirical success.

This paper analyzes the effect of structural and temporal heterogeneities in decentralized federated learning over temporal networks, showing that ignoring these heterogeneities leads to unrealistically rapid convergence and that real-world networks slow down diffusion.

The author shares thoughts on making convergence a reliable halting signal for iterative weight-tied models, discussing tricks from papers like DEQ, Huggin, Ouro, and EqR, and highlighting the roles of pre-norm and input injection.

This paper presents counterexamples showing that Monte Carlo Exploring Starts can converge to suboptimal solutions in tabular reinforcement learning, and provides a modification that guarantees convergence to optimality by scaling learning rates inversely to update frequencies.

AC-ODM uses reinforcement learning to dynamically optimize pretraining data composition for LLMs, achieving faster convergence and higher downstream accuracy with negligible computational overhead.

This paper introduces the Hierarchical Emergence Framework (HEF), which explains how diverse systems such as neural networks and biological evolution converge to similar internal representations through phase transitions in mechanism landscapes under physical and informational constraints. The framework is validated empirically with 111 grokking experiments that confirm universal convergence and identify a critical energy threshold.

Proposes SSD-FL, a serverless semi-decentralized federated learning methodology that optimizes cluster formation in heterogeneous environments using effective loss functions and Cheeger inequality-based iterative clustering, improving convergence and communication efficiency.

This paper generalizes non-uniform smoothness assumptions to objectives whose curvature is affine in the objective value, proving convergence rates for steepest descent and diagonal variants of RMSProp and Adam, with applications to logistic regression and neural networks.

This paper presents a unified theoretical framework for gradient aggregation in multi-objective optimization, establishing convergence rates to Pareto stationarity. The authors introduce a sufficient alignment condition and demonstrate its application to existing and new algorithms, such as capped MGDA.

A comprehensive research guide from Veso detailing the universal architecture patterns that have converged across major AI agent systems (Claude Code, OpenAI Codex, Gemini CLI, etc.), presenting 8 postulates for building production-grade agentic systems.

This paper introduces DynMuon, a dynamic spectral shaping optimizer that schedules the update parameter p from positive to mildly negative during training, consistently achieving lower validation loss and requiring 10.6-26.5% fewer steps than the standard Muon optimizer.