riemannian-optimization

This paper introduces Stiefel Attention, which constrains transformer query and key projection matrices to the Stiefel manifold using Riemannian optimization, demonstrating improved performance on modular arithmetic grokking and CIFAR-10 patches.

This paper presents distributed Riemannian online gradient descent on Hadamard manifolds with curvature-independent regret bounds for horospherical convex functions, achieving rates matching Euclidean optimization.

GeoSteer is an optimization-based method for norm-preserving activation steering in large language models, using geodesic updates on the representation manifold to improve control over model behavior while preserving activation norms.