shampoo

Tag

Cards List
#shampoo

Why don't optimizes like Shampoo, SOAP, and Adam measure the actual per sample second moments like the Fischer Information Matrix does? [d]

Reddit r/MachineLearning ↗ · 2d ago

A technical discussion arguing that Shampoo, Muon, and Adam all drop per-sample covariance terms when approximating second moments (like the Fisher Information Matrix), and questioning why using the true second moments underperforms these approximations.

0 favorites 0 likes
#shampoo

Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storage

arXiv cs.LG ↗ · 2026-05-27 Cached

This paper proposes a reparametrization of the preconditioner in Shampoo-based optimization methods (like KL-Shampoo and SOAP) to support BFloat16 storage and reduce computational overhead by updating only part of the basis via QR decomposition in a subspace, making these methods more memory- and time-efficient.

0 favorites 0 likes
← Back to home

Submit Feedback