Tag
This paper introduces Fisher8, an output-layer gradient correction that uses Fisher geometry instead of Euclidean geometry to stabilize neural heteroscedastic regression, improving uncertainty calibration and likelihood-error tradeoffs.
This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.
Introduces Energy Manifold Natural Gradient Descent (EMNGD), a manifold optimization framework for neural PDE solvers that aligns parameter updates with function-space energy curvature while respecting parameter constraints. Theoretical guarantees and empirical results show improved accuracy and convergence.
This paper introduces GeoSD, a geometric self-distillation objective that uses Hellinger loss and a proximal Fisher-Rao distance term to counter drift in on-policy self-distillation, improving out-of-distribution reasoning accuracy by 5.7–8.6 points across model scales.