Tag
This paper develops a statistical-mechanical framework for analyzing learning dynamics in deep neural networks by shifting from parameter space to function space, deriving exact error dynamics and fluctuation-induced effects.
The paper introduces Fora (Function-space Orthogonal Residual Adaptation), a method to protect existing capabilities during fine-tuning by projecting updates onto function-space directions derived from activations rather than weight-space directions. Experiments on Qwen3-1.7B show it outperforms weight-space projection and standard regularization in preserving translation and math abilities.