Tag
This paper investigates using symbolic regression to discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks, achieving an aggregate MSE reduction of 44.47% in 25 out of 30 benchmark/network combinations.
This paper introduces OPTScientist, a theory-guided multi-agent framework for automatically discovering typed optimizer programs using a domain-specific language and closed-loop experimentation. The framework discovered RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines.