transformer-pretraining

This paper introduces OPTScientist, a theory-guided multi-agent framework for automatically discovering typed optimizer programs using a domain-specific language and closed-loop experimentation. The framework discovered RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines.