Tag
The paper discovers "forking," a generalization failure in NanoGPT autoresearch where training and validation loss sharply diverge at epoch boundaries when models with an over-encoding n-gram memory branch are trained under data replay. It mechanistically explains the phenomenon and cautions about unintended side effects of techniques produced by autoresearch agents.
IntologyAI releases NanoGPT-Bench, an internal benchmark to evaluate coding agents on AI R&D tasks. Current agents recover only 9.3% of human progress, mostly through hyperparameter tuning, highlighting gaps in algorithmic research capabilities.