Tag
This paper studies how unlabeled, AI-generated web text affects language model pretraining by training 800 models and fitting new scaling laws. It finds AI text helps data-starved models initially but quickly turns harmful for human-heavy budgets, proposes a scaling law with separate benefit and harm terms that outperforms existing laws, and releases the WildAI 83B-token corpus and all models.