Tag
The paper formulates data mixing for large language model pretraining as a mixture experiment, applying response surface methodology and optimal experimental design to improve the efficiency and interpretability of proxy training runs.