@rohanpaul_ai: New Meta paper shows, small models may not be bad predictors of scale; they may just be getting under-tuned. Finds scal…

New Meta paper shows, small models may not be bad predictors of scale; they may just be getting under-tuned. Finds scaling laws emerge around 4M parameters, where models can train in under 1 hour on 1 GPU. Small models are unusually sensitive to hyperparameters. With 4 or 16 configurations per scale, the law is basically invisible; at 64 it appears but extrapolates poorly, and at 256 it becomes accurate. As models get larger, good settings occupy more of the search space, while the effective number of hyperparameters near the optimum drops toward 1. That helps explain why scaling laws look cleaner at larger sizes: the models are easier to tune. As a check, small-scale runs recover that pre-norm transformers scale better than post-norm over the tested range. There is a limit: extrapolate too far beyond the measured scales, and statistical errors can dominate. For model research, cheap experiments may need more tuning breadth, not more model size. – arxiv. org/abs/2608.11859 Title: "Small-Scale Experiments: Are They There Yet?"