over-optimization

The paper revisits scaling laws for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}) where M is the amount of preference data and K is the KL-divergence budget, and provides an information-theoretic model plus extensive empirical validation with proxy reward models.