Tag
The paper reveals that the growth of Weibull weight-scale in transformer training is determined by data predictability, specifically through bigram conditional entropy, and establishes a predictive law for this growth.