Tag
该论文定位了LLM持续预训练/微调中灾难性遗忘的根源——语料库中罕见词元的输出嵌入，并提出仅提升输出投影层Adam ε值的单行优化器改动，在160M至12B参数规模上消除了39.4%至67.9%的遗忘且不损害目标学习。
This paper provides a theoretical analysis of the edge-of-stability phenomenon for the Adam optimizer on a one-dimensional quadratic function, proving that Adam exhibits a restoring mechanism that pushes sharpness toward a stability threshold.
This tweet lists 7 fundamental math ideas used daily by ML engineers, with brief explanations emphasizing the underlying derivations, such as why gradient descent moves in the steepest direction and why softmax plus cross-entropy yields a clean gradient.