Tag
This paper introduces CvAdamW, an AdamW variant that monitors attention specific heat to detect grokking phase transitions and dynamically scales weight decay, achieving grokking on modular arithmetic where baseline fails.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This paper audits the measurement validity of representation metrics in grokking, showing that values at the grokking transition overstate converged circuit complexity and that compression lags generalization. It provides tooling to separate onset from compression and reports negative results on generality.
Introduces cross-trajectory chimera interventions to dissociate the roles of weight magnitude and direction in grokking, showing that direction carries transferable circuit identity while norm affects susceptibility to overwriting.
This paper investigates how weight decay acts as a control parameter for transitioning between memorization and generalization in transformers trained on modular arithmetic, and introduces two cheap online diagnostic metrics from attention activations that track these dynamics.
Study reveals that diverse language-model architectures independently evolve similar periodic Fourier features for representing numbers, with only some achieving geometric separability for modular arithmetic.