Tag
This paper audits the measurement validity of representation metrics in grokking, showing that values at the grokking transition overstate converged circuit complexity and that compression lags generalization. It provides tooling to separate onset from compression and reports negative results on generality.
Introduces cross-trajectory chimera interventions to dissociate the roles of weight magnitude and direction in grokking, showing that direction carries transferable circuit identity while norm affects susceptibility to overwriting.
This paper investigates how weight decay acts as a control parameter for transitioning between memorization and generalization in transformers trained on modular arithmetic, and introduces two cheap online diagnostic metrics from attention activations that track these dynamics.
Study reveals that diverse language-model architectures independently evolve similar periodic Fourier features for representing numbers, with only some achieving geometric separability for modular arithmetic.