Tag
This paper investigates component roles in warm-start transfer for grokking in neural networks, showing that transferring internal weights improves early performance but risks instability, and proposes methods to stabilize the process.
The paper demonstrates that standard neural scaling law derivations fail when data forms group orbits, as intrinsic dimension is undefined, leading to exponential rather than power law scaling in model performance.
The article investigates if regex can match valid credit card numbers by constructing a Deterministic Finite Automaton (DFA) based on the Luhn algorithm, resulting in a massive but executable regex pattern.
This paper introduces CvAdamW, an AdamW variant that monitors attention specific heat to detect grokking phase transitions and dynamically scales weight decay, achieving grokking on modular arithmetic where baseline fails.
This paper ablates the Muon optimizer to find that orthogonalization (Newton-Schulz iteration), not spectral scaling, is the key ingredient behind its faster grokking on modular arithmetic, and introduces a stability-aware metric for measuring grokking speed.
This paper audits the measurement validity of representation metrics in grokking, showing that values at the grokking transition overstate converged circuit complexity and that compression lags generalization. It provides tooling to separate onset from compression and reports negative results on generality.
Introduces cross-trajectory chimera interventions to dissociate the roles of weight magnitude and direction in grokking, showing that direction carries transferable circuit identity while norm affects susceptibility to overwriting.
This paper investigates how weight decay acts as a control parameter for transitioning between memorization and generalization in transformers trained on modular arithmetic, and introduces two cheap online diagnostic metrics from attention activations that track these dynamics.
Study reveals that diverse language-model architectures independently evolve similar periodic Fourier features for representing numbers, with only some achieving geometric separability for modular arithmetic.