Tag
Hazy Research showed that an MLP can be initialized with embedded knowledge without training, and a transformer can then query and use that knowledge, hinting at continual learning capabilities.
This paper analyzes spectral patterns in pretrained GPT-2-style language models and tests whether these patterns can be used for initialization, finding that coarse spectral matching does not improve pretraining performance over standard methods.
S-GAI is a spectral geometry-aware initialization framework for one-hidden-layer sigmoidal MLPs that uses class-wise spectral geometry from image data to initialize weights, outperforming random initialization in terms of starting hidden state quality and achieving comparable final accuracy on benchmarks like MNIST and CIFAR-10.
Taylor-Calibrate proposes a principled initialization method for hybrid linear attention models that significantly improves the efficiency of distilling pretrained Transformers into Gated DeltaNet students, achieving up to 88x improvement and reducing training tokens by 4.9x-9.2x.
A deep dive into what happens before the main function in Rust binaries, exploring runtime initialization, entry points, and novel techniques for mutable data initialization.
This paper introduces the concept of 'initialization memory' to study how much of the random initialization bias survives training in deep networks, showing that low-learning-rate SGD preserves initialization while Adam-family optimizers erase it, and linking this to forgetting dynamics.