initialization

TaRA is a training-aware initialization method for Low-Rank Adaptation (LoRA) that improves gradient fidelity, leading to better fine-tuning performance for large language models across various tasks.

This paper studies what transfers between transformer models of different sizes in the same family (Pythia), showing that representations align while weights don't, and that conversion works best via initialization rather than direct weight projection.

Hazy Research showed that an MLP can be initialized with embedded knowledge without training, and a transformer can then query and use that knowledge, hinting at continual learning capabilities.

This paper analyzes spectral patterns in pretrained GPT-2-style language models and tests whether these patterns can be used for initialization, finding that coarse spectral matching does not improve pretraining performance over standard methods.

S-GAI is a spectral geometry-aware initialization framework for one-hidden-layer sigmoidal MLPs that uses class-wise spectral geometry from image data to initialize weights, outperforming random initialization in terms of starting hidden state quality and achieving comparable final accuracy on benchmarks like MNIST and CIFAR-10.

Taylor-Calibrate proposes a principled initialization method for hybrid linear attention models that significantly improves the efficiency of distilling pretrained Transformers into Gated DeltaNet students, achieving up to 88x improvement and reducing training tokens by 4.9x-9.2x.

A deep dive into what happens before the main function in Rust binaries, exploring runtime initialization, entry points, and novel techniques for mutable data initialization.

This paper introduces the concept of 'initialization memory' to study how much of the random initialization bias survives training in deep networks, showing that low-learning-rate SGD preserves initialization while Adam-family optimizers erase it, and linking this to forgetting dynamics.