weight-initialization

This paper explores using quasi-Monte Carlo (QMC) methods for weight initialization in meta-reinforcement learning, showing improved convergence in similar control tasks compared to orthogonal initialization.

This paper analyzes spectral patterns in pretrained GPT-2-style language models and tests whether these patterns can be used for initialization, finding that coarse spectral matching does not improve pretraining performance over standard methods.

Introduces CAWI, a copula-based weight initialization method for randomized neural networks that models inter-feature dependence, improving predictive performance across 83 classification benchmarks.