llm-pre-training

This paper investigates Hidden Layer Distillation (HLD) for Large Language Model pre-training, comparing it against standard logit-based knowledge distillation using Gemma3. The study finds that while HLD does not consistently outperform standard methods on downstream tasks, it yields systematic perplexity gains, suggesting potential for future improvements in extracting latent signals during pre-training.