Tag
This paper analyzes off-policy distillation for LLM pre-training, characterizing how training objectives shape token-level supervision and downstream capabilities, and proposes adaptive objective routing that applies different objectives to different data domains, reframing pre-training as a data-conditional supervision design problem.