Tag
This paper finds that smaller frozen models can generate better negative samples (rejects) for preference distillation than self-generated responses, reducing inference cost while training stronger 7B–72B students on code generation and math reasoning. The authors derive a finite-horizon utility bound for DPO in a linearized feature model to explain why task structure and lower-likelihood candidate selection improve reject quality.
MINT introduces min-selection preference distillation to balance multiple objectives in language agent alignment by ranking candidates based on their weakest objective, significantly reducing imbalance.
Introduces Implicit Preference Distillation, a method to extract preference signals from version control merge history to align AI agents with institutional practices cheaply.