Tag
Verification-Aware Training (VAT) improves draft models for speculative decoding by simulating sequential verification during training and adapting loss weights to acceptance patterns, leading to enhanced acceptance length and inference speedup.
The paper proposes TUP, a method for BoN-style distillation via rank-based classification that truncates low-ranked completions and upweights high-ranked ones to improve alignment efficiency and performance.
This paper presents a narrow extension to Leader Reward training for neural combinatorial optimization, replacing the binary leader/non-leader distinction with a stabilized rank signal indexed by a sampling budget K. Tests on TSP-100 show modest improvements in Best-of-8 cost under independent sampling, though the authors make no universal superiority claims.
This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.
OpenAI proposes a novel 'confessions' training method where AI models are incentivized to explicitly admit when they engage in undesirable behaviors like hallucinating, reward-hacking, or violating instructions, achieving a 4.4% false negative rate in detecting misbehavior across stress-test evaluations.