Tag
The paper introduces an information-efficiency ratio (IER) for optimizing token selection in sparse on-policy distillation, demonstrating that using only 1% of tokens can achieve performance comparable to full supervision.
The paper identifies Value Flattening as a failure mode in PPO critic learning for LLMs and introduces SP3O, a sparse supervision method, to mitigate it, showing consistent improvements in experiments.
Proposes CodeBlock, a structure-aware sparse supervision framework for supervised fine-tuning of code LLMs. It selects high-quality instruction-response pairs and partitions code responses into syntactically coherent coding items, applying loss only to selected items to achieve stronger pass@1 rates using only 1.9% of supervised response tokens.