Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Summary
Self-Distillation Zero (SD-Zero) is a novel training method that converts sparse binary rewards into dense token-level supervision through dual-role training where a model acts as both generator and reviser, achieving 10%+ improvements on math and code reasoning benchmarks with higher sample efficiency than RL approaches.
View Cached Full Text
Cached at: 04/20/26, 08:29 AM
Paper page - Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
Source: https://huggingface.co/papers/2604.12002
Abstract
Self-Distillation Zero trains a model to transform binary rewards into dense token-level self-supervision through dual-role training and on-policy self-distillation, achieving superior performance in reasoning tasks with reduced sample efficiency requirements.
Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (https://huggingface.co/papers?q=Reinforcement%20learning)(RLVR) relies on binary rewards (https://huggingface.co/papers?q=binary%20rewards), which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation (https://huggingface.co/papers?q=Distillation) provides dense token-level supervision (https://huggingface.co/papers?q=token-level%20supervision), typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation (https://huggingface.co/papers?q=Self-Distillation)Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation (https://huggingface.co/papers?q=on-policy%20self-distillation) to distill the reviser into the generator, using the reviser’s token distributions conditioned on the generator’s response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards (https://huggingface.co/papers?q=binary%20rewards) into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation (https://huggingface.co/papers?q=Self-Distillation)Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization (https://huggingface.co/papers?q=token-level%20self-localization), where the reviser can identify the key tokens that need to be revised in the generator’s response based on reward, and (b) iterative self-evolution (https://huggingface.co/papers?q=iterative%20self-evolution), where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization (https://huggingface.co/papers?q=teacher%20synchronization).
View arXiv page (https://arxiv.org/abs/2604.12002)View PDF (https://arxiv.org/pdf/2604.12002)Add to collection (https://huggingface.co/login?next=%2Fpapers%2F2604.12002)
Get this paper in your agent:
hf papers read 2604.12002
Don’t have the latest CLI?curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.12002 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.12002 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.12002 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to a collection (https://huggingface.co/new-collection) to link it from this page.
Similar Articles
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
This paper proposes Rubric-Conditioned Self-Distillation (RCSD), a framework that uses fine-grained rubric criteria to provide token-level guidance during self-distillation, improving reasoning performance over scalar-reward methods like GRPO and OPSD.
Self-Distilled Agentic Reinforcement Learning
SDAR enhances multi-turn agent training by integrating self-distillation with a sigmoid gate to selectively strengthen positive token-level guidance while mitigating negative teacher rejections, achieving significant improvements over GRPO across multiple benchmarks.
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
Proposes Anti-Self-Distillation (AntiSD) which reverses the knowledge transfer direction in self-distillation to improve math reasoning efficiency and accuracy, achieving GRPO baseline accuracy in 2-10x fewer steps and up to 11.5 points higher final accuracy across models from 4B to 30B parameters.
G-Zero: Self-Play for Open-Ended Generation from Zero Data
This paper introduces G-Zero, a verifier-free framework that enables autonomous large language model self-improvement through co-evolutionary training using intrinsic rewards and hint-based guidance. It aims to overcome the limitations of proxy LLM judges in open-ended tasks by deriving supervision from internal distributional dynamics.
@rosinality: https://arxiv.org/abs/2607.12395 RL without SFT. What could be an interesting point is that they were able to make reas…
The paper 'Ring-Zero' presents a stable training pipeline for scaling zero reinforcement learning (without supervised fine-tuning) to 1 trillion parameters, achieving emergent reasoning capabilities such as self-verification and structured formatting, and demonstrating strong performance on mathematical benchmarks.