On-Policy Delta Distillation for Multilingual Math Reasoning

Hugging Face Daily Papers Papers
distillation math-reasoning multilingual llm on-policy qwen3

Summary

This paper studies On-Policy Delta Distillation (OPD^2) for multilingual math reasoning in English, Korean, and Japanese, showing consistent improvements over standard OPD and narrowing language gaps.

On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD^2), for mathematical reasoning in English, Korean, and Japanese. OPD^2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD^2 consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:56 AM

Paper page - On-Policy Delta Distillation for Multilingual Math Reasoning

Source: https://huggingface.co/papers/2608.05802

Abstract

On-PolicyDistillation(OPD)isemergingasapromisingalternativetoreinforcementlearningforLLMpost-training,yetitseffectivenessinmultilingualsettingsremainsunderexplored.WestudyOPDanditsadvancedvariant,On-PolicyDeltaDistillation(OPD^2),formathematicalreasoninginEnglish,Korean,andJapanese.OPD^2improvesOPDbyusingtheprobabilitygapbetweenapost-trainedteacheranditsbasemodelasthelearningsignal.ExperimentswithQwen3showthatOPD^2consistentlyoutperformstheoriginalOPD,withparticularlystrongimprovementsinKoreanandJapanese,andgenerallynarrowstheEnglish-Koreanperformancegap.WefurtherfindthatEnglish-onlyOPDcanalsoincreaseperformanceforKoreanandJapanese,butoftenshiftstheresponsestowardEnglish,highlightingtheimportanceofmultilingualdatatopreservingtarget-languageresponses.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.05802

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.05802 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.05802 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.05802 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

On-Policy Delta Distillation

Hugging Face Daily Papers

The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.

Crosslingual On-Policy Self-Distillation for Multilingual Reasoning

Hugging Face Daily Papers

The paper proposes Crosslingual On-Policy Self-Distillation (COPSD), a method to transfer high-resource language reasoning capabilities to low-resource languages using a shared student-teacher architecture. Experiments across 17 African languages show significant improvements in mathematical reasoning and answer-format adherence, outperforming Group Relative Policy Optimization (GRPO).

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.

OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification

Hugging Face Daily Papers

OmniOPD introduces a logit-free on-policy distillation method that uses chunk-level semantic similarity and speculative verification to train student models with black-box teachers, achieving up to +28.64% improvement on math benchmarks over standard OPD.