Paper page - TREK: Distill to Explore, Reinforce to Refine

Source: https://huggingface.co/papers/2607.05339 Published on Jul 6

·

Submitted byhttps://huggingface.co/xuyd16

XYXon Jul 8

#3 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

TREK expands exploration support for policy optimization by using distillation for exploration rather than imitation, improving performance on challenging mathematical reasoning and agentic tasks.

Group Relative Policy Optimization(GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student’son-policy support. We propose TREK (Teacher-Routed ExplorationviaForward KL), a simple staged procedure that usesdistillationnot for imitation but forexploration support expansion. A key advantage of TREK is its generality: because it only consumesverified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-r proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student’s support, and then returns to standard on-policy GRPO refinement. Onmathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. Onagentic tasks, TREK raisesALFWorldsuccess rate from 75.8 to 82.8 andScienceWorldsuccess rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.05339

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.05339 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.05339 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.05339 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.