LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Hugging Face Daily Papers Papers
reinforcement-learning llm instruction-following prompt-adaptation policy-aware tutor-framework non-verifiable-rl

Summary

LLM-as-a-Tutor introduces a framework that extends LLM's role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.
Original Article
View Cached Full Text

Cached at: 07/09/26, 07:51 AM

Paper page - LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Source: https://huggingface.co/papers/2607.04412

Abstract

LLM-as-a-Tutor framework extends LLM role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.

Reinforcement learning(RL) for non-verifiableinstruction followingincreasingly relies on LLM judges with prompt-specific rubrics asreward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM’s role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appendsatomic constraintsto them. This append-only design monotonically raises difficulty in step with the policy’s capability, producing aself-calibrating training signalwithout external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggestingprompt adaptationas a missing axis of policy-awareness in non-verifiable RL.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.04412 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.04412 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.04412 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles