LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
Summary
LLM-as-a-Tutor introduces a framework that extends LLM's role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.
View Cached Full Text
Cached at: 07/09/26, 07:51 AM
Paper page - LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
Source: https://huggingface.co/papers/2607.04412
Abstract
LLM-as-a-Tutor framework extends LLM role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.
Reinforcement learning(RL) for non-verifiableinstruction followingincreasingly relies on LLM judges with prompt-specific rubrics asreward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM’s role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appendsatomic constraintsto them. This append-only design monotonically raises difficulty in step with the policy’s capability, producing aself-calibrating training signalwithout external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggestingprompt adaptationas a missing axis of policy-awareness in non-verifiable RL.
View arXiv pageView PDFAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.04412 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.04412 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.04412 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring
This paper proposes an adaptive, subject-aware prompt routing framework for LLM-based high-school tutoring, using 14 pedagogical features to switch strategies. A/B testing with 359 students shows improved efficiency and conversion rates over static baselines.
@ethantsliu: meta-rl induces agentic LLM exploration traditional RL trains LLM agents to use fixed policies that struggle with activ…
A new research paper reframes LLM agent training as a cross-episode Meta-RL problem, using critic-free policy gradients to enable in-context adaptation without gradient updates. The LAMER framework improves test-time performance by 11-19% over standard RL baselines on long-horizon tasks and generalizes better to unseen environments.
Reflective Dialogue or Prompt Refinement? Effects of Tutor Scaffolding on Students' Independent LLM Use for Programming
This study compares two LLM-based tutoring approaches (Socratic guidance vs prompt refinement) for programming education, finding that Socratic guidance fosters better learning outcomes and more understanding-driven prompting strategies when students later use unconstrained LLMs.
LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
LLMZero uses LLM agents to search over training trajectories via tree search, discovering adaptive multi-parameter transitions for RL post-training that outperform fixed schedules and grid search across diverse tasks.
LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks
This paper introduces Experiential Learning (EL), a method that repurposes an LLM-as-a-Judge into an LLM-as-a-Coach to provide rich textual feedback instead of scalar rewards, improving performance and generalization on open-ended non-verifiable tasks.