@itarutomy: Microsoft Research researchers propose a method called "Experiential Learning..." that replaces RL (reinforcement learning) rewards in LLM post-training (post-training, additional fine-tuning after pre-training) with "coach's advice" instead of "scores"

X AI KOLs Timeline Papers
reinforcement-learning post-training llm experiential-learning microsoft-research arxiv

Summary

Researchers at Microsoft Research propose "Experiential Learning (EL)," which replaces RL rewards with "coach's advice" instead of scalar scores in LLM post-training. In experiments with Qwen3-8B and OLMo-3-7B, it consistently outperformed standard rubric-based RL (GRPO) on multiple benchmarks.

Researchers from Microsoft Research have proposed a method called "Experiential Learning (EL)," which replaces RL (reinforcement learning) rewards in LLM post-training (i.e., additional fine-tuning after pre-training) with "coach's advice" instead of "scores" (https://arxiv[.]org/html/2607.18110v1). Unlike "verifiable tasks" such as math or code, which have a single correct answer, in RL for "non-verifiable tasks" that measure text quality, the grading LLM (LLM-as-a-Judge) compresses responses into scalar (single numerical) rewards of 1–10 points according to a rubric (evaluation criteria) before using them for training. The rich information in the grading rationale is discarded here, making it a weakness that policy models can no longer distinguish between responses that receive the same score. EL repurposes the same grading LLM as a "coach." Instead of outputting scores, it extracts advice sentences (experiential knowledge) such as "use specific and diverse adjectives," which can be transferred to similar tasks, and provides them as hints to the teacher model. On top of that, it uses a technique called "on-policy context distillation," which brings the policy model's output distribution closer to the teacher model (by minimizing reverse KL divergence), directly infusing the coach's advice into the policy model's weights. It is interesting to compare in terms of information volume: a 1–10 point scalar reward can theoretically carry at most 3.3 bits per sample, and even a reward model trained with bf16 (16-bit floating point) has an upper limit of 16 bits, whereas a 1024-token advice sentence, assuming a vocabulary of 150,000, theoretically carries about 17,600 bits per sample—over 1,000 times the bandwidth of bf16 rewards, and over 5,000 times that of discrete 1–10 point rewards. When tested with Qwen3-8B and OLMo-3-7B as policy models, in both cases where the feedback role was self-evaluation and GPT-4o, EL consistently outperformed standard rubric RL (GRPO) not only on the held-out test of WildChat-IF, which consists of 7,500 real user instructions, but also on benchmarks not used in training: AlpacaEval v2.0, WildBench, ArenaHard v2.0, and CreativeWritingV3. In the OLMo-3-7B self-combination, the win rate on AlpacaEval improved from 45.9% with RL to 50.8% with EL, and the WildBench score rose from 42.1 to 47.5. Even more intriguing is the fact that while RL shows greater gains on the training data itself, EL performs better on benchmarks not used in training. The paper explains that scalar rewards, due to their low information volume, make it easier for the policy to drift toward "reward hacking," where it games only the scores, while EL, by chasing the distribution itself, is less prone to overfitting to the training data.
Original Article
View Cached Full Text

Cached at: 08/04/26, 02:10 PM

Researchers from Microsoft Research have proposed a method called “Experiential Learning (EL),” which replaces RL (reinforcement learning) rewards in LLM post-training (the additional fine-tuning after pre-training) from “scores” to “coach’s advice” (https://arxiv[.]org/html/2607.18110v1).

Unlike “verifiable tasks” such as math or code, which have a single correct answer, in RL for “non-verifiable tasks” that measure the quality of text, the grading LLM (LLM-as-a-Judge) compresses responses into a scalar (single numerical) reward of 1–10 points according to a rubric (evaluation criteria) before using it for training. The rich information contained in the grading rationale is discarded here, which was a weakness: the policy model could no longer distinguish between responses that received the same score.

EL repurposes the same grading LLM as a “coach.” Instead of outputting a score, it extracts advice sentences (experiential knowledge)—such as “use specific and diverse adjectives”—that can be transferred to similar tasks, and provides them to the teacher model as hints. On top of that, it uses a technique called “on-policy context distillation,” which brings the policy model’s output distribution closer to the teacher model (by minimizing reverse KL divergence), directly infusing the coach’s advice into the policy model’s weights.

The comparison in terms of information volume is interesting: a 1–10 point scalar reward can theoretically carry at most 3.3 bits per sample, and even a reward model trained in bf16 (16-bit floating point) has an upper limit of 16 bits, whereas a 1024-token advice sentence, with a vocabulary of 150,000, theoretically carries about 17,600 bits per sample—over 1,000 times the bandwidth of bf16 rewards, and over 5,000 times that of discrete 1–10 point rewards.

When tested with Qwen3-8B and OLMo-3-7B as the policy models, in both configurations where the feedback role was played by the model itself and by GPT-4o, EL consistently outperformed standard rubric RL (GRPO) not only on the held-out WildChat-IF test set of 7,500 real user instructions, but also on benchmarks never used in training: AlpacaEval v2.0, WildBench, ArenaHard v2.0, and CreativeWritingV3. In the OLMo-3-7B self-evaluation setup, the win rate on AlpacaEval rose from 45.9% with RL to 50.8% with EL, and the WildBench score climbed from 42.1 to 47.5.

Even more intriguing is that while RL shows larger gains on the training data itself, EL performs better on benchmarks not used in training. The paper explains that scalar rewards, carrying limited information, make policies more prone to drifting toward “reward hacking”—inflating the score alone—whereas EL, by pursuing the distribution itself, is less susceptible to overfitting to the training data.

Similar Articles

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Hugging Face Daily Papers

This paper introduces Experiential Learning (EL), a method that repurposes an LLM-as-a-Judge into an LLM-as-a-Coach to provide rich textual feedback instead of scalar rewards, improving performance and generalization on open-ended non-verifiable tasks.

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Hugging Face Daily Papers

LLM-as-a-Tutor introduces a framework that extends LLM's role from judge to tutor by dynamically adjusting prompt difficulty through pairwise comparison and constraint addition, improving instruction-following performance in reinforcement learning.

ExpRL: Exploratory RL for LLM Mid-Training

Hugging Face Daily Papers

ExpRL is a new RL-based mid-training method that uses human-written reference solutions as dense reward scaffolds (never shown to the policy) to improve LLM reasoning, achieving significant gains on hard math benchmarks like AIME-2026.

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

arXiv cs.LG

Harvard researchers challenge the standard LLM training pipeline by showing RL can be effectively applied during pre-training rather than only after SFT, finding that data composition matters more than model scale, and proposing parallel averaging of RL and SFT objectives that outperforms sequential approaches while preserving general capabilities.