@cwolferesearch: Agentic RL requires new algorithm modifications. In GRPO, the “group” used starts to change when training agents… In va…

X AI KOLs Timeline Papers
reinforcement-learning grpo agentic-rl reward-normalization multi-task-learning algorithm

Summary

This thread discusses modifications to GRPO for agentic RL, focusing on different levels of advantage normalization (prompt-level, task-level, environment-level) to handle higher reward variance in multi-task, multi-turn environments.

Agentic RL requires new algorithm modifications. In GRPO, the “group” used starts to change when training agents… In vanilla GRPO, we sample multiple rollouts per prompt and compute the advantage by normalizing each rollout’s reward (i.e., subtracting the mean reward and dividing by the standard deviation of rewards in the group). This works well for single-turn tasks, but agentic RL can have much higher reward variance due to rollouts having multiple turns, different environments, etc. To cope with higher variance in agentic RL, several papers modify GRPO by changing the group over which advantages are normalized: (1) Prompt-level normalization. This is standard GRPO. For each prompt, sample multiple trajectories, compute the mean / standard deviation of rewards within that prompt group, and use these statistics to estimate an advantage for each trajectory. (2) Task-level normalization. AgentRL first computes GRPO-style trajectory advantages, assigns these advantages to agent-generated tokens, and then normalizes token-level advantages across an entire task / domain. This reduces optimization scale differences across domains and prevents one task from dominating the update. (3) Environment-level normalization. ERPO keeps the prompt-level reward mean from GRPO, but computes the reward standard deviation across all valid trajectories from questions in the same environment. In other words, the denominator in the GRPO-style advantage estimate pools rewards across a wider environment-level group. Why does this help? Agentic RL is multi-task and multi-turn. Different environments can have different reward scales, difficulty levels, trajectory lengths, and variance. If we normalize only within each prompt group, the policy update can become noisy or dominated by a subset of tasks. By widening the normalization group from a prompt to an entire task / domain / environment, we can make advantage estimates more stable during agent training.
Original Article
View Cached Full Text

Cached at: 07/07/26, 09:38 PM

Agentic RL requires new algorithm modifications. In GRPO, the “group” used starts to change when training agents…

In vanilla GRPO, we sample multiple rollouts per prompt and compute the advantage by normalizing each rollout’s reward (i.e., subtracting the mean reward and dividing by the standard deviation of rewards in the group). This works well for single-turn tasks, but agentic RL can have much higher reward variance due to rollouts having multiple turns, different environments, etc.

To cope with higher variance in agentic RL, several papers modify GRPO by changing the group over which advantages are normalized:

(1) Prompt-level normalization. This is standard GRPO. For each prompt, sample multiple trajectories, compute the mean / standard deviation of rewards within that prompt group, and use these statistics to estimate an advantage for each trajectory.

(2) Task-level normalization. AgentRL first computes GRPO-style trajectory advantages, assigns these advantages to agent-generated tokens, and then normalizes token-level advantages across an entire task / domain. This reduces optimization scale differences across domains and prevents one task from dominating the update.

(3) Environment-level normalization. ERPO keeps the prompt-level reward mean from GRPO, but computes the reward standard deviation across all valid trajectories from questions in the same environment. In other words, the denominator in the GRPO-style advantage estimate pools rewards across a wider environment-level group.

Why does this help? Agentic RL is multi-task and multi-turn. Different environments can have different reward scales, difficulty levels, trajectory lengths, and variance. If we normalize only within each prompt group, the policy update can become noisy or dominated by a subset of tasks. By widening the normalization group from a prompt to an entire task / domain / environment, we can make advantage estimates more stable during agent training.

Relevant links for further reading:

  • My blog: https://cameronrwolfe.substack.com/p/agentic-rl
  • ERPO: https://arxiv.org/abs/2512.22857
  • Task-Level Adv. Norm.: https://arxiv.org/abs/2510.04206

Similar Articles