Tag
A new research paper reframes LLM agent training as a cross-episode Meta-RL problem, using critic-free policy gradients to enable in-context adaptation without gradient updates. The LAMER framework improves test-time performance by 11-19% over standard RL baselines on long-horizon tasks and generalizes better to unseen environments.
This paper formalises counterfactual policy optimisation for Markov Decision Processes under probabilistic nondeterministic causal models, which separate latent confounding from inherent stochasticity, and proposes a practical optimisation procedure for deriving robust counterfactual policies. The approach is validated on a sepsis treatment simulator with diabetes as an unobserved global confounder.
The paper proposes OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization by masking gradients on answer spans, and introduces Contrast-Augmented Reward to refine reward estimation without extra rollouts. It consistently outperforms existing label-free methods and matches supervised ground-truth reward training across reasoning benchmarks.
Introduces Q-Steer, a rollout-time action-value guidance for molecular policy optimization that uses a frozen prefix-action value scorer to improve sampling without changing the online oracle budget.
This paper introduces β-OPSD, a generalization of on-policy self-distillation that frames it as a policy optimization family with a controllable KL penalty. The method uses distillation to approximate expensive policy optimization, improving stability and reasoning performance on mathematical benchmarks.
Proposes Progress-conditioned Group Policy Optimization (ProGPO) to overcome credit assignment issues in long-horizon agentic tasks by using first-visit observation coverage when all group samples fail, improving performance on ALFWorld and WebShop.
This paper identifies a reward-variance collapse failure mode in GRPO for multi-turn evidence-reading agents and proposes CIGPO, which uses per-turn contextual information-gain rewards to maintain gradient signal, achieving +105% F1 improvement on HotpotQA.
This paper proposes Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy to perform entropy-conditioned asymmetric advantage shaping, addressing heterogeneous entropy regimes across tasks during RL-based alignment of LLMs. Experiments show consistent improvements over GRPO and recent entropy-controlled methods across multiple benchmarks.
This paper introduces Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping in reinforcement learning with verifiable rewards. CPO outperforms entropy-based RLVR methods on both in-domain and out-of-domain benchmarks.
This paper reveals that PPO-Clipping's use of Euclidean metric causes exploration collapse in LLM RL, and proposes Riemannian Isometric Policy Optimization (RIPO) to ensure geometrically consistent policy updates, achieving up to 60% improvement over GRPO on AIME24.
Proposes IGRPO, a framework that adaptively allocates rollout budget based on informativeness of intermediate states for multi-turn LLM agents, unifying adaptive tree-structured exploration with policy learning. Experiments on seven search-augmented QA benchmarks show consistent improvements over baselines.
Introduces ACPO, a token-level credit assignment framework for reinforcement learning in LLMs that uses fine-grained surrogate entropy to improve reasoning performance on math and coding benchmarks, outperforming strong baselines like DAPO, GTPO, and SAPO.
TREK is a staged procedure that uses distillation to expand exploration support for policy optimization, improving performance on mathematical reasoning and agentic tasks beyond standard GRPO.
This paper proves that GRPO, Dr. GRPO, and DAPO are three formulations of the same underlying mechanism: adjusting the standard deviation of rewards within a group of sampled answers. The group-standard-deviation identity shows that unanimous groups teach nothing while split groups drive learning, revealing a unified dial for training language models to reason.
ReGRPO introduces a reflection-augmented policy optimization framework for tool-using vision-language agents, leveraging structured failure observations and joint optimization of reflection tokens and actions to improve recovery from tool failures, achieving state-of-the-art results on GTA and GAIA benchmarks.
This paper extends Frictive Policy Optimization (FPO) to handle perceptual asymmetry in dialogue, where participants hold asymmetric partial information. It demonstrates that evaluating friction from each participant's perspective is more effective than omniscient access, and proposes annotation refinements for grounding states.
VideoSearch-R1 introduces an agentic framework that iteratively retrieves videos and refines search queries using continuous latent space refinement and policy optimization, achieving state-of-the-art performance on video corpus moment retrieval and temporal grounding tasks.
JERP introduces a method for LLM agents to jointly learn interpretable natural-language rules and update policy parameters from the same interaction trajectories, improving performance on AlfWorld and WebShop while maintaining inspectability.
BiPACE introduces a drop-in advantage estimator that fixes state-action credit mismatch in stepwise group-based RL for LLM agents, using bisimulation-guided state clustering and action counterfactual estimation, achieving significant performance gains on ALFWorld, WebShop, and TextCraft with Qwen2.5 models.
Introduces Strategy-Guided Policy Optimization (SGPO) for LLM reasoning, which replaces trajectory imitation with strategy distillation, improving generalization on math benchmarks.