Tag
This paper introduces Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping in reinforcement learning with verifiable rewards. CPO outperforms entropy-based RLVR methods on both in-domain and out-of-domain benchmarks.
This paper reveals that PPO-Clipping's use of Euclidean metric causes exploration collapse in LLM RL, and proposes Riemannian Isometric Policy Optimization (RIPO) to ensure geometrically consistent policy updates, achieving up to 60% improvement over GRPO on AIME24.
Proposes IGRPO, a framework that adaptively allocates rollout budget based on informativeness of intermediate states for multi-turn LLM agents, unifying adaptive tree-structured exploration with policy learning. Experiments on seven search-augmented QA benchmarks show consistent improvements over baselines.
Introduces ACPO, a token-level credit assignment framework for reinforcement learning in LLMs that uses fine-grained surrogate entropy to improve reasoning performance on math and coding benchmarks, outperforming strong baselines like DAPO, GTPO, and SAPO.
TREK is a staged procedure that uses distillation to expand exploration support for policy optimization, improving performance on mathematical reasoning and agentic tasks beyond standard GRPO.
This paper proves that GRPO, Dr. GRPO, and DAPO are three formulations of the same underlying mechanism: adjusting the standard deviation of rewards within a group of sampled answers. The group-standard-deviation identity shows that unanimous groups teach nothing while split groups drive learning, revealing a unified dial for training language models to reason.
ReGRPO introduces a reflection-augmented policy optimization framework for tool-using vision-language agents, leveraging structured failure observations and joint optimization of reflection tokens and actions to improve recovery from tool failures, achieving state-of-the-art results on GTA and GAIA benchmarks.
This paper extends Frictive Policy Optimization (FPO) to handle perceptual asymmetry in dialogue, where participants hold asymmetric partial information. It demonstrates that evaluating friction from each participant's perspective is more effective than omniscient access, and proposes annotation refinements for grounding states.
VideoSearch-R1 introduces an agentic framework that iteratively retrieves videos and refines search queries using continuous latent space refinement and policy optimization, achieving state-of-the-art performance on video corpus moment retrieval and temporal grounding tasks.
JERP introduces a method for LLM agents to jointly learn interpretable natural-language rules and update policy parameters from the same interaction trajectories, improving performance on AlfWorld and WebShop while maintaining inspectability.
BiPACE introduces a drop-in advantage estimator that fixes state-action credit mismatch in stepwise group-based RL for LLM agents, using bisimulation-guided state clustering and action counterfactual estimation, achieving significant performance gains on ALFWorld, WebShop, and TextCraft with Qwen2.5 models.
Introduces Strategy-Guided Policy Optimization (SGPO) for LLM reasoning, which replaces trajectory imitation with strategy distillation, improving generalization on math benchmarks.
Self-Reset Policy Optimization (SRPO) addresses credit assignment in multi-step reasoning RL post-training by localizing the first wrong reasoning step and learning from counterfactual continuations without external supervision.
DR-MV3D presents a map-grounded learning framework with dense rewards to improve multi-view 3D visual question answering through global map construction, view-trajectory planning, and egocentric grounding.
PolicyTrim is a reinforcement learning-based post-training framework that improves action chunk utilization by 3× and reduces physical execution steps by 51.4% in Vision-Language-Action models, delivering up to 5.83× deployment speedup.
Introduces Independent Combinatorial Tokens (ICT) framework that uses Jensen-Shannon divergence between token logit distributions to identify critical branching points, preventing entropy collapse and explosion in RLVR for LLM reasoning. Achieves up to 14.9% pass@4 improvement on Qwen models.
ZPPO introduces a replay buffer for hard questions in reinforcement learning for LLMs/VLMs, allowing repeated exposure to gradually improve rollout accuracy without policy drift. The method graduates more hard questions than GRPO, especially those with near-zero initial accuracy.
GraphPO is a novel graph-based reinforcement learning framework that represents rollouts as a directed acyclic graph, merging semantically equivalent reasoning paths to reduce redundant exploration and improve credit assignment for large reasoning models.
MobileForge presents an annotation-free adaptation system for mobile GUI agents that uses real app interaction and hierarchical feedback-guided policy optimization to improve performance, achieving near state-of-the-art results on AndroidWorld with open data.
NVIDIA GEAR lab introduces ENPIRE, a framework for autonomous real-world robot policy self-improvement that achieves 99% success on dexterous manipulation tasks like GPU insertion and zip-tying, with multi-robot parallel learning and open-source release.