grpo

Video-DeepResearch (Video-DR) extends multimodal agents from static images to continuous video streams, introducing a decoupled perception-exploration pipeline and a new benchmark Video-DR-Bench. Their Video-DeepResearch-35B-A3B model achieves 64.0% accuracy, surpassing Claude-4.5-Sonnet, GPT-5, and Gemini 2.5 Pro.

TAPR is a lightweight model trained with reinforcement learning to rewrite user prompts into task-optimized prompts, improving downstream LLM performance on benchmarks like Natural Questions and GSM8K.

This paper proposes LSPO (LoRA Scaffolded Policy Optimization) to solve the vanishing-gradient problem of GRPO on zero-reward "cliff problems": a temporary LoRA adapter is used for a short SFT, successful trajectories are sampled, corrected via importance sampling, and then fed back into the RL batch; ultimately, only the base model is updated. Experiments on DeepMath-103K show it outperforms the DAPO baseline by an average of 3.8 points.

A Meta paper analyzes why standard RL recipes fail for code optimization and rebuilds the entire feedback pipeline with calibrated timing, problem-relative ranking, and GRPO changes, improving Qwen 2.5 7B speed threshold from 18.0% to 31.3%.

Presents B1ade, a minimalist RAG architecture with a 335M zero-training embedding model and a 1B SLM trained via GRPO on 723M tokens, showing emergent attribution behavior and competitive QA performance without large-scale pretraining.

SAF-OPD introduces a Stable Advantage Fusion framework that combines RLVR with on-policy distillation, addressing magnitude and temporal mismatches to improve training stability and performance on math and code benchmarks.

This paper proposes Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages for long-CoT mathematical reasoning. It consistently outperforms GRPO baselines, while also revealing that privileged token-shift directions are unreliable and mostly reflect counterfactual sensitivity rather than learning value.

MemHarness is a framework that enables LLM agents to reconstruct past experiences conditioned on the current context instead of replaying them verbatim, improving performance on ALFWorld and WebShop while reducing negative transfer.

CoRT proposes a token-level credit weighting method for GRPO that uses counterfactual replay to compute token-wise log-likelihood contrasts, redistributing the signed advantage across tokens without an auxiliary scorer, achieving average gains of 4.4 percentage points over response-level GRPO.

This paper addresses challenges in using reinforcement learning for code optimization by proposing three stages: improved testing with DMC-Optim, converting execution time into reward via correctness-speed composition and an offline simulator, and adapting GRPO for noisy timing rewards. The method achieves substantial improvements on code optimization benchmarks.

Reminder for Class 3 of the Training Agents live series, covering reinforcement learning (GRPO) for training agents, how to implement it in TRL, and end-to-end examples, streamed on Hugging Face's X, YouTube, and LinkedIn on Tuesday, July 28.

This paper identifies a stylistic inconsistency in MLLMs where their comprehension is robust but safety can be bypassed by stylistic triggers. It proposes Adversarial Style Optimization (ASO) using GRPO to fine-tune an image-editing model to enhance jailbreak attacks.

Combining GRPO and OPSD, CriPO is a rubric-based reinforcement learning method from ByteDance and Zhejiang University that addresses unexplored and suppressed criteria via self-distillation, achieving better performance and compute efficiency.

This paper identifies a reward-variance collapse failure mode in GRPO for multi-turn evidence-reading agents and proposes CIGPO, which uses per-turn contextual information-gain rewards to maintain gradient signal, achieving +105% F1 improvement on HotpotQA.

This paper proposes Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy to perform entropy-conditioned asymmetric advantage shaping, addressing heterogeneous entropy regimes across tasks during RL-based alignment of LLMs. Experiments show consistent improvements over GRPO and recent entropy-controlled methods across multiple benchmarks.

This paper investigates whether GRPO post-training improves a small (4B-8B) language and vision-language model web agent. It finds a controlled null result: no configuration yields credible gains on mastered tasks, and moderate-to-high learning rates cause degradation or collapse, revealing a double dissociation between degrade and collapse regimes.

MAG introduces a benchmark and harness for multimodal web agents that both execute tasks and generate step-by-step guide text, using screenshots and grounding schemes. The work includes a GRPO training method that nearly doubles the success rate of a 9B agent.

Proposes AgentKGV, an agentic LLM-RAG framework with two-stage training (distillation SFT and trajectory-level GRPO) for verifying facts in knowledge graphs, achieving significant improvements on the T-REx benchmark while reducing retrieval calls.

A new post-training method called IAPO (Information-Aware Policy Optimization) assigns token-wise advantages based on conditional mutual information with the final answer, achieving up to 47% shorter reasoning while improving accuracy on math benchmarks.

Introduces SVR-R1, a multi-turn reinforcement learning framework that uses the model's own verification as a learning signal for multi-modal reasoning, achieving significant accuracy improvements over standard GRPO baselines on vision-language reasoning benchmarks.