Tag
Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method that reformulates Policy Mirror Descent using the Bellman equation for autoregressive generation with terminal rewards, improving mathematical reasoning in large language models.
The paper proposes belief-shift branching for tree-structured reinforcement learning to enhance step-level credit assignment, demonstrating improved performance in mathematical and coding benchmarks.
The paper proposes DATPO, a difficulty-adaptive tree-structured policy optimization method that enhances reasoning coverage in large models through sentence-entropy-guided branching and diversity-aware optimization, outperforming baselines in pass@k metrics.
DataFlex-RL is an evaluation platform for RLVR data policies that finds uniform sampling performs comparably to adaptive methods across mathematics, logic, and science benchmarks.
DE-Venus is a unified framework for data-efficient Reinforcement Learning with Verifiable Rewards in large language models, reducing annotation and training costs while maintaining performance.
The paper proposes Group Adaptive Clustering Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to rollout advantage, improving Pass@1 and Pass@k on math reasoning and coding benchmarks.
Cliff improves reinforcement learning with verifiable rewards by using an off-the-shelf LLM to detect the first reasoning mistake and shape token-level advantages, leading to consistent performance gains over existing methods.
This paper investigates how reinforcement learning with verifiable rewards (RLVR) narrows the solution space in LLM reasoning by analyzing where diversity is lost, finding it concentrated at the 'entrance' of trajectories. It demonstrates interventions to recover breadth without compromising accuracy.
The article describes a reinforcement learning method with verifiable rewards to fine-tune AI models for text-to-SQL, achieving human-level accuracy on the BIRD benchmark without scaffolding.
CBPO introduces a contrastive branch policy optimization method for fine-grained credit assignment in reinforcement learning with verifiable rewards, enhancing language model performance in tool-integrated reasoning tasks across multiple benchmarks.
The paper introduces Continual Reasoning Gym to diagnose and improve continual reinforcement learning with verifiable rewards (RLVR), proposing Continual Prompt Replay (CPR) to harness shared reasoning structures and achieve multitask-level performance.
This paper introduces PAIR, a pairwise-aware inclusion reweighting method for adaptive rollout allocation in RLVR, improving sample efficiency and accuracy over pointwise allocators by correcting biases in pairwise gradient estimation.
Simon Willison analyzes the timeline of OpenAI's accidental attack on Hugging Face, suggesting that RLVR training of a new model explains the lack of safety behaviors and lax monitoring.
Explains why frontier AI models often behave rudely or disobediently, citing former Meta engineer Kun Chen on RLHF and RLVR training that optimizes for task success over human-friendly communication.
This paper introduces RSTG, a method that selectively applies on-policy distillation to recover learning signals from zero-variance GRPO groups during LLM post-training, achieving substantial gains on math and code reasoning benchmarks.
The paper introduces verifier-induced support reshaping, showing that on-policy RL with verifiable rewards can improve the current objective while making successful behaviors for later objectives too rare to sample. Experiments across math reasoning and instruction following demonstrate that endpoint improvements do not guarantee future trainability.
Introduces RLSVR, a task-transformation paradigm that extends RLVR to open-ended tasks via self-verifiable rewards from self-play games, instantiated in SpyRL and Vision-Zero. It improves LLM performance on summarization, creative writing, and math reasoning.
This paper proposes LSPO (LoRA Scaffolded Policy Optimization) to solve the vanishing-gradient problem of GRPO on zero-reward "cliff problems": a temporary LoRA adapter is used for a short SFT, successful trajectories are sampled, corrected via importance sampling, and then fed back into the RL batch; ultimately, only the base model is updated. Experiments on DeepMath-103K show it outperforms the DAPO baseline by an average of 3.8 points.
This paper introduces LEEPS, a latent-guided explore-exploit prompt sampler for efficient reinforcement learning with verifiable rewards (RLVR) in LLMs. It adaptively balances reuse of informative prompts and exploration of uncertain ones, improving reasoning benchmark scores by 2.6-3.7% over baselines while adding only ~2 seconds of overhead per training step.
SAF-OPD introduces a Stable Advantage Fusion framework that combines RLVR with on-policy distillation, addressing magnitude and temporal mismatches to improve training stability and performance on math and code benchmarks.