Bellman Policy Optimization
Summary
Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method that reformulates Policy Mirror Descent using the Bellman equation for autoregressive generation with terminal rewards, improving mathematical reasoning in large language models.
View Cached Full Text
Cached at: 09/23/26, 03:31 AM
Paper page - Bellman Policy Optimization
Source: https://huggingface.co/papers/2609.15987
Abstract
Reinforcementlearningwithverifiablerewards(RLVR)improvesthereasoningcapabilitiesoflargelanguagemodels(LLMs).WeintroduceBellmanPolicyOptimization(BPO),acritic-freemethodderivedfromPolicyMirrorDescent(PMD).Forautoregressivegenerationwithterminalrewards,BPOusestheBellmanequationstoreformulatePMDasatrajectory-levelobjective.Thereformulationavoidsestimatingstatevaluesatintermediatestates.WeprovethatithasthesameuniqueoptimalsolutionastheoriginalPMDobjective.WederivethepracticalBPOlossbyapproximatingthisobjective.Itsmismatch-correctionweightisasmoothedratioofcomplementarytokenprobabilities.ExperimentsonmathematicalreasoningbenchmarksdemonstratetheeffectivenessofBPO.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2609\.15987
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.15987 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.15987 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.15987 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Contrastive Branch Policy Optimization
CBPO introduces a contrastive branch policy optimization method for fine-grained credit assignment in reinforcement learning with verifiable rewards, enhancing language model performance in tool-integrated reasoning tasks across multiple benchmarks.
Proximal Policy Optimization
OpenAI introduces Proximal Policy Optimization (PPO), a reinforcement learning algorithm that matches or outperforms state-of-the-art methods while being simpler to implement and tune. PPO uses a novel clipped objective function to constrain policy updates and has since become OpenAI's default RL algorithm.
Best Practice Critic Optimization
BPCO introduces a stable recipe for critic-based reinforcement learning in language models, combining bounded value predictions and Monte Carlo targets to match group-based methods with single-response sampling.
Deterministic Pareto-Optimal Policy Synthesis for Multi-Objective Reinforcement Learning
This paper introduces a novel preference-conditioned Bellman operator based on Chebyshev scalarization to compute deterministic Pareto-optimal policies for Multi-Objective Markov Decision Processes, proving its convergence and effectiveness in capturing the entire Pareto frontier.
LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models
Introduces LambdaPO, a novel reinforcement learning framework that improves upon GRPO by decomposing advantage estimation into pairwise preference comparisons and adding a semantic density reward, achieving better performance on math reasoning tasks.