RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
Summary
This paper introduces RubricEM, a reinforcement learning framework that uses rubric-guided policy decomposition and reflection-based meta-policy evolution to train deep research agents for long-form tasks. The resulting RubricEM-8B model demonstrates strong performance on long-form research benchmarks by leveraging stage-aware planning and denser semantic feedback.
View Cached Full Text
Cached at: 05/13/26, 04:13 AM
Paper page - RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
Source: https://huggingface.co/papers/2605.10899 Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
Deep research agents trained using RubricEM framework demonstrate superior performance on long-form research tasks through rubric-guided reinforcement learning with stage-aware planning and reflection-based meta-policy evolution.
Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushesreinforcement learningbeyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structurespolicy execution,judge feedback, andagent memory. Based on this view, we introduce RubricEM, arubric-guided reinforcement learningframework that combinesstagewise policy decompositionwithreflection-based meta-policy evolution. RubricEM first makesresearch trajectoriesstage-aware by conditioning planning,evidence gathering, review, andsynthesison self-generated rubrics. It then assigns credit withStage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback forlong-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.10899
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.10899 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.10899 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.10899 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
This paper introduces POW3R, a policy-aware rubric reward framework for reinforcement learning with verifiable rewards (RLVR). It shows that static rubric aggregation misallocates learning signal, and POW3R achieves faster convergence and better performance across multiple settings.
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
SERPO introduces a self-evolving rubric policy optimization framework for test-time reinforcement learning in open-ended generation, replacing answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters, achieving significant improvements on health and research benchmarks.
Enhancing Rubric-based RL via Self-Distillation
Introduces CriPO, a method that enhances rubric-based reinforcement learning for LLMs via on-policy self-distillation, addressing unexplored and suppressed criteria. Experiments on medicine and science benchmarks show stronger final performance with roughly 2x fewer optimization steps.
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
This paper introduces the Auto-Rubric as Reward (ARR) framework, which externalizes implicit preference knowledge into explicit rubrics for multimodal alignment. It proposes Rubric Policy Optimization (RPO) to stabilize policy gradients, achieving better performance in text-to-image and image editing tasks.
Rubrics as Privileged Information for Open-Ended Generation
This paper introduces Rubrics as Privileged Information (RuPI), extending on-policy self-distillation to open-ended generation by conditioning the teacher on rubrics as soft privileged information. The method outperforms rubric-as-reward RL and reference-completion distillation across multiple LLMs and benchmarks like HealthBench.