DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
Summary
DRACO is a reinforcement learning method that dynamically generates rubrics and redistributes trajectory scores to improve long-horizon agent performance without verifiers.
View Cached Full Text
Cached at: 09/04/26, 03:58 PM
Paper page - DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
Source: https://huggingface.co/papers/2609.04094
Abstract
DRACO dynamically generates rubrics and redistributes trajectory-level scores into per-step advantages for reinforcement learning without verifiers, improving long-horizon agent performance.
Reinforcement Learning from Verifiable Rewardsworks well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in theoutcome-blindsetting, where ground-truth success signals are not available.Multi-criteria rubricsare a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We proposeDRACO: DistributingRubric-based AdvantageforCredit Optimization. It generates rubrics dynamically during training to track the policy’s evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiatedper-step advantagesinGRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld,DRACOgains 15.9 points over the base model and 5.3 points overGRPOtrained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code forDRACOis available at https://github.com/IBM/draco.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.04094
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.04094 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.04094 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.04094 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
This paper identifies weaknesses in existing reinforcement learning methods for diffusion language models—lack of temporal credit assignment and biased likelihood estimates—and proposes DACA-GRPO, a plug-and-play enhancement that introduces denoising progress scores and stratified masking likelihood, achieving consistent improvements across reasoning, code generation, and constrained generation benchmarks.
Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
RubricForge induces evaluation rubrics from labeled trajectories to reduce over-crediting in language model agent evaluation, enhancing faithfulness without environment access.
Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
The paper proposes GACA, a granularity-adaptive credit assignment method for long-horizon LLM agent reinforcement learning that improves task success by adapting resolution to step importance.
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
This paper introduces POW3R, a policy-aware rubric reward framework for reinforcement learning with verifiable rewards (RLVR). It shows that static rubric aggregation misallocates learning signal, and POW3R achieves faster convergence and better performance across multiple settings.
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
A Scale AI research paper proposes Rubric Dropout, a dropout-style regularization for rubric criteria in rubric-as-reward RL, showing it mitigates reward hacking and improves out-of-distribution gold judge scores on medical and science benchmarks.