Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Summary
AutoSciRub is an evaluation-first framework that improves autonomous scientific agents by generating task-specific executable rubrics to guide experiments and verification, achieving consistent performance gains on benchmarks.
View Cached Full Text
Cached at: 09/01/26, 11:53 AM
Paper page - Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Source: https://huggingface.co/papers/2608.31076
Abstract
AutoSciRub improves autonomous scientific agents by generating task-specific executable rubrics that guide experiments, verify criteria, and iteratively refine outputs.
Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we presentAutoSciRub, an evaluation-first framework that induces a task-specificexecutable rubricbefore research execution, and uses it to guide execution, criterion-level verification as well as iterative revision.AutoSciRubdecomposes an underspecified instruction intoatomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision,rubric-guided verificationidentifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. OnResearchClawBench,AutoSciRubconsistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset ofAstaBenchE2E Discovery,AutoSciRubfurther achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).
View arXiv pageView PDFGitHub14Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.31076 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.31076 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.31076 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
This paper introduces AutoResearchEval, an evaluation framework for AI agents in automated scientific research, revealing a critical lack of metacognitive abilities as a recurring failure pattern across models.
An Empirical Study of Automating Agent Evaluation
This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.
Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
RubricForge induces evaluation rubrics from labeled trajectories to reduce over-crediting in language model agent evaluation, enhancing faithfulness without environment access.
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
This paper introduces the Auto-Rubric as Reward (ARR) framework, which externalizes implicit preference knowledge into explicit rubrics for multimodal alignment. It proposes Rubric Policy Optimization (RPO) to stabilize policy gradients, achieving better performance in text-to-image and image editing tasks.
RUBAS: Rubric-Based Reinforcement Learning for Agent Safety
RUBAS is a rubric-based reinforcement learning framework for agent safety that decomposes LLM agent behavior into four dimensions—tool-use safety, argument safety, response safety, and helpfulness—providing fine-grained rewards over complete trajectories. Experiments show RUBAS improves safety over standard alignment baselines while reducing tool-grounded hallucinations and maintaining competitive utility.