PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Hugging Face Daily Papers Papers
reinforcement-learning research-planning rubric-evaluation scientific-papers ai-training benchmark-improvement

Summary

PaperGym is a framework that converts scientific papers into training environments by separating research questions from evaluation rubrics, enabling reinforcement learning to improve research planning. It demonstrates improved performance over existing methods, with trained models outperforming larger ones on benchmarks.

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.
Original Article
View Cached Full Text

Cached at: 09/01/26, 11:55 AM

Paper page - PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Source: https://huggingface.co/papers/2608.31119

Abstract

PaperGym converts scientific papers into training environments by separating research questions from evaluation rubrics, enabling reinforcement learning that improves research planning across multiple model sizes.

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, soreinforcement learninglacks the environment it requires: tasks paired with a critic.Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. Therubricis further compressed into a single scalar per rollout. We introducePaperGym, a unified framework that turns each research paper into a complete training environment.PaperGymexploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, andcriterion leakagefalls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses therubrictwice: first as privileged context forOPSD’sself-teacher, then as the reward forGRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained onPaperGym-20kwin 58.1% of three-way comparisons, against 28.2% forRubricHub Science. The trained Qwen3-8B reaches 73.48 onResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpusPaperGym-20k, and the benchmarksPaperGym-Innov andPaperGym-Design.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2608\.31119

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### CabbageWyh/PaperGym-Model Updatedabout 9 hours ago • 2

Datasets citing this paper1

#### CabbageWyh/PaperGym-Data Viewer• Updatedabout 9 hours ago • 22.4k • 38

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.31119 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Hugging Face Daily Papers

This paper introduces RubricEM, a reinforcement learning framework that uses rubric-guided policy decomposition and reflection-based meta-policy evolution to train deep research agents for long-form tasks. The resulting RubricEM-8B model demonstrates strong performance on long-form research benchmarks by leveraging stage-aware planning and denser semantic feedback.

OpenAI Gym Beta

OpenAI Blog

OpenAI releases OpenAI Gym, a public beta toolkit for developing and comparing reinforcement learning algorithms with a growing suite of environments and a platform for reproducible research. The toolkit aims to standardize RL benchmarks and address the lack of diverse, easy-to-use environments for the research community.