Banger paper from Meta on AI research agents.

(bookmark it)

If you build AI-scientist systems, this one is worth your time.

The result:

A 27B open model beats the strongest open autoresearch baseline by 14.0%, mostly on novelty, and beats Claude Code SDK and Codex SDK setups by up to 5.9%.

How it works:

IdeaScientist splits ideation into three roles trained separately with RL. A gap finder reads related work for limitations, an innovator retrieves mechanisms that solved similar problems in other fields, and a writer turns the result into a full proposal.

Retrieval runs over a corpus of 2.77M decomposed research ideas.

The evaluation only allows literature published before a cutoff date and scores proposals against directions explored later in 15K human-written papers.

Paper: https://arxiv.org/abs/2610.04074

Chat with Paper: https://academy.dair.ai/papers/ideascientist-orchestrating-agents-for-grounded-scientific-ideation-2610.04074…

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Source: https://arxiv.org/html/2610.04074 Renjie TaoAffiliation:Meta Reality LabsYiwei LiaoAffiliation:Meta Reality LabsChuanyang JinAffiliation:FAIR at MetaKai SunAffiliation:Meta Reality LabsXiao YangAffiliation:Meta Reality LabsXinyuan ZhangAffiliation:Meta Reality LabsXilun ChenAffiliation:Meta Reality LabsZhuangqun HuangAffiliation:Meta Reality LabsLechen ZhangAffiliation:University of Illinois Urbana-ChampaignYongjin YangAffiliation:University of TorontoYinghui HeAffiliation:Princeton UniversityWeihao XuanAffiliation:The University of TokyoAffiliation:RIKEN AIPRakesh WangaAffiliation:Meta Reality LabsAnuj KumarAffiliation:Meta Reality LabsMona T. DiabAffiliation:Carnegie Mellon UniversityWen-tau YihAffiliation:FAIR at MetaXin Luna DongAffiliation:Meta Reality Labs

Abstract

Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%.

††date:October 2, 2026††correspondence:Jiarui Liu at[email protected]††Code:https://github.com/jiarui-liu/IdeaScientist††Dataset:https://huggingface.co/datasets/Jerry999/SvalbardIdeaVault## 1Introduction

As coding agents take on a growing share of implementation work,Autonomous Research (AutoResearch)has attracted increasing attention(Lu et al., 2024;Yamada et al., 2025;Gottweis et al., 2026;Schmidgall et al., 2025). Yet most of this progress sits downstream: existing systems couple solution proposal with implementation, experimentation, and writing, making it difficult to determine whether success comes from strong research ideas or strong downstream execution. Frontier labs report the same asymmetry: OpenAI recently reported its “automated research intern” focuses on well-defined tasks under human direction, whereas high-level planning remains a minimal fraction of agent output(OpenAI, 2026).

In this paper we focus on the problem ofresearch ideation, which we define as follows: given a problem definition (the task input and the desired output) together with achallengethat current solutions fail to meet, produce a grounded research proposal that addresses the challenge. The challenge may concern any desideratum of the output or of the process producing it, such as inferior quality, high inference latency, or privacy risks. We ask two questions: (1)how can agents generate high-quality research proposals for open research problems?(2)how can such proposals be evaluated reliablybeforeexperimentation,when many plausible solutions exist and none is uniquely correct?

Where, then, do research ideas come from? Our key intuition is rooted in a recurring view inCognitive ScienceandPsychonomic Science: novelty emerges by connecting existing knowledge in new ways, and a solution developed for one problem transfers to a seemingly unrelated one when the two share the same underlying structure(Mednick, 1962;Uzzi et al., 2013;Gentner, 1983;Gick and Holyoak, 1983;Loewenstein et al., 1999). We thus draw insightsnotfrom prior work in the same domain, but from workaddressing a similar challenge in a different field. For example, web search cuts latency by low-cost retrieval of a candidate set before applying a sophisticated ranker; the same two-stage structure answers the latency challenge in recommendation, where first-stage filtering precedes fine-grained scoring.

We introduce IdeaScientist, an agent framework that operationalizes this process with three specialized sub-agents. Agap finderexamines closely related prior solutions in the target domain to identify the shared methodological limitation that leaves the challenge unaddressed, or even amplifies it. Aninnovatorsearches for mechanisms that have addressed a similar challenge in other fields, then adapts them to the target setting. Finally, areport writerturns the resulting intuitions into a complete research proposal(Baek et al., 2025;Keya et al., 2026). We train each sub-agent with asynchronous GRPO and role-specific, rubric-based rewards.

Figure 1:Overview of IdeaScientist. An orchestrator coordinates specialized agents—gap finder, innovator, and report writer—through shared intermediate artifacts, and the agents access the Svalbard Idea Vault for across-domain insights.Both training and inference rely on aSvalbard Idea Vault111This is named after theSvalbard Global Seed Vault, as a metaphor for the repository of research ideas., an offline corpus built throughidea decomposition, which extracts 2.77 million (problem definition, challenge, research gap, intuition, proposal) tuples from research papers. We set the cutoff date to January 1, 2026, and treat the 14,977 methodology papers published after the cutoff aspost-cutoff targets. The idea vault serves three purposes. Pre-cutoff papers form the literature that the gap finder and the innovator search. A portion of the post-cutoff targets generates training data: the agent receives only the problem definition and challenge and produces a proposal, and comparing it against the target paper’s actual solution yields the supervision signal for reinforcement learning. The remaining targets are held out for evaluation—because the agent cannot search literature published after the cutoff during evaluation, this design tests whether it can infer a research direction before that direction appeared.

In summary, this paper makes the following contributions:

1. 1.We separate out research ideation from AutoResearch, and formalize it as producing a grounded proposal from a problem definition and a challenge.
2. 2.We introduce IdeaScientist, an agent framework built on the intuition of searching for mechanisms that address similar challenges in different fields, and apply role-specific reinforcement learning to improve gap finding, innovation, and report writing.
3. 3.We construct the Svalbard Idea Vault, an offline corpus of 2.77 million (problem definition, challenge, research gap, intuition, proposal) tuples produced through idea decomposition. The Idea Vault is used for literature search, training data generation, and evaluation.
4. 4.We conduct extensive experiments showing the benefits of IdeaScientist. On Qwen3.6-27B, it outperforms the strongest state-of-the-art autoresearch baseline by 14.0% in overall score, which averages novelty and proposal quality. On the same open backbone, it also outperforms Claude Code SDK with Claude-4.8-Opus by 5.9% and Codex SDK with GPT-5.4 by 5.1%.

We believe IdeaScientist takes an important step toward fully automated research, and that it can be combined with automated research-topic identification and automated experimentation for stronger end-to-end auto-research.

2Related Work

2.1Autonomous Research Systems and Research Ideation

End-to-end autoresearch systems combine ideation with implementation through agent workflows, experiment search, debate, or persistent memory(Lu et al., 2024;Baek et al., 2025;Ghafarollahi and Buehler, 2025;Ifargan et al., 2025;Schmidgall et al., 2025;Liu et al., 2026a;Song et al., 2026;Tang et al., 2026;Lu et al., 2026;Yamada et al., 2025;Lyu et al., 2026;Jin et al., 2026;Wang et al., 2026a;Gottweis et al., 2026;Su et al., 2025;Weng et al., 2026;Mitchener et al., 2025). Their objectives typically reward experiment selection or downstream success, entangling proposal quality with coding and empirical outcomes(Falck et al., 2026;Yang et al., 2026b). Recent work supervises ideation more directly through execution feedback, supervised fine-tuning followed by rewards for technical depth and semantic alignment, debate-derived rewards, or paper-specific reference-free rubrics(Zhang et al., 2026b;Gu et al., 2026;Salimi et al., 2026;Wang et al., 2026c). However, these methods do not train literature search as an inspectable sequence of research operations.

Retrieval-based deepresearch systems structure evidence use through knowledge graphs, hybrid retrieval, citation paths, iterative search, or outline-guided synthesis(Wang et al., 2024a;Wang et al., 2024b;Li et al., 2025a;Hu et al., 2024;Wang et al., 2024c;Yan et al., 2025;Shao et al., 2024a). Other methods rank research gaps or recombine ideas across scholarly trajectories(Shahid et al., 2025;Xu et al., 2025b;Li et al., 2026c;Radensky et al., 2024;Suzuki and Banaei-Kashani, 2026;Choi and Song, 2026). IdeaScientist combines these threads in an inspectable ideation process, separately training agents to identify limitations in closely related work, seek inspiration from how other fields address similar challenges, and develop those insights into complete proposals.

2.2Evaluating Generated Research Ideas

Pre-execution evaluation must substitute observable proxies for eventual scientific value, but different proxies answer different questions. Reference-free rubrics use LLM judges to assess whether an idea appears high quality across dimensions such as novelty and feasibility(Guo et al., 2025;Ruan et al., 2026;Chen et al., 2026;Schopf and Färber, 2026;Wang et al., 2026d). Learned reviewers estimate acceptance-like quality(Zhu et al., 2025;Idahl and Ahmadi, 2025;Weng et al., 2025;Zeng et al., 2025;Nguyen et al., 2026;Deng et al., 2026;Li et al., 2026a;Camelo-Guerrero and Diaz-Rodriguez, 2026;Akella et al., 2025;Wen et al., 2026). Factuality metrics test whether generated claims are directly supported by cited or retrieved scientific evidence(Min et al., 2023;Song et al., 2024;Dhuliawala et al., 2024). However, LLM judges can disagree, exhibit self-preference, or reward stylistic features(Zheng et al., 2023;Panickssery et al., 2024;Li et al., 2026b;Ji et al., 2026;Norman et al., 2026;Sinhahajari et al., 2026;Zhang et al., 2026a). Expert studies also find that greater novelty may come at the cost of feasibility in actual execution(Si et al., 2025;Ho et al., 2026;Si et al., 2026;Jiang, 2026). Direct execution offers a more direct outcome-based signal, but it is costly and mixes ideation with implementation skill(Foster et al., 2026). We validate and justify our evaluation metrics through cross-evaluation and manual annotation.

2.3Training LLM Agents for Non-Verifiable, Long-Horizon Tasks

Ideation is difficult to train because unlike code, a proposal has no directly verifiable reward. Rubric-based reinforcement learning converts qualitative criteria into scalar supervision(Gunjal et al., 2026;Huang et al., 2025;Zhou et al., 2025;Arora et al., 2025;Mahmoud et al., 2026;Liu et al., 2026d;Xu et al., 2025a;Shao et al., 2025;Goel et al., 2025), but a single proposal-level rubric provides weak credit assignment and invites reward hacking(Wang et al., 2026b;Yang et al., 2026a). Outcome feedback can optimize long tool-use trajectories(Jin et al., 2025;Chen et al., 2025;Luo et al., 2025;Wei et al., 2026). In multi-agent settings, however, prior methods either specialize roles under a shared terminal objective or distill their trajectories into one policy(Motwani et al., 2024;Zhao et al., 2026;Park et al., 2025;Li et al., 2026d;Li et al., 2025b). This leaves unclear which reasoning stage improved and whether specialization survives deployment. To make long-horizon research ideation trainable, we use Group Relative Policy Optimization (GRPO)(Shao et al., 2024b)to train each subagent role with a stage-matched human reference artifact and rubric, then recompose the trained roles within the full agentic loop.

3IdeaScientist: A Framework for Autoresearch Ideation

3.1Harness Design

IdeaScientist decomposes research ideation into three specialized sub-agents coordinated by an orchestrator: agap finderexamines closely related works to identify methodological limitations, aninnovatordraws on work from different problem settings to develop transferable solution intuitions, and areport writerconsolidates the surviving intuition into a complete proposal. The report writer follows a fixed output schema that requires it to formalize the problem setting, state the central novelty, distinguish the proposed mechanism from the closest prior methods, and provide sufficient architectural and training detail to make the proposal actionable. It also specifies an evaluation plan, falsifiable predictions and their refutation conditions, and the proposal’s limitations, while supporting claims about prior work with inline database identifiers. Table5provides the complete schema.

The orchestrator maintains the evolving research plan and determines which sub-agent to invoke next, allowing the system to revisit earlier stages when additional evidence or refinement is needed. The specialized agents execute the corresponding stages of research ideation through predefined interfaces.

Because full papers are often too long to include directly in every agent context, we separate planning and synthesis from full paper reading and reviewing. A dedicated second-tierpaper readeraccesses full paper text in a separate context, performs targeted QA, and returns its responses to the requesting agent. A second-tierreviewerevaluates quality of generated research intuitions and proposals.

Figure1shows the resulting flow from the problem and challenge to a grounded gap, a research intuition that may be adapted from another field, and a complete proposal. These intermediate artifacts let us check and train the gap finder, innovator, and report writer separately. The complete action space for each role is listed in AppendixA.1.

3.2Training Design

We train the gap finder, innovator, and report writer separately before recomposing them in the shared harness, while preserving the same inputs, outputs, information access, and actions during training and deployment. Per-role training is possible because each agent has a clearly defined input and output, while human-authored papers provide valuable information from which corresponding reference artifacts can be reconstructed. This decomposition turns a sparse proposal-level outcome into stage-specific learning signals without prescribing a single tool-use trajectory. We optimize a separate LoRA adapter for each role using asynchronous Group Relative Policy Optimization (GRPO)(Shao et al., 2024b). We keep the orchestrator fixed because its deliberately narrow action space to maintain the research plan and schedule specialized roles has little room for improvement. AppendixE.1reports the training configuration details.

Rather than relying only on reference-free LLM judgments, we design a reference-grounded reward that evaluates both the standalone quality of each generated artifact and its agreement with the corresponding human reference. The resulting reward provides an outcome-level learning signal for the complete trajectory of decisions and tool calls that produces the artifact. We use three complementary reward components.

• •Reference-free scorerfreer_{\mathrm{free}}: An LLM judge rates the specificity, methodological significance, feasibility, and evidential support of the generated artifact, measuring whether it is well formed and coherent independently of the reference.
• •Reference-grounded scorerrefr_{\mathrm{ref}}: A field-level comparison measures agreement with the reference on substantive research decisions such as the core limitations, proposed method, evaluation design, and related work.
• •Citation F1rcitr_{\mathrm{cit}}: Deterministic set F1 measures overlap between the papers cited by the generated artifact and those cited by the reference, capturing alignment in literature grounding and retrieval.

The rubric reward combines these three components subject to a validity gate:

rrubric=g⋅(λfree​rfree+λref​rref+λcit​rcit),r_{\mathrm{rubric}}=g\cdot\left(\lambda_{\mathrm{free}}r_{\mathrm{free}}+\lambda_{\mathrm{ref}}r_{\mathrm{ref}}+\lambda_{\mathrm{cit}}r_{\mathrm{cit}}\right),where all three weights are1/31/3in the reported runs. The gateg∈{0,1}g\in\{0,1\}is determined by structural, task-compliance, citation-grounding, and tool-validity checks, together with a judged methodological criterion for the gap finder and innovator.

Finally, the training reward isr=0.85​rrubric+0.15​rprocr=0.85r_{\mathrm{rubric}}+0.15r_{\mathrm{proc}}, whererprocr_{\mathrm{proc}}is a rule-based term that penalizes protocol violations (e.g., disallowed tool calls, invalid formats, missing required fields). AppendixB.1provides detailed rubric definitions and reward design choices.

4Dataset Construction

4.1Idea Vault construction

We constructed the Svalbard Idea Vault as an offline literature database using the official arXiv API. The database covers all arXiv subject areas to allow drawing on relevant work from neighboring fields. The collection contains papers published from 1986 through June 18, 2026, with duplicates removed and full text collected whenever possible. For papers collected on or after January 1, 2026, we additionally follow their reference lists one hop and add cited papers that are missing from the database, retrieving full text whenever available. The resulting database contains 4.7 million papers, including 2.8 million (60%) with full text.

For each paper, we applyidea decompositionto reconstruct the problem definition and challenge as inputs and three chained reference artifacts as targets: the methodological gap, the proposed research intuition, and the complete research proposal. These artifacts correspond to the outputs of the gap finder, innovator, and report writer.

4.2Training data construction

We construct training examples from machine learning papers published on or after January 1, 2026. Starting from an initial corpus of 104K candidate papers, we first retained papers whose LaTeX source is available for download and for which more than70%70\%of cited works have usable full text in our database. We then restricted the corpus to methodological papers, such as contributing a new method, model, algorithm, architecture, training procedure, or technical framework, excluding surveys, position papers, datasets and benchmarks, and purely empirical engineering reports.

These filters yielded14,97714{,}977target papers. We randomly partitioned them into14,50014{,}500training papers,200200validation papers, and277277held-out test papers, and used the same partition for all three agents. Qwen3.6-27B converts each human-authored target paper into the required proposal schema, preserving the research problem, proposed method, model design, related work, and limitations, while removing all measured experimental outcomes. This results-masked representation records directions that human researchers have pursued without exposing their experimental success.

We generate training data using the extracted gaps, intuitions, and research proposal, which provides the skeleton about the research direction; supporting claims and citations must be grounded in papers read through the environment. We validate every artifact carefully before adding to the training set. AppendixDreports details of the training reference construction.

5Experiments

5.1Experiment Setup

We evaluate IdeaScientist-Base and IdeaScientist-Trained on Qwen3.6-27B. In the trained variant, the gap finder, innovator, and report writer use their trained checkpoints. Every run uses a fixed generation token budget per turn and a fixed turn limit for each role.

We compare against three families of methods: (1) general-purpose research agents, including Claude Code with Claude Opus 4.8 and Codex with GPT 5.4; (2) open-source autoresearch systems including end-to-end systems such as AutoResearchClaw(Liu et al., 2026a), Dr. Claw(Song et al., 2026), and DeepScientist(Weng et al., 2026), and research ideation systems including ResearchAgent(Baek et al., 2025), CycleResearcher paired with its DeepReviewer critic(Weng et al., 2025;Zhu et al., 2025), SciAgents(Ghafarollahi and Buehler, 2025), and AI-Researcher(Tang et al., 2026), with different backbones including Qwen3.6-27B, Gemma-4-31B, and GPT-OSS-120B; (3) IdeaScientist-Base (our harness) on the same backbones, and IdeaScientist-Trained.

For a fair comparison, all systems receive the same problem definition and output schema and use the same tools to access our offline paper database for literature retrieval, while the target paper remains hidden and internet access is disabled. External AutoResearch systems and general-purpose agents retain their native agent loops. For end-to-end research systems, we instruct the agent to stop once a machine learning research proposal is complete rather than continuing to implementation or experimentation. AppendixE.1reports the full configuration.

Evaluation Metrics.

We evaluate the generated research proposal for each held-out target paper using two complementary groups of metrics. First,Noveltymeasures how difficult it is to derive the proposal’s central mechanism from the comparison literature in the Idea Vault. We compare with three scopes of literature: against all literature (All), against literature before the cutoff date (Cutoff), and against literature in the same problem domain. Second,Proposal qualitymeasures the research proposal on six criteria commonly used in conference peer review:relevance, clarity, specificity, actionability, soundness,andimpact. AppendixCgives the judgment prompts.

5.2Q1. Can Our Agents Generate High-Quality Research Proposals?

Table 1:Proposal-level evaluation on the held-out 277-paper test split. We measure novelty against all literature (All), pre-cutoff literature (Cutoff), and same-domain literature (In-domain), with their mean (Avg). We measure proposal quality along six dimensions: with relevance (Relev.), clarity (Clar.), specificity (Spec.), actionability (Act.), soundness (Sound.), and impact (Impact), with their mean (Avg). The overall score is the mean of the novelty Avg and the proposal-quality Avg. All metrics are scored by a Qwen3.6-27B judge. Within each group, the best result is bolded and the second-best is underlined. IdeaScientist outperforms state-of-the-art systems by up to 14%, especially on novelty.Table1evaluates the ideation systems using the metrics defined in Section5.1. Even without subagent training, IdeaScientist-Base is competitive on every backbone. Against general-purpose agents running in their native SDK harnesses, IdeaScientist-Base leads Claude Code SDK on Claude-4.8-Opus by 9.0% on the overall score (77.7% vs. 68.7%) and leads Codex SDK on GPT-5.4 by 2.4% (71.9% vs. 69.5%). Against state-of-the-art autoresearch systems, IdeaScientist-Base ranks first on all three open backbones. Its margin over the strongest matched-backbone baseline ranges from 3.3% on GPT-OSS-120B to 12.9% on Qwen3.6-27B.

Our reward design enables training to improve all three producer roles. In isolated held-out evaluation, the final checkpoints raise the reward from 56.6% to 66.7% (+10.1%) for the gap finder, from 58.8% to 69.5% (+10.7%) for the innovator, and from 78.6% to 89.1% (+10.5%) for the report writer (Appendix Figure5and Table11). Recomposing the three trained roles carries these gains over to proposal generation. IdeaScientist-Trained improves over IdeaScientist-Base by 1.9% on average novelty and 0.4% on average proposal quality. It reaches an overall score of 74.6%, which makes it the strongest open system and puts it 14.0% above the best baseline on Qwen3.6-27B.

Across dimensions, IdeaScientist’s advantage comes mainly from novelty. On average proposal quality, IdeaScientist-Base is on par with or modestly above the strongest baseline, ranging from1.8%1.8\%on Gemma-4-31B to+2.5%+2.5\%on Qwen3.6-27B. On average novelty, its gains are much larger, up to+23.1%+23.1\%on Qwen3.6-27B. On that backbone, the gain holds against all literature (+8.4%+8.4\%), against pre-cutoff literature (+8.6%+8.6\%), and most strongly against same-domain literature (+48.0%+48.0\%). Being novel relative to a proposal’s own research area requires thinking outside that area’s established approaches; this is arguably the hardest form of novelty, and it is where our method stands strongest. Figure2(a) shows that this advantage spans all six domain groups, with the estimated IdeaScientist-Trained profile exceeding Base in five.

Human-annotation-based win-rate.

We ran a blind pairwise human evaluation on 50 randomly selected papers from the held-out test split. For each paper, two annotators compared the proposal from IdeaScientist-Trained with proposals from other systems on the same backbone, including IdeaScientist-Base. System names were hidden. Figure2(b) reports the win rates: IdeaScientist-Trained beats every other state-of-the-art system by a statistically significant margin, with DeepScientist the only exception. IdeaScientist-Trained also wins more often than IdeaScientist-Base, although the difference is not statistically significant.

(a)Novelty across research domains

(b)IdeaScientist-Trained vs. each system

Figure 2:Proposal-level results.(a)Proposal novelty across research domains for different systems on Qwen3.6-27B; the number of test papers per domain is given in parentheses.(b)Human paired win rate of IdeaScientist-Trained against each system on matched papers. Quality averages all seven dimensions, and error bars show bootstrap95%95\%confidence intervals.

Case Studies.

We manually examined differences between proposals generated by different systems, with representative examples presented in AppendixI. IdeaScientist-Trained generally makes a tighter connection between the stated failure and the mechanism used to address it. Its key decision rules follow from the structure of the problem, making the underlying assumptions and expected effects easier to test. In comparison, baseline methods introduce plausible control rules with weaker justification for why those rules address the failure.

5.3Q2: How Much Does Each Element Contribute?

(a)Sub-agent ablation: Qwen3.6-27B

(b)Sub-agent ablation: Gemma-4-31B

(c)Innovator retrieval ablation: Qwen3.6-27B

Figure 3:Ablation study.(a–b)Sub-agent ablations: adding the Gap Finder and Innovator on top of the Report Writer improves All and Cutoff novelty by up to 7.7% on Qwen3.6-27B and up to 13.0% on Gemma-4-31B.(c)Innovator retrieval ablation: cross-domain retrieval improves novelty over same-domain retrieval, most strongly on In-domain novelty (+30.7%).#### Sub-agent contributions.

We first measure how much each sub-agent contributes to novelty. Figure3(a–b) compares three configurations: the Report Writer alone (the minimal setup that can write a proposal), the Report Writer with the Gap Finder, and the full pipeline, which adds the Innovator. Each added sub-agent raises both All and Cutoff novelty scores, and the weaker backbone (Gemma-4-31B) benefits more from the pipeline. Identifying research gaps first and then innovating on them helps the system reach ideas that are hard to derive from the problem statement alone.

Innovator retrieval scope.

We next vary the Innovator’s retrieval scope, comparing same-domain retrieval with cross-domain retrieval. As in Table1, we measure novelty against three scopes of literature. We add a fourth, stricter measure,Mechanism non-obviousness: we prompt Qwen3.6-27B to independently derive candidate solutions from close prior work and score the proposal’s novelty against these derived solutions. As Figure3(c) shows, cross-domain retrieval improves every novelty measure: All (+5.3%), Cutoff (+3.7%), and most strongly In-domain (+30.7%, from 36.3% to 67.0%). Mechanism non-obviousness is the hardest measure for both variants, but cross-domain retrieval still raises it from 33.5% to 35.7% (+2.2%).

5.4Q3: Can We Evaluate Research Proposal Quality Reliably?

(a)Pearson correlation

(b)Consensus agreement

(c)Self-preference bias

Figure 4:Judge reliability supports our use of the Qwen3.6-27B judge.(a)Pearson correlation between the two human annotators, or between each judge and their averaged ratings; the last column averages across the seven dimensions.(b)System-level ranking agreement with the leave-one-out consensus of the other four judges.(c)Excess self-preference bias measures how much more a judge rates its own model family than the other judges do.To check whether the LLM-judged quality scores are trustworthy, we compare them with scores from two human experts on 50 proposals generated by 9 systems. The two annotators scored the same quality dimensions as the LLM judges. Figure4(a) reports two Pearson correlations: between the two annotators on individual proposals, and between each LLM judge and the averaged human scores. We make two observations. First, scoring research proposals is inherently subjective, and even the two experts disagree to a notable degree (r=0.56r=0.56), much as reviewers do in conference peer review. Second, each LLM judge correlates with the human average at a level comparable to the agreement between the two annotators (r=0.51r=0.51–0.590.59for top-3 judges). LLM judges are therefore a reasonable proxy for expert assessment.

Figure4(b–c) further measures two properties of each judge: its agreement with the cross-judge consensus, and its self-preference bias toward generation systems from its own model family. Qwen3.6-27B agrees strongly with the consensus, second only to the frontier Claude-4.8 judge, and the difference between the two is not statistically significant. Its self-preference bias is negligible and lower than Claude-4.8’s, justifying why we chose it as the main judge. AppendixHreports report-level agreement and the full cross-judge breakdown.

6Conclusion

IdeaScientist isolates research ideation as a capability that can be trained and evaluated independently of downstream implementation. Our system identifies limitations in closely related work, searches other problem settings for mechanisms that may address the same challenge, and develops them into grounded research proposals. Using role-specific training with temporally held-out human papers as reference directions, IdeaScientist improves each stage of ideation and becomes the strongest open system in our proposal-level evaluation. Our results suggest that structured agent environments can make scientific ideation amenable to controlled learning and evaluation, rather than treating it as an opaque byproduct of end-to-end autoresearch.

AI use statement

We used generative AI tools only to polish phrasing and correct grammatical errors. We reviewed all AI-assisted edits and take responsibility for the final content of this work.

References

• Akellaet al.(2025)A. P. Akella, H. V. Siravuri, and S. RohatgiPre-review to peer review: pitfalls of automating reviews using large language models.arXiv preprint arXiv:2512.22145.Cited by:§2.2.
• Aroraet al.(2025)R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel,et al.Healthbench: evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775.Cited by:§2.3.
• Baeket al.(2025)J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. HwangResearchagent: iterative research idea generation over scientific literature with large language models.InProceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: human language technologies (volume 1: long papers),pp. 6709–6738.Cited by:§1,§2.1,§5.1.
• Camelo-Guerrero and Diaz-Rodriguez (2026)A. Camelo-Guerrero and J. Diaz-RodriguezHow closely do llm reviews align with human peer review?.arXiv preprint arXiv:2608.03659.Cited by:§2.2.
• Chenet al.(2025)M. Chen, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang,et al.Research: learning to reason with search for llms via reinforcement learning.arXiv preprint arXiv:2503.194702(3),pp. 7.Cited by:§2.3.
• Chenet al.(2026)Z. Chen, K. Zhao, J. Fu, D. Liang, Y. Wu, J. Li, H. Xue, X. Zeng, Y. Zhen, F. Xu,et al.Ideation arena: evaluating llm generated research ideas with battle-style human expert assessment.arXiv preprint arXiv:2608.29696.Cited by:§2.2.
• Choi and Song (2026)Y. Choi and M. SongRead, grep, and synthesize: diagnosing cross-domain seed exposure for llm research ideation.arXiv preprint arXiv:2605.11532.Cited by:§2.1.
• Denget al.(2026)H. Deng, X. Ke, Y. Li, R. Hu, D. Huang, D. F. Wong, Y. Wang, X. Liu, and M. ZhangCoCoReviewBench: a completeness-and correctness-oriented benchmark for ai reviewers.arXiv preprint arXiv:2605.07905.Cited by:§2.2.
• Dhuliawalaet al.(2024)S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. WestonChain-of-verification reduces hallucination in large language models.InFindings of the association for computational linguistics: ACL 2024,pp. 3563–3578.Cited by:§2.2.
• Falcket al.(2026)D. Falck, S. Sabri, A. Surina, T. Foster, A. Sims, S. Devlin, D. Rogers, T. Collins, K. Aleksiev, L. Kirsch,et al.Training ai scientists to replicate research.arXiv preprint arXiv:2608.13331.Cited by:§2.1.
• Fosteret al.(2026)T. S. Foster, B. A. Omari, T. Fu, T. Mann, C. Domond, L. Cipolina-Kun, B. Gauri, M. Aghamelu, A. D. Goldie, E. Helenowski,et al.AI research preference models.arXiv preprint arXiv:2608.13940.Cited by:§2.2.
• Gentner (1983)D. GentnerStructure-mapping: a theoretical framework for analogy.Cognitive science7(2),pp. 155–170.Cited by:§1.
• Ghafarollahi and Buehler (2025)A. Ghafarollahi and M. J. BuehlerSciAgents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning.Advanced Materials37(22),pp. 2413523.Cited by:§2.1,§5.1.
• Gick and Holyoak (1983)M. L. Gick and K. J. HolyoakSchema induction and analogical transfer.Cognitive psychology15(1),pp. 1–38.Cited by:§1.
• Goelet al.(2025)S. Goel, R. Hazra, D. Jayalath, T. Willi, P. Jain, W. F. Shen, I. Leontiadis, F. Barbieri, Y. Bachrach, J. Geiping,et al.Training ai co-scientists using rubric rewards.arXiv preprint arXiv:2512.23707.Cited by:§2.3.
• Gottweiset al.(2026)J. Gottweis, W. Weng, A. Daryin, T. Tu, P. Sirkovic, A. Myaskovsky, G. Glowaty, F. Weissenberger, A. Orlandi, D. Popovici,et al.Accelerating scientific discovery with co-scientist.Nature,pp. 1–3.Cited by:§1,§2.1.
• Guet al.(2026)C. Gu, J. Cheng, M. Zhang, P. Zheng, J. Zheng, and G. HeMoRI: learning motivation-grounded reasoning for scientific ideation in large language models.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 34838–34869.Cited by:§2.1.
• Gunjalet al.(2026)A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains.InInternational Conference on Learning Representations,Vol.2026,pp. 127924–127945.Cited by:§2.3.
• Guoet al.(2025)S. Guo, A. H. Shariatmadari, G. Xiong, A. Huang, M. Kim, C. M. Williams, S. Bekiranov, and A. ZhangIdeabench: benchmarking large language models for research idea generation.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2,pp. 5888–5899.Cited by:§2.2.
• Hoet al.(2026)S. Ho, M. Liu, H. Nghiem, and F. HuangSoundnessBench: can your ai scientist really tell good research ideas from bad ones?.arXiv preprint arXiv:2605.30329.Cited by:§2.2.
• Huet al.(2024)X. Hu, H. Fu, J. Wang, Y. Wang, Z. Li, R. Xu, Y. Lu, Y. Jin, L. Pan, and Z. LanNova: an iterative planning and search approach to enhance novelty and diversity of llm generated ideas.arXiv preprint arXiv:2410.14255.Cited by:§2.1.
• Huanget al.(2025)Z. Huang, Y. Zhuang, G. Lu, Z. Qin, H. Xu, T. Zhao, R. Peng, J. Hu, Z. Shen, X. Hu,et al.Reinforcement learning with rubric anchors.arXiv preprint arXiv:2508.12790.Cited by:§2.3.
• Idahl and Ahmadi (2025)M. Idahl and Z. AhmadiOpenReviewer: a specialized large language model for generating critical scientific paper reviews.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations),pp. 550–562.Cited by:§2.2.
• Ifarganet al.(2025)T. Ifargan, L. Hafner, M. Kern, O. Alcalay, and R. KishonyAutonomous llm-driven research—from data to human-verifiable research papers.NEJM AI2(1),pp. AIoa2400555.Cited by:§2.1.
• Jiet al.(2026)F. Ji, Y. Li, J. Yang, J. Wu, F. Zhang, Z. Cui, Y. Xie, M. Peng, Q. Xie, X. Chen,et al.Style wins, substance loses: a diagnosis of llm-as-judge in idea generation.arXiv preprint arXiv:2608.01666.Cited by:§2.2.
• Jiang (2026)B. JiangHindSight: evaluating llm-generated research ideas via future impact.arXiv preprint arXiv:2603.15164.Cited by:§2.2.
• Jinet al.(2025)B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. HanSearch-r1: training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516.Cited by:§2.3.
• Jinet al.(2026)J. Jin, Y. Hu, K. Qiu, Q. Dai, C. Luo, G. Dong, X. Li, T. Zhao, X. Ma, G. Zhang,et al.Toward generalist autonomous research via hypothesis-tree refinement.arXiv preprint arXiv:2606.11926.Cited by:§2.1.
• Keyaet al.(2026)F. Keya, G. Rabby, S. Auer, S. Vahdati, P. Mitra, and M. Y. JaradehSci-idea: context-aware scientific ideation using token and sentence embeddings.Machine Learning115(5),pp. 103.Cited by:§1.
• Liet al.(2026a)L. Li, J. Xiong, C. Zhu, R. Yu, C. Chen, J. Wang, R. Ma, and Z. LuLLM-as-a-reviewer: benchmarking their ability, divergence, and prompt injection resistance as paper reviewers.arXiv preprint arXiv:2605.25415.Cited by:§2.2.
• Liet al.(2025a)L. Li, W. Xu, J. Guo, R. Zhao, X. Li, Y. Yuan, B. Zhang, Y. Jiang, Y. Xin, R. Dang,et al.Chain of ideas: revolutionizing research via novel idea development with llm agents..InEMNLP (Findings),pp. 8971–9004.Cited by:§2.1.
• Liet al.(2026b)M. Li, C. Wang, X. Li, X. Zeng, D. Li, P. Shi, D. Zhou, and T. ZhouHow can rhetoric reward-hack ai reviewers? dissecting rhetorical sensitivity in ai-based peer review.arXiv preprint arXiv:2608.08975.Cited by:§2.2.
• Liet al.(2025b)W. Li, J. Lin, Z. Jiang, J. Cao, X. Liu, J. Zhang, Z. Huang, Q. Chen, W. Sun, Q. Wang,et al.Chain-of-agents: end-to-end agent foundation models via multi-agent distillation and agentic rl.arXiv preprint arXiv:2508.13167.Cited by:§2.3.
• Liet al.(2026c)X. Li, Y. Yang, P. Li, X. Yao, S. Liu, X. Ye, Z. Lu, Y. Yao, Y. Li, Y. Li,et al.Tree-of-ideas: automated research ideation via cross-trajectory reasoning over scholarly evolution.arXiv preprint arXiv:2608.10740.Cited by:§2.1.
• Liet al.(2026d)Z. Li, H. Zhang, S. Han, S. Liu, J. Xie, Y. Zhang, Y. Choi, J. Y. Zou, and P. LuIn-the-flow agentic system optimization for effective planning and tool use.InInternational Conference on Learning Representations,Vol.2026,pp. 50524–50570.Cited by:§2.3.
• Liuet al.(2026a)J. Liu, S. Qiu, M. Li, B. Li, H. Ji, S. Han, X. Ye, P. Xia, Z. Dong, M. Chen,et al.Autoresearchclaw: self-reinforcing autonomous research with human-ai collaboration.arXiv preprint arXiv:2605.20025.Cited by:§2.1,§5.1.
• Liuet al.(2025)J. Liu, K. Dhole, Y. Wang, H. Wen, S. Zhang, H. Mao, G. Li, N. Varshney, J. Liu, and X. PanToward honest language models for deductive reasoning.arXiv preprint arXiv:2511.09222.Cited by:§B.2.
• Liuet al.(2026b)J. Liu, L. Zhang, Y. Yang, Y. He, Y. Wang, W. Xuan, Z. Jin, and M. DiabMixSD: mixed contextual self-distillation for knowledge injection.arXiv preprint arXiv:2605.16865.Cited by:Table 13.
• Liuet al.(2026c)J. Liu, T. J. Zhang, R. Faulkner, X. A. Huang, V. Zouhar, D. Glandorf, I. Dahlgren, R. Dagli, Y. Chen, F. Leeb, V. Q. Truong, P. S. Pandey, Y. Bicker, S. Majumder, W. Jiang, Z. Qiu, S. Pal Chowdhury, M. Sachan, B. Schölkopf, M. T. Diab, and Z. JinPaperMentor: a human-centered multi-agent writing tutor for AI research papers in overleaf.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),G. Durrett and P. Jian (Eds.),San Diego, California, United States,pp. 397–407.External Links:Link,Document,ISBN 979-8-89176-392-0Cited by:Appendix C.
• Liuet al.(2026d)T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. WangOpenrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 17417–17437.Cited by:§2.3.
• Loewensteinet al.(1999)J. Loewenstein, L. Thompson, and D. GentnerAnalogical encoding facilitates knowledge transfer in negotiation.Psychonomic Bulletin & Review6(4),pp. 586–597.Cited by:§1.
• Luet al.(2024)C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. HaThe ai scientist: towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292.Cited by:§1,§2.1.
• Luet al.(2026)C. Lu, C. Lu, R. T. Lange, Y. Yamada, S. Hu, J. Foerster, D. Ha, and J. CluneTowards end-to-end automation of ai research.Nature651(8107),pp. 914–919.Cited by:§2.1.
• Luoet al.(2025)M. Luo, N. Jain, J. Singh, S. Tan, A. Patel, Q. Wu, A. Ariyak, C. Cai, T. Venkat, S. Zhu, B. Athiwaratkun, M. Roongta, C. Zhang, L. E. Li, R. A. Popa, K. Sen, and I. StoicaDeepSWE: training a fully open-sourced, state-of-the-art coding agent by scaling RL.Together AI.External Links:LinkCited by:§2.3.
• Lyuet al.(2026)Y. Lyu, X. Zhang, X. Yi, Y. Zhao, S. Guo, W. Hu, J. Piotrowski, J. Kaliski, J. Urbani, Z. Meng,et al.Evoscientist: towards multi-agent evolving ai scientists for end-to-end scientific discovery.arXiv preprint arXiv:2603.08127.Cited by:§2.1.
• Mahmoudet al.(2026)A. Mahmoud, M. Rezaei, Z. Wang, A. Gunjal, B. Liu, and Y. HeReward hacking in rubric-based reinforcement learning.arXiv preprint arXiv:2605.12474.Cited by:§2.3.
• Mednick (1962)S. MednickThe associative basis of the creative process..Psychological review69(3),pp. 220.Cited by:§1.
• Minet al.(2023)S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. HajishirziFActScore: fine-grained atomic evaluation of factual precision in long form text generation.InProceedings of the 2023 conference on empirical methods in natural language processing,pp. 12076–12100.Cited by:§2.2.
• Mitcheneret al.(2025)L. Mitchener, A. Yiu, B. Chang, M. Bourdenx, T. Nadolski, A. Sulovari, E. C. Landsness, D. L. Barabasi, S. Narayanan, N. Evans,et al.Kosmos: an ai scientist for autonomous discovery.arXiv preprint arXiv:2511.02824.Cited by:§2.1.
• Motwaniet al.(2024)S. R. Motwani, C. Smith, R. J. Das, R. Rafailov, I. Laptev, P. H. Torr, F. Pizzati, R. Clark, and C. S. de WittMalt: improving reasoning with multi-agent llm training.arXiv preprint arXiv:2412.01928.Cited by:§2.3.
• Nguyenet al.(2026)D. Nguyen, W. Hao, Y. Elazar, and C. TanBenchmarking agentic review systems.arXiv preprint arXiv:2606.19749.Cited by:§2.2.
• Normanet al.(2026)J. D. Norman, M. U. Rivera, and D. A. HughesReliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias.arXiv preprint arXiv:2606.19544.Cited by:§2.2.
• OpenAI (2026)OpenAIResearch acceleration: the view inside OpenAI.Note:https://openai.com/index/research-acceleration-view-inside-openai/Accessed: 2026-09-14Cited by:§1.
• Panicksseryet al.(2024)A. Panickssery, S. R. Bowman, and S. FengLlm evaluators recognize and favor their own generations.Advances in Neural Information Processing Systems37,pp. 68772–68802.Cited by:§2.2.
• Parket al.(2025)C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J. KimMaporl: multi-agent post-co-training for collaborative large language models with reinforcement learning.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 30215–30248.Cited by:§2.3.
• Radenskyet al.(2024)M. Radensky, S. Shahid, R. Fok, P. Siangliulue, T. Hope, and D. S. WeldHuman-llm compound system for scientific ideation through facet recombination and novelty evaluation.arXiv preprint arXiv:2409.14634.Cited by:§2.1.
• Ruanet al.(2026)K. Ruan, X. Wang, J. Hong, P. Wang, Y. Liu, and H. SunEvaluating llms’ divergent thinking capabilities for scientific idea generation with minimal context.Nature communications17(1),pp. 3625.Cited by:§2.2.
• Salimiet al.(2026)M. Salimi, B. H. Mohtasham, A. Aghakasiri, M. Naieni, A. H. Qeysarbeigi, M. M. S. Nazer, Z. Azar, M. J. Siavoshani, and M. H. RohbanDebate as reward: a multi-agent reward system for scientific ideation via rl post-training.arXiv preprint arXiv:2604.16723.Cited by:§2.1.
• Schmidgallet al.(2025)S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. BarsoumAgent laboratory: using llm agents as research assistants.Findings of the Association for Computational Linguistics: EMNLP 2025,pp. 5977–6043.Cited by:§1,§2.1.
• Schopf and Färber (2026)T. Schopf and M. FärberIs this idea novel? an automated benchmark for judgment of research ideas.arXiv preprint arXiv:2603.10303.Cited by:§2.2.
• Shahidet al.(2025)S. Shahid, M. Radensky, R. Fok, P. Siangliulue, D. S. Weld, and T. HopeLiterature-grounded novelty assessment of scientific ideas.InProceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025),pp. 96–113.Cited by:§2.1.
• Shaoet al.(2025)R. Shao, A. Asai, S. Z. Shen, H. Ivison, V. Kishore, J. Zhuo, X. Zhao, M. Park, S. G. Finlayson, D. Sontag,et al.Dr tulu: reinforcement learning with evolving rubrics for deep research.arXiv preprint arXiv:2511.19399.Cited by:§2.3.
• Shaoet al.(2024a)Y. Shao, Y. Jiang, T. Kanell, P. Xu, O. Khattab, and M. LamAssisting in writing wikipedia-like articles from scratch with large language models.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),pp. 6252–6278.Cited by:§2.1.
• Shaoet al.(2024b)Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu,et al.Deepseekmath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§2.3,§3.2.
• Siet al.(2026)C. Si, T. Hashimoto, and D. YangThe ideation-execution gap: execution outcomes of llm-generated versus human research ideas.InInternational conference on learning representations,Vol.2026,pp. 74195–74240.Cited by:§2.2.
• Siet al.(2025)C. Si, D. Yang, and T. HashimotoCan llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.InInternational Conference on Learning Representations,Vol.2025,pp. 94003–94092.Cited by:§2.2.
• Sinhahajariet al.(2026)S. Sinhahajari, N. Majumder, and S. PoriaOn the limits of llm-as-judge for scientific novelty assessment.arXiv preprint arXiv:2606.12071.Cited by:§2.2.
• Songet al.(2026)D. Song, H. Zhang, D. Liu, Y. Liu, Z. Li, Z. Yuan, S. Zhang, H. P. Zou, Z. Yan, Y. Zhang,et al.Dr. claw: an ai scientist workspace for vibe research.arXiv preprint arXiv:2609.00365.Cited by:§2.1,§5.1.
• Songet al.(2024)Y. Song, Y. Kim, and M. IyyerVeriScore: evaluating the factuality of verifiable claims in long-form text generation.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp. 9447–9474.Cited by:§2.2.
• Suet al.(2025)H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Li, B. Qi, Q. Wu, H. Li, W. Ouyang,et al.Many heads are better than one: improved scientific idea generation by a llm-based multi-agent system.InProceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: long papers),pp. 28201–28240.Cited by:§2.1.
• Suzuki and Banaei-Kashani (2026)Y. Suzuki and F. Banaei-KashaniTCA-sir: learning target-conditioned abstractions for scientific inspiration retrieval.arXiv preprint arXiv:2607.28498.Cited by:§2.1.
• Tanget al.(2026)J. Tang, L. Xia, Z. Li, and C. HuangAi-researcher: autonomous scientific innovation.Advances in Neural Information Processing Systems38,pp. 9481–9520.Cited by:§2.1,§5.1.
• Uzziet al.(2013)B. Uzzi, S. Mukherjee, M. Stringer, and B. JonesAtypical combinations and scientific impact.Science342(6157),pp. 468–472.Cited by:§1.
• Wanget al.(2024a)Q. Wang, D. Downey, H. Ji, and T. HopeScimon: scientific inspiration machines optimized for novelty.InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers),pp. 279–299.Cited by:§2.1.
• Wanget al.(2024b)W. Wang, L. Gu, L. Zhang, Y. Luo, Y. Dai, C. Shen, L. Xie, B. Lin, X. He, and J. YeScipip: an llm-based scientific paper idea proposer.arXiv preprint arXiv:2410.23166.Cited by:§2.1.
• Wanget al.(2026a)X. J. Wang, H. Cui, M. P. Brenner, and S. VenugopalanIdea search: guiding tree search with ideas to explore diverse scientific methods.arXiv preprint arXiv:2608.08958.Cited by:§2.1.
• Wanget al.(2026b)X. Wang, Z. Hao, S. Hou, H. Peng, J. Li, and X. WangReproducing, analyzing, and detecting reward hacking in rubric-based reinforcement learning.arXiv preprint arXiv:2606.04923.Cited by:§2.3.
• Wanget al.(2024c)Y. Wang, Q. Guo, W. Yao, H. Zhang, X. Zhang, Z. Wu, M. Zhang, X. Dai, M. Zhang, Q. Wen,et al.Autosurvey: large language models can automatically write surveys.Advances in neural information processing systems37,pp. 115119–115145.Cited by:§2.1.
• Wanget al.(2026c)Y. Wang, Z. Lu, Y. Yan, K. Song, W. Zhang, W. Lu, J. Xiao, Y. Zhuang, and Y. ShenPaperGym: rubric-centered evolution for research-plan generation.arXiv preprint arXiv:2608.31119.Cited by:§2.1.
• Wanget al.(2026d)Z. Wang, A. Yuan, Y. Yao, L. Yao, H. Zuo, Z. Gong, Y. Liu, S. Li, Y. Cai, T. Yang,et al.What proves you wrong: benchmarking language models on falsifiable research ideation.arXiv preprint arXiv:2608.22948.Cited by:§2.2.
• Weiet al.(2026)Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. WangSwe-rl: advancing llm reasoning via reinforcement learning on open software evolution.Advances in Neural Information Processing Systems38,pp. 78500–78525.Cited by:§2.3.
• Wenet al.(2026)J. Wen, C. Si, Y. Chen, H. He, and S. FengPredicting empirical ai research outcomes with language models.Advances in Neural Information Processing Systems38,pp. 2663–2680.Cited by:§2.2.
• Wenget al.(2025)Y. Weng, M. Zhu, G. Bao, H. Zhang, J. Wang, Y. Zhang, and L. YangCycleresearcher: improving automated research via automated review.InInternational Conference on Learning Representations,Vol.2025,pp. 3669–3709.Cited by:§2.2,§5.1.
• Wenget al.(2026)Y. Weng, M. Zhu, Q. Xie, Q. Sun, Z. Lin, S. Liu, and Y. ZhangDeepscientist: advancing frontier-pushing scientific findings progressively.InInternational Conference on Learning Representations,Vol.2026,pp. 47981–48037.Cited by:§2.1,§5.1.
• Xuet al.(2025a)Y. Xu, T. Chakraborty, S. Sharma, L. Nunes, S. Sharma, K. D. Demopulos, E. Kıcıman, S. Lu, and R. ChandraDirect reasoning optimization: token-level reasoning reflectivity meets rubric gates for unverifiable tasks.arXiv preprint arXiv:2506.13351.Cited by:§2.3.
• Xuet al.(2025b)Z. Xu, Y. Zhao, M. Patwardhan, L. Vig, and A. CohanCan llms identify critical limitations within scientific research? a systematic evaluation on ai research papers.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 20652–20706.Cited by:§2.1.
• Yamadaet al.(2025)Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. HaThe ai scientist-v2: workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066.Cited by:§1,§2.1.
• Yanet al.(2025)X. Yan, S. Feng, J. Yuan, R. Xia, B. Wang, L. Bai, and B. ZhangSurveyforge: on the outline heuristics, memory-driven generation, and multi-dimensional evaluation for automated survey writing.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 12444–12465.Cited by:§2.1.
• Yanget al.(2026a)M. Yang, X. Guo, U. Tyagi, M. Zhang, R. Dumitru, S. Hou, Y. He, D. Y. Zhang, and Y. LiuRubric dropout: a simple way to mitigate reward hacking in rubric-as-reward rl.arXiv preprint arXiv:2608.11669.Cited by:§2.3.
• Yanget al.(2026b)X. Yang, S. Sarwar, J. Cheng, Z. Shi, D. Li, H. Chen, H. Zhang, C. Guo, J. He, and Z. LiaoScaling automatic research agents via world models.arXiv preprint arXiv:2608.12564.Cited by:§2.1.
• Zenget al.(2025)S. Zeng, K. Tian, K. Zhang, Y. Wang, J. Gao, R. Liu, S. Yang, J. Li, X. Long, J. Ma,et al.Reviewrl: towards automated scientific review with rl.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp. 16942–16954.Cited by:§2.2.
• Zhanget al.(2026a)L. Zhang, J. Liu, and T. AugustRe-centering humans in llm personalization.arXiv preprint arXiv:2606.06614.Cited by:§2.2.
• Zhanget al.(2026b)Y. Zhang, K. Zhou, Z. Xu, K. Ramnath, Y. Zhou, S. Woo, H. Ding, and L. L. CheongLearning to ideate for machine learning engineering agents.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers),pp. 436–447.Cited by:§2.1.
• Zhaoet al.(2026)W. Zhao, M. Yuksekgonul, S. Wu, and J. ZouSirius: self-improving multi-agent systems via bootstrapped reasoning.Advances in Neural Information Processing Systems38,pp. 124475–124504.Cited by:§2.3.
• Zhenget al.(2023)L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,et al.Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36,pp. 46595–46623.Cited by:§2.2.
• Zhouet al.(2025)Y. Zhou, S. Li, S. Liu, W. Fang, K. Zhang, J. Zhao, J. Yang, Y. Zhou, J. Lv, T. Zheng,et al.Breaking the exploration bottleneck: rubric-scaffolded reinforcement learning for general llm reasoning.arXiv preprint arXiv:2508.16949.Cited by:§2.3.
• Zhuet al.(2025)M. Zhu, Y. Weng, L. Yang, and Y. ZhangDeepreview: improving llm-based paper review with human-like deep thinking process.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 29330–29355.Cited by:§2.2,§5.1.

Appendix AFramework Details

A.1Action Space and Retrieval

Each agent acts in turns. On a turn it may perform one or more permitted operations in parallel or hand control back. For the orchestrator, handing back control ends the run; for a sub-agent, it returns the result to the role that invoked it. Table2summarizes these permissions in terms of what each role can access, modify, and return.

The orchestrator performs no literature retrieval or paper reading. It can spawn any producer or reviewer sub-agent, view any file a sub-agent wrote, and modify only the research plan. If it attempts to stop before the final proposal exists, it receives up to eight continuation prompts; the run is also bounded by a maximum turn count. Each producer and reviewer sub-agent can retrieve and request readings of papers and can write its working file. A producer that attempts to return before writing at least 20 non-whitespace characters to its required artifact receives up to three completion prompts. The reviewer has no analogous artifact gate. Each first-tier sub-agent may request at most 20 paper readings per task and is bounded by a role-specific turn limit. For the reported runs, the paper reader receives one paper’s full text in a separate single-turn context and returns a digest, so full text never enters a first-tier agent’s context.

Retrieval regimes.

Every retrieving role uses the same keyword-search interface over the offline corpus, but the roles are instructed to use it in the two different regimes described in §3.1. The regime is therefore determined by how a role formulates queries rather than by a separate tool. The gap finder is given one challenge axis and instructed to issue four to eight queries phrased in the exact terminology of that axis and of the target problem definition. It re-queries with rephrasings while the returned pool is thin so that the same-challenge, same-problem-definition prior art is retrieved as completely as possible. Its gap schema then requires the near-miss papers and the reason each falls short (Table3), which is only satisfiable from that regime. The innovator is instructed to issue six to ten queries hunting for a mechanism whose logic could transfer to the assigned gap, preferring queries that leave the problem’s own sub-field and span at least three distinct sub-fields. An in-domain source is admissible only when the transfer is non-trivial rather than a rediscovery of the closest prior work, and the resulting candidate must name the source paper, the mechanism, and the condition the transfer relies on. The reviewer searches in the gap finder’s regime, since its task is to falsify the novelty claim against same-problem prior art. The report writer searches in whichever regime the claim it is grounding requires, because it must cite both the related work the proposal displaces and the source of the mechanism it proposes.

Table 2:Role-specific information access, workspace permissions, and handoffs. Only the paper reader sees full paper text; all other roles use bibliographic metadata and task-focused reader digests.

A.2Agent Output Schemas

Each producing role must write its artifact in a fixed schema, which is identical at training and deployment and is the surface the reward and the deterministic checks operate on. Tables3through7list the entries of each schema and the instruction attached to each entry. The gap finder appends oneAxissection per assigned axis, containing a mechanism-level summary of what has been done and a list of at most two gaps, each following the gap schema. The innovator appends one candidate block per call, the reviewer appends one verdict block per reviewed candidate, and the report writer emits a single JSON object in the results-masked schema with every cited work referenced inline by its database id. The orchestrator’s plan file is a free-form planning document and is not schema-constrained.

Table 3:Gap finder output schema (gaps.md). Each assigned axis produces one section with a summary of prior work and at most two gaps. Every gap fills the entries below.Table 4:Innovator output schema (candidates.md). Each call appends one candidate block with the entries below.Table 5:Report writer output schema. The report is stored as a single JSON object in the results-masked schema; nested entries are summarized here. Every cited work is referenced inline by its database id, and no measured results are included.Table 6:Reviewer output schema (scores.md). The reviewer appends one verdict block per reviewed candidate.Table 7:Paper reader output schema (papers/<id>.md), a cumulative digest returned to the requesting agent.

Appendix BTraining Details

B.1Role-Specific Training and Rewards

Why we train roles separately.

Our initial design optimized the complete pipeline using only the final proposal reward. The base model rarely completed the pipeline, so most examples received no informative signal. Training each role separately avoids this cold start and connects every update to a clearly defined artifact reconstructed from a human paper.

Role environments.

Each training example contains the information expected at the corresponding stage of the harness. The gap finder receives a problem definition and challenge axis, the innovator receives a verified gap analysis, and the report writer receives an accepted candidate. Every role can search the offline database, request targeted readings, and revise its artifact over multiple turns using the same interface available at deployment.

Validity checks.

Rubric rewards are computed only after an output passes the applicable structural and citation checks. For the gap finder and innovator, the rubric reward also requires full credit on a judged methodological-focus or methodological-contribution criterion. Gap analyses must remain on the assigned axis and contain no more than two gaps. Reports must complete every required field. Every cited paper must have been read during the current stage or supplied through a verified earlier stage.

Role-specific quality rubrics.

The reference-free score captures methodological specificity, importance, feasibility, evidence, and clarity. The reference-grounded score compares the substantive fields of an output with the privileged reference while allowing a valid alternative to score well. The citation F1 score measures overlap between generated and reference citation sets. For every role, the complete reward is

r=0.85​g​(rfree+rref+rcit3)+0.15​rproc,r=0.85\,g\left(\frac{r_{\mathrm{free}}+r_{\mathrm{ref}}+r_{\mathrm{cit}}}{3}\right)+0.15\,r_{\mathrm{proc}},whereg∈{0,1}g\in\{0,1\}is the validity gate andrprocr_{\mathrm{proc}}is a rule-based compliance term that penalizes protocol violations, including disallowed tool calls, invalid formats, and missing required fields. It does not positively reward the volume of searching or reading. The gap finder’s reference-grounded score averages its fields uniformly; the innovator and report writer give important fields weight22and optional fields weight11(Table8).

Table 8:Per-role validity gates. For every role,r=0.85​g​(rfree+rref+rcit)/3+0.15​rprocr=0.85\,g(r_{\mathrm{free}}+r_{\mathrm{ref}}+r_{\mathrm{cit}})/3+0.15\,r_{\mathrm{proc}}, whereg∈{0,1}g\in\{0,1\}andrprocr_{\mathrm{proc}}is the rule-based protocol-compliance term defined in AppendixB.1. Thus a failed validity gate removes the rubric reward while preserving the compliance term. Reference-grounded fields are averaged uniformly for the gap finder and by importance tier for the innovator and report writer.

Judge and optimization summary.

Qwen3.6-27B scores the rubric with reasoning disabled. Invalid judge responses are retried and excluded if they remain unparseable. A failed validity gate sets the rubric component to zero, while the small process-shaping term remains available. We train one LoRA adapter per role using asynchronous GRPO with a maximum policy staleness of one step and an importance sampling correction for the resulting staleness. Sequence lengths and the number of training steps are calibrated per role, and the full optimization configuration is reported in AppendixE.1.

B.2Training Design Choices

Before settling on the final reward and environment, we explored several alternatives.

Full-text reading over metadata.

We compared giving the reader the paper full text against a faster metadata-only digest. The metadata variant collapsed training, with a near-zero mean reward and most rollouts truncated at initialization, against a mean reward around0.50.5for the full-text reader, so we kept full-text reading and recovered throughput with asynchronous GRPO rather than by shortening the input.

Comparative innovator reward.

We also tested a pairwise innovator reward that asks whether each candidate field is better than its reference counterpart, in place of the absolute reference-grounded score. This variant increasingly favored longer and more elaborate answers: its training reward spiked to∼0.83{\sim}0.83, suggesting that the judge rewarded length and elaboration independently of substance, and then repeatedly collapsed. By contrast, the absolute-rubric reward remained stable around0.660.66, so we kept the absolute reference-anchored rubric.

Process shaping and validity gates.

We keep a small rule-based compliance term that deducts reward for protocol violations such as disallowed tool calls, invalid formats, and missing required fields. We also experimented with richer per-turn process rewards, including a deterministic reward for retrieval overlap with the reference citations and the injection of a privileged-reference rollout into each group(Liu et al., 2025). These were run as isolated experiments and not adopted into the final reward. We refined the validity gates over time. Gap depth moved from a soft penalty to a hard cap of two gaps per axis. The report writer switched from a reference list with anchors to bare inline citation IDs after about half of its early episodes failed the read-before-cite check. A brittle rule that rejected any numeric string was replaced by a judged penalty for stating predictions as measured results.

Optimization choices.

We tuned optimization for stability under asynchronous GRPO, using an asymmetric probability ratio clip with no KL penalty and generation budgets calibrated to the observed rollout-length distribution. During development we also tried higher learning rates and smaller global batch sizes and varied the number of training epochs per role. The final values are reported in AppendixE.1.

Appendix CEvaluation Metrics and Judge Prompts

Novelty measures.

We use the three novelty names reported in Table1.Allis a 1–10 judgment of how much the proposal adds beyond the supplied closest papers from the complete comparison corpus, including pre- and post-cutoff work.Cutoffapplies the same judgment using only papers published before the target cutoff.In-domainis the percentage of proposals that exhibit structural cross-setting transfer. A proposal receives one when it imports a mechanism from a different problem setting and adapts it through a shared underlying structure, and zero when the transfer is absent, superficial, or unsupported. Thus, unlike All and Cutoff, In-domain is a binary transfer judgment rather than a 1–10 comparison against a separate literature set. The innovator retrieval ablation additionally reportsMechanism non-obviousness, which asks how readily the central mechanism follows from the three nearest pre-cutoff mechanism-side papers and standard techniques available at the cutoff. We implement this as two separate judge calls. The first receives the target problem, challenge, and prior papers but never the proposal, and independently derives candidate mechanisms. The second receives these frozen candidates and the proposal, then measures whether any ex-ante candidate reaches the proposal’s central mechanism without adding a new hindsight-derived recipe. Four scored levels map to{0,1/3,2/3,1}\{0,1/3,2/3,1\}for trivial combination, routine extension, non-obvious step, and surprising, respectively; an invalid candidate set is unscored and retried. In-domain novelty and Mechanism non-obviousness are averaged over the fixed test-set denominator, with an unscored proposal contributing zero.

Proposal-quality measures and judge models.

Relevance measures how directly the solution addresses the supplied problem and challenge; clarity measures organization and unambiguous presentation; specificity measures concrete technical detail; actionability measures whether the method and evaluation can be implemented(Liu et al., 2026c); soundness measures the plausibility and internal consistency of the technical reasoning; and impact measures the importance of the contribution if successful. All are scored from 1 to 10. Clarity, specificity, actionability, soundness, and impact are assessed jointly from the proposal alone, whereas relevance is assessed with the target problem definition and challenge. The primary Qwen3.6-27B judge and the four corroborating judges—Claude-4.8, Gemma-4-31B, GPT-OSS-120B, and GPT-5.4—receive the same prompts and inputs. We therefore reproduce each distinct prompt once rather than duplicating it by judge model. Dynamic proposal, literature, and few-shot-example blocks are shown as angle-bracketed placeholders; line breaks are normalized for print.

Proposal quality: clarity, specificity, actionability, soundness, and impact.

SYSTEM:

Youareameticulousseniorreviewerforatop-tierML/NLPvenue.Youwillbegiventhefulltextofamachine-generatedresearchproposal/report.JudgeitONITSOWNTERMS--youhavenoreferencepaperorindependentexperimentalresults.Scoreeachofthefivedimensionsbelowonaninteger1-10scaleandgiveasingleconcisesentenceofreasoningforeach.Judgeonlywhatisstated;donotinventsupportingdetailsorevidence.

Usethesesharedanchors:1-2=seriouslydeficient;3-4=majorgaps;5-6=partlyadequatewithconsequentialgaps;7-8=strongwithlimitedgaps;9-10=exceptionallystrongandwellsupportedbythesuppliedtext.

Dimensions:

-actionability:Couldacompetentresearcheractuallyexecutethis?Arethemethodsteps,designchoices,andexperimentsconcreteenoughtoimplement,notjustaspirational?

-specificity:Istheproposalpreciseanddetailed(concretemechanisms,equations,nameddatasets/baselines/metrics)ratherthanvagueorgeneric?

-clarity:Isitwell-organized,unambiguous,andeasytofollow?Areclaims,notation,andcontributionsstatedcleanly?

-impact:Ifitworks,howsignificantisthecontributiontothefield?Doesitaddressaproblemthatmatters?

-soundness:Isthetechnicalreasoningcorrectandinternallyconsistent?Areclaimssupported,assumptionsexplicit,andfailuremodesacknowledged(noobviouserrorsorover-claiming)?

ReturnONLYaJSONobjectofthisexactshape:

{

“actionability”:{“score”:<1-10>,“reasoning”:“<onesentence>”},

“specificity”:{“score”:<1-10>,“reasoning”:“<onesentence>”},

“clarity”:{“score”:<1-10>,“reasoning”:“<onesentence>”},

“impact”:{“score”:<1-10>,“reasoning”:“<onesentence>”},

“soundness”:{“score”:<1-10>,“reasoning”:“<onesentence>”}

}

FEW-SHOTEXAMPLE:

<few-shotexample>

USER:

#RESEARCHREPORTTOEVALUATE

<fullgeneratedproposal>

Proposal quality: relevance.

SYSTEM:

YouareanexpertreviewerassessingtheRELEVANCEofamachine-generatedproposedsolutiontotheresearchproblemitwasaskedtoaddress.YouaregiventhetargetPROBLEMDEFINITIONandCHALLENGE,thentheproposal(report).

JudgehowdirectlyandcompletelytheproposedsolutionaddressesTHISspecificproblemdefinitionandchallenge:afocusedsolutionthatsquarelytargetsthestatedchallengescoreshigh;asolutionthatdriftstoadifferentproblem,onlypartiallyaddressesit,orisgenericscoreslow.Scorerelevanceindependentlyofnovelty,correctness,andwritingquality.

Usetheseanchors:1-2=unrelatedoralmostunrelated;3-4=weakorgenericconnection;5-6=addressespartofthechallenge;7-8=directlyaddressesmostofit;9-10=directlyandfullytargetsit.

ReturnONLYaJSONobject:

{“relevance”:{“score”:<1-10>,“reasoning”:“<onesentenceonhowwellthesolutiontargetsthestatedproblem/challenge>”}}

FEW-SHOTEXAMPLE:

<few-shotexample>

USER:

#TARGETPROBLEMDEFINITION

<problemdefinition>

#TARGETCHALLENGE

<challenge>

#PROPOSEDSOLUTION(report)

<fullgeneratedproposal>

Novelty: All and Cutoff.

SYSTEM:

YouareanexpertreviewerassessingtheNOVELTYofamachine-generatedresearchproposalrelativetothesuppliedclosestrelatedwork.Youaregiventheproposaland,asreference,theproblem-definition,challenge,andsolutionoftheexistingpapersthattacklethemostsimilarproblem.Thecomparisonsetmayincludepaperspublishedaftertheproposal’stargetcutoff;judgeagainstthesuppliedreferences.

JudgehowmuchtheproposalcontributesBEYONDthisrelatedwork:genuinelynewmechanisms/ideas/combinationsscorehigh;rediscovering,renaming,ortriviallyrecombiningwhatthereferencepapersalreadydoscoreslow.Rewardnoveltythatisrealandnon-obvious,notnovelty-by-jargon.

Usetheseanchors:1-2=essentiallythesamecentralmechanismasareference;3-4=minoradaptationorstraightforwardcombination;5-6=adistinctstepwithsubstantialoverlap;7-8=amateriallydifferentcontribution;9-10=stronglydistinctrelativetothesuppliedreferences.

ReturnONLYaJSONobject:

{

“novelty”:{“score”:<1-10>,“reasoning”:“<onesentencenamingwhatisnewvs.whatalreadyexistsinthereferences>”},

“overlapping_prior_work”:[“<shorttitlesorideastheproposaloverlapswith>”]

}

FEW-SHOTEXAMPLE:

<few-shotexample>

USER:

#PROPOSALTOEVALUATE

<fullgeneratedproposal>

#CLOSESTPRIORWORK(referencepaperswithsimilarchallenge+problemdefinition)

<reference-paperblocksfromtheselectedtemporalcomparisonset>

Mechanism non-obviousness.

Call 1: proposal-blind candidate derivation.

SYSTEM:

Youareaseniorresearcherattemptingtosolveatargetresearchproblemusingonlythesuppliedpriorpapersandstandardtechniquesavailableatthetargetcutoff.Youhavenotseen,andmustnotspeculateabout,anyproposalthatwilllaterbeevaluated.

Independentlyderiveuptothreeconcretecandidatemechanismsthataddressthetargetchallenge.Eachcandidatemustspecify:

-itscentraltechnicaloperation;

-astep-by-stepderivationfromthesuppliedpapersandstandardtechniques;

-whichreferencesupplieseachnonstandardstep;and

-thekeyassumptionneededforthemechanismtowork.

Donotmerelyrestatetheproblem,listbroadresearchdirections,orusemechanismsunsupportedbythesuppliedmaterial.Anewcombinationisallowedonlywhenyouexplainconcretelyhowitspiecescompose.Ifnoconcretemechanismcanbederived,returnanemptycandidatelist.

ReturnONLYthisJSONobject:

{

“candidates”:[

{

“candidate_id”:“C1”,

“central_mechanism”:“<onesentence>”,

“derivation_steps”:[“<step1>”,“<step2>”,“...”],

“reference_uses”:[“<papertitleorid>:<piecesupplied>”],

“key_assumption”:“<onesentence>”

}

]

}

USER:

#TARGETPROBLEMDEFINITION

<problemdefinition>

#TARGETCHALLENGE

<challenge>

#NEARESTPRIORWORK(allpublishedbeforethetarget)

<threenearestmechanism-sideprior-paperblocks>

Derivecandidatemechanismsusingonlytheinformationabove.JSONonly.

Call 2: comparison against frozen candidates.

SYSTEM:

Youareaskepticalseniorreviewerassessingwhetheraproposal’scentralmechanismwasindependentlyderivablebeforetheproposalwasrevealed.Youreceivecandidatemechanismsproducedinaseparateproposal-blindcall,alongwiththesourcepriorpapersandtheproposalunderreview.

Thecandidatelistisfrozen.DoNOTinvent,repair,merge,orextendcandidatesafterreadingtheproposal.Priorpapersmaybeconsultedonlytoverifyafrozencandidate’sstatedderivation,nottoconstructanewproposal-conditionedrecipe.

Workinthisorder.

STEP1--Statetheproposal’scentraltechnicaloperationinonesentence.

STEP2--Comparethatoperationwitheveryfrozencandidate.Selecttheclosestcandidate,ifany,andidentifytheexactsharedstepsandtheconsequentialstepsabsentfromthecandidate.Ifnocandidatereachestheproposalmechanism,useNONE.Superficialtopicalsimilaritydoesnotcountasaderivation.

STEP3--Ifthecandidatelistisempty,malformed,ortoovaguetosupportastep-levelcomparison,setcandidate_set_validtofalseandreturnunscored;donotinfernon-obviousnessfromafailedderivationcall.Otherwise,assignonescoredlevel:

-“trivial_combination”:afrozencandidatealreadycontainsthecentralmechanismthroughdirectreuseorcombinationwithnomeaningfuladaptation.

-“routine_extension”:afrozencandidatereachesthecentralmechanismusingonlyordinaryimplementationsteps,withoutanewstructuralobservation.

-“non_obvious_step”:everyfrozencandidatemissesaconsequentialsteporstructuralobservationsuppliedbytheproposal.

-“surprising”:themissingconsequentialstepalsorunsagainstaclearexpectationestablishedbythesuppliedpapers.

Usetheupperlevelsonlywhenyoucanidentifytheconsequentialmissingstep.Donotrewardimpressivevocabulary,length,orconfidentphrasing.

ReturnONLYthisJSONobject:

{

“candidate_set_valid”:true|false,

“central_mechanism”:“<onesentence>”,

“closest_candidate_id”:“<candidateid,orNONE>”,

“shared_steps”:[“<steppresentinbothcandidateandproposal>”],

“missing_consequential_steps”:[“<stepabsentfromthecandidate>”],

“obviousness”:“trivial_combination”|“routine_extension”|“non_obvious_step”|“surprising”|“unscored”,

“hardest_to_guess_step”:“<onesentence,ornone>”,

“confidence”:<1-5>

}

FEW-SHOTEXAMPLE:

<few-shotexample>

USER:

#PROPOSAL-BLINDCANDIDATES(verbatimoutputofCall1)

<frozencandidateJSON>

#NEARESTPRIORWORK(allpublishedbeforethetarget)

<threenearestmechanism-sideprior-paperblocks>

#PROPOSALUNDERREVIEW

<proposalmechanismblock>

Comparetheproposalonlyagainstthefrozencandidates,thengradeit.JSONonly.

Novelty: In-domain (structural cross-setting transfer).

SYSTEM:

YouareaseniormachinelearningresearcherjudgingwhetheraresearchproposalexhibitsSTRUCTURALCROSS-SETTINGTRANSFER.ThisbinaryjudgmentisreportedasIn-domainnovelty.DoNOTjudgetheproposal’soverallqualityorcorrectness.

Assignscore1onlywhenALLofthefollowingaresupportedbytheproposal:

1.Itidentifiesasourceproblemsettingthatdiffersfromthetargetsetting.

2.Itimportsaconcretemechanism,abstraction,oranalysistechniquefromthatsourcesetting.Acitationorageneral-purposetoolalonedoesnotcount.

3.Itidentifiesasharedunderlyingstructurethatexplainswhythemechanismshouldtransfer.

4.Itspecifieshowthemechanismisadaptedtoexploitthatstructureinthetargetsetting.

Otherwiseassignscore0.Thisincludesnocross-settingtransfer,unsupportedtransferclaims,andsuperficialanalogiesthatnameanothersettingwithoutestablishingthesharedstructureandadaptation.

Supportthejudgmentwithoneverbatimquoteofatmost35wordsfromtheproposal.Ifnoquotesupportsallfourrequirements,assignscore0.

ReturnONLYthisJSONobject:

{

“source_setting”:“<shortphrase,ornull>”,

“target_setting”:“<shortphrase,ornull>”,

“transferred_mechanism”:“<shortphrase,ornull>”,

“shared_structure”:“<shortphrase,ornull>”,

“adaptation”:“<shortphrase,ornull>”,

“evidence_quote”:“<verbatimproposalspan,<=35words,ornull>”,

“reasoning”:“<onesentence>”,

“score”:<0or1>

}

FEW-SHOTEXAMPLE:

<few-shotexample>

USER:

#PROPOSAL

<proposalmechanismblock>

Judgestructuralcross-settingtransfer.JSONonly.

Appendix DData Construction and Ideation Process Decomposition

Qwen3.6-27B generates the primary reference artifacts for all three stages. A small residual set of target papers, on which Qwen3.6-27B fails validation after regeneration, is completed with GPT-5.4. Table9reports the per-stage counts across the training, validation, and test partitions. GPT-5.4 accounts for only7676of the44,93144{,}931stage-level artifacts, under0.2%0.2\%of the total, so the artifacts are overwhelmingly produced by the open model.

We fully deduplicate the partitions, ensuring that neither the validation nor test set overlaps the training set and that the validation and test sets are mutually disjoint.

Problem-formulation leakage audit.

Because the problem definition and challenge are reconstructed from the complete target paper, theoretically they could inadvertently retain method-specific clues to the target solution if the LLM that performs this task doesn’t follow instructions well. We manually inspected the extracted problem definition and challenge for 50 randomly sampled target examples, comparing each with its source paper. We found no instances (0/500/50) in which the input exposed the target method, including through method-specific terminology, assumptions, examples, or problem decompositions that would reveal the solution.

Table 9:Number of accepted reference artifacts generated by each model, per stage, across the training, validation, and test partitions. Qwen3.6-27B produces the primary set, and GPT-5.4 completes a small residual set of target papers on which Qwen3.6-27B does not converge after regeneration.

Appendix EExperiment Configuration

E.1Training and Inference Configuration

We train one LoRA adapter per role on the Qwen3.6-27B backbone with asynchronous GRPO. Each adapter uses rank3232, scalingα=64\alpha=64, and no dropout, applied to all linear modules except the output projection and the visual tower, which is unused for this text-only task. Optimization uses AdamW with a constant learning rate of5×10−65\times 10^{-6}, weight decay0.010.01, no warmup, and gradient norm clipping at1.01.0, in bfloat16 with tensor parallel size88on H100 GPUs. Each GRPO step draws6464prompts with88generations per prompt, giving512512rollouts per step, and applies two gradient updates per step at a training batch of256256rollouts. Training uses an asymmetric probability ratio clip of[0.2,0.28][0.2,0.28]with no KL penalty in the reward, a maximum policy staleness of one step, and an importance sampling correction. Sequence lengths and training limits are calibrated per role: the gap finder uses a maximum sequence length of26,62426{,}624tokens for up to22epochs or800800steps, the innovator30,72030{,}720tokens for up to11epoch or600600steps, and the report writer49,15249{,}152tokens for up to22epochs or600600steps. For each sub-agent, we select the checkpoint with the highest held-out validation reward; the test split is used only for final evaluation. The reported system uses gap-finder step 380, innovator step 310, and report-writer step 194. The judge is Qwen3.6-27B with reasoning disabled, sampled at temperature0.70.7and top-pp0.80.8. At evaluation the orchestrator, reviewer, and paper reader use the base backbone while the three selected adapters are composed into IdeaScientist-Trained. The trained variant and IdeaScientist-Base share the same offline retrieval interface, output schema, and reading and generation budget.

Reading and generation budget (IdeaScientist variants and backbone controls).

Within the harness each producer sub-agent runs under a fixed per-turn generation budget and a per-role turn cap: the gap finder up to1515turns at8,1928{,}192tokens per turn, the innovator up to2020turns at6,1446{,}144tokens per turn, and the report writer up to2020turns at16,38416{,}384tokens per turn. The orchestrator is bounded by at most100100turns overall. Reading is done only by the paper reader, which runs as a single-turn tool with a30,00030{,}000-token input cap per read and returns a digest. The three IdeaScientist-Base controls (Qwen3.6-27B, Gemma-4-31B, and GPT-OSS-120B) run under these identical caps. The open source autoresearch systems and the general purpose agents run their own native agent loops, and we do not impose these caps on them. Each role is trained on4040nodes of eight H100 GPUs, with3232nodes dedicated to policy optimization and88nodes to asynchronous rollout generation.

E.2Temporal Cutoff and Model-Memory Contamination

Our temporal split restricts retrieval to papers published before January 1, 2026, although knowledge stored in model parameters cannot be fully controlled. We randomly sampled 10 post-cutoff papers and queried each evaluated model about their contents, finding no evidence of reliable recall sufficient to compromise the evaluation; moreover, the task requires substantial reasoning and synthesis beyond memorizing a target paper. We therefore consider January 1, 2026 a reasonable cutoff that balances model capability with contamination risk.

Table 10:Public release dates and sources for the models used in our experiments.As a direct screening check, we manually queried each of the five models about 10 randomly sampled post-cutoff target papers (50 model–paper queries in total). Each query supplied only the paper title and asked the model to describe the paper’s contents, without retrieval or an abstract. None of the models identified or described any of the queried papers as known work. We use a title-only probe because supplying the abstract would itself reveal substantive content. This result provides evidence against obvious title-level memorization in the sampled cases, but it cannot detect latent, paraphrased, or method-level memory and therefore cannot rule out contamination.

We chose January 1, 2026 as a pragmatic tradeoff between model capability and temporal validity: it supplies a sufficiently broad pre-cutoff literature for this demanding task while reserving papers on or after that date from retrieval as future directions. The training, validation, and test partitions are all drawn from these post-cutoff targets, and the test partition remains held out from optimization. Accordingly, our results should be interpreted as performance under a controlled retrieval cutoff, not as proof that every evaluated model was parametrically unaware of every target paper. Residual model-memory contamination is a limitation of the retrospective evaluation.

Appendix FAdditional Experimental Results

F.1Role-Level Validation Reward

Table11compares each role’s held-out validation reward for IdeaScientist-Base with the reward at the checkpoint selected for IdeaScientist-Trained. All three roles improve: the gap finder by0.1010.101, the innovator by0.1070.107, and the report writer by0.1050.105. Figure5shows the complete training and validation trajectories.

Table 11:Held-out validation reward for IdeaScientist-Base and at each role’s selected IdeaScientist-Trained checkpoint.

F.2Training Diagnostics

Training and validation reward.

Figure5shows the per-step training and held-out validation rewards for all three trained roles.

Figure 5:GRPO reward over training for the gap finder (left,), innovator (middle,), and report writer (right,). Curves show the per-step training reward together with the held-out validation reward.

Reward components.

Decomposing the reward into its terms shows that the reference-free score begins high, while the reference-grounded and citation F1 scores provide more room for improvement.

Reward diversity.

Across27,22527{,}225gap finder episodes, the median reward is0.6030.603and no reward is nonpositive. The mean within group standard deviation is0.1040.104across3,4733{,}473groups, and only0.4%0.4\%of groups have zero variance. The reward therefore supplies a useful learning signal for nearly every sampled group.

Rollout length.

At the selected gap-finder checkpoint (step 380), the512512training episodes have a median sequence length of17.817.8K tokens, a 95th percentile of22.422.4K, and a maximum of25.825.8K under the26,62426{,}624-token limit. The median rollout contains66turns and reads66distinct papers; the corresponding 95th percentiles are99turns and99papers. Reward remains essentially uncorrelated with sequence length (r=−0.024r=-0.024), turns (r=−0.031r=-0.031), and papers read (r=−0.017r=-0.017).

F.3Generation Failure Analysis

We recompute generation failures from the final 277-problem evaluation, defining a failure as a test problem without a scorable proposal. IdeaScientist-Trained and all three open-backbone IdeaScientist-Base runs have no failures, whereas Claude Code SDK has55and Codex SDK has88.

F.4Checkpoint Selection

We trained each role over multiple checkpoints and evaluated more than one checkpoint combination. Table12compares the selected combination (gap finder step 380, innovator step 310, and report writer step 194) with the superseded combination (steps 122, 116, and 80) on the full test split under the same judge and metrics. The selected checkpoints improve valid generation, Citation F1, proposal quality, the reference-grounded score, the validity-gate pass rate, and the checkpoint-selection average.

Table 12:Reported IdeaScientist-Trained checkpoint versus an earlier checkpoint, scored by the same primary Qwen3.6-27B judge as the main results. Selection avg. averages all checkpoint-selection fields, including the proposal-quality dimensions, novelty scores, and field-level reference-grounded scores; it is distinct from Overall in Table1.

Appendix GResource Use and Test-Time Scaling

G.1Test-Time Compute

On the same Qwen3.6-27B backbone and harness, IdeaScientist-Trained uses168,916168{,}916output tokens per delivered proposal,11.1%11.1\%fewer than the189,942189{,}942used by IdeaScientist-Base, so its gains do not result from a larger generation budget. Within the fixed per-role turn and token caps, the per-run generation budget still varies across papers. That variation is endogenous: harder papers can drive more turns while scoring lower, whereas runs that fail the validity gate early spend less and score zero. Correlating the report-writer reward with per-run turns, output tokens, or LLM calls under the caps therefore shows no positive association; it is small and slightly negative for IdeaScientist-Trained and near zero for IdeaScientist-Base, reflecting paper difficulty rather than a compute effect. To isolate the effect of compute, we ran a controlled sweep on a fixed subset of 40 test papers with IdeaScientist-Base on Qwen3.6-27B, varying only the per-role turn budget across three levels while holding the reader settings, generation length, and retrieval constant. Figure6shows that raising the budget from the low to the high level lifts the report-writer reward from 0.290 to 0.465 and the completion rate, the fraction of papers that yield a proposal passing the validity gate, from 0.60 to 1.00, while the mean orchestrator turns actually consumed rise from 34 to 55, confirming that the smallest budget is genuinely compute-starved. Additional test-time compute thus raises quality once the caps themselves are relaxed, complementing the role-training gains reported in the main results.

Figure 6:Controlled test-time compute sweep on a fixed4040paper subset with IdeaScientist-Base on Qwen3.6-27B. Only the per-role turn budget varies across the three levels. The reader settings, generation length, and retrieval are held constant. Left: mean report-writer reward with standard error and completion rate, the fraction of papers that produced a proposal and passed the validity gate. Right: mean orchestrator turns consumed, confirming that the low level is genuinely compute-starved. Reward and completion rise with budget, showing that the harness benefits from additional test-time compute.

Appendix HJudge Reliability

The system-level agreement in Figure4(b) is computed by ranking the generation systems under each judge’s mean score for each metric family and taking the Spearman correlation with the leave-one-out mean of the other four judges, and with the Claude-4.8 judge, averaged across the metric families. The excess self-preference bias of a judge is the amount by which it rates its own model family’s generations above the other judges’ mean for that system, minus the judge’s average deviation from the other judges across all systems, so a uniformly lenient or harsh judge is not counted as biased. At the individual-report level the judges cluster similarly: the score-level Pearson agreement with the leave-one-out consensus is0.710.71for Qwen3.6-27B,0.770.77for Claude-4.8,0.770.77for Gemma-4-31B,0.670.67for GPT-OSS-120B, and0.780.78for GPT-5.4, placing every judge in the same reliable band.

Appendix ICase Studies

The four cases below compare the human method, IdeaScientist-Base, IdeaScientist-Trained, and seven external systems under the same problem definition, model setting, inputs, and desired output. For the first three methods, each table separates the core idea, mechanism, and distinguishing contribution or limitation; the seven baselines are grouped by method family and summarized individually. The accompanying analysis focuses on how directly each mechanism addresses the stated challenge and reports when the evaluation scores do not support a strict ranking.

I.1Composable Safety Alignment

Table 13:Case study on composable safety alignment. Every method receives a user questionxxand a setSSof safety rules enabled for that request and must produce an answer that follows those rules. The language model’s existing weights remain frozen.ComponentDescriptionProblem definitionLetfθf_{\theta}be a language model whose weightsθ\thetaremain fixed, and letC={c1,…,cK}C=\{c_{1},\ldots,c_{K}\}be the available safety categories. For each request, the system receives a user questionxxand a selected subsetS⊆CS\subseteq Cspecifying which categories are active. It must produce a responseyythat follows the rules inSS, including refusing or adapting an answer when an active rule requires it. It should still answer permitted questions helpfully and avoid refusing solely because of a category that is inactive. Small control components may be trained using examples of questions, active rule sets, and appropriate responses; the underlying model weights remain unchanged.ChallengeSafety rules vary by user and context, but a model trained for one fixed policy cannot easily switch between them. Written instructions may not reliably enforce several rules at once. The challenge is to make one frozen model apply the rules selected for each request while still answering permitted questions.Human paperCore idea — MOSAIC.Represent every safety category with a small set of learned control tokens and prepend the tokens for exactly the rules enabled on the current request.Mechanism.Train only these soft tokens on question–rule-set–answer examples, including harmless questions and repeated questions under different rule subsets; the language model remains frozen.Distinction / limit.Policy switching happens entirely at the input, making composition simple and modular: unlike Base it does not alter hidden states, and unlike Trained it performs no generation-time checking. Success still depends on the frozen model composing the tokens reliably.IdeaScientist-BaseCore idea.Model the enabled policies as a set whose members can interact, rather than adding independent control vectors: a Set Transformer produces an interaction-aware policy representation.Mechanism.Train rule embeddings, the set aggregator, and a FiLM controller on matched rule-set responses; FiLM uses the aggregate to scale and shift selected hidden signals while the language model stays frozen.Distinction / limit.Base is the only proposal here that explicitly learns nonlinear interactions among the active rules before steering. This may capture conflicts missed by additive methods, but it needs combination coverage and can destabilize pretrained representations(Liu et al., 2026b).IdeaScientist-TrainedCore idea.Treat composition as constraint satisfaction: learn an acceptable-state boundary for each rule and move the current activation into the intersection of all enabled boundaries.Mechanism.Train per-rule probes from acceptable and violating generations; at decoding time, solve for a nearby activation satisfying every active probe, then continue generation from the corrected state.Distinction / limit.Unlike weighted vectors, adapters, or reward scores, Trained seeks joint satisfaction rather than a compromise and minimizes activation displacement. Linear probe boundaries are only behavioral proxies, however, and feasible/nearest projection is not automatically guaranteed.Baselines: modular adaptersAutoResearchClawParses the active rules and routes through category-specific, disentangled adapters; a lightweight router combines the selected modules before frozen-model generation.DeepScientistTrains one nearly orthogonal LoRA adapter per category and an input-aware network to weight active adapters, optionally projecting to preserve general capabilities.Baselines: conditioning and additive steeringResearchAgentEncodes active rules as conditioning embeddings and guides the frozen model through a safety router, switching policy through the conditioning signal.CycleResearcherLearns one logit-steering vector per category, combines active vectors with a policy encoder, and trains first on single rules and then combinations.AI-ResearcherCombines rules into a hidden-state control vector, removes components associated with general capabilities, and context-gates the intervention strength.Baselines: decoding-time scoring or instructionsDr. ClawTrains a separate autoregressive reward model per category and combines the enabled models’ scores during decoding; a policy manager selects the active set.SciAgentsClassifies a safety action from the active policies and writes a refusal or redirection instruction, relying on the frozen model to follow that instruction.The human paper composes safety policies with learned input tokens, whereas Base models interactions among the active rules with a Set Transformer and FiLM. Trained changes the formulation more fundamentally: each rule defines a learned boundary over model states, and generation projects toward their joint feasible region. The baselines instead compose adapters, additive steering signals, reward scores, or generated instructions; even AI-Researcher’s projection protects a capability subspace rather than enforcing the intersection of active policy constraints.

I.2Adaptive Rollout Allocation

Table 14:Case study on adaptive rollout allocation for reasoning RL. Every method uses training questions with reference answers or an automatic correctness checker together with a student model. The methods differ in how they spend the rollout budget and construct the groups used for GRPO.ComponentDescriptionProblem definitionGiven a datasetDDof question–answer pairs(q,a)(q,a)and a student policyfθf_{\theta}, the model generates multiple answers, orrollouts, for each sampled question. A verifier compares each answer with the reference answer and assigns a binary correctness reward.The goal is to improve the student’s reasoning accuracy under a fixed compute budget by deciding which questions receive additional rollouts and how their samples contribute to training. GRPO calculates advantages by comparing rewards within each question’s rollout group. The baseline objective also includes a KL penalty to limit deviation from a reference policy.Inputs.A batch of questions with reference answers, the student policy, a compute budget, and the generated rollouts with their token log probabilities and correctness rewards.Output.Updated student policy parameters, evaluated on the model’s ability to answer questions independently.Every method uses training questions with reference answers or an automatic correctness checker together with a student model. Arolloutis one answer generated by the student, including its reasoning. Several rollouts for the same question form agroup. GRPO compares their rewards to calculate anadvantage, which is positive for answers that perform better than the group average and negative for those that perform worse. If every answer receives the same reward, their relative reward advantages are zero.ChallengeStandard GRPO training typically assigns the same number of rollouts to every question, despite differences in difficulty and potential training value. This can waste computation on consistently solved questions while giving difficult but learnable questions too few attempts to produce a correct answer and a useful training signal. The challenge is to allocate a fixed sampling budget more effectively to improve reasoning performance.Human paperCore idea — CoDaPO.Prioritize questions using both empirical difficulty (fraction incorrect) and token-probability confidence, favoring questions that are moderately difficult but that the student appears ready to learn.Mechanism.Generate an initial rollout group for every question, repeat the highest-valued questions to form a fresh second batch, and use the same values to weight the training loss; no teacher is required.Distinction / limit.CoDaPO changes bothwhich questions are resampledandhow strongly they train the model; unlike the methods below, it couples allocation with loss weighting. A fair efficiency comparison must charge both rollout stages.IdeaScientist-BaseCore idea.Treat questions as arms in a UCB-style allocator: empirical success estimates current solvability, while an uncertainty bonus gives under-sampled questions an optimistic chance to receive more compute.Mechanism.Generate an initial group per question, repeatedly assign the remaining budget to the highest upper-confidence score, and train GRPO on the resulting variable-size whole-question groups; no teacher model is required.Distinction / limit.The exploration bonus can sample apparently very hard questions that variance-based rules discard, then decays if evidence shows they are unreachable. The rule is standard bandit machinery and assumes question difficulty is sufficiently stationary as the policy changes.IdeaScientist-TrainedCore idea.Estimate distance to the learning frontier from partial reasoning: prefix-conditional success separates questions that are close to a solution from those that are merely hard and currently unreachable.Mechanism.From a small initial group, take the most confident trajectory and an intermediate prefix, sample short continuations, and classify the question as saturated, learnable, or beyond reach; use that estimate to allocate fresh full rollouts.Distinction / limit.Unlike baselines driven by final-answer statistics or gradients, Trained asks whether a promising partial solution can be completed. Its diagnostic continuations are discarded rather than trained on, and estimates from a small sample and fixed regime thresholds may be noisy.Baselines: outcome-statistic schedulersAutoResearchClawUses recent success-rate variation as a learnability signal, while reserving some easy questions for calibration and behavioral stability.Dr. ClawCombines rollout accuracy, reward variance, and entropy to favor hard-but-learnable questions; it also relaxes KL regularization for hard questions and strengthens it for easy ones.ResearchAgentAllocates from early reward variance and verifier feedback, then normalizes advantages within difficulty bands to stabilize unequal group sizes.CycleResearcherAllocates from initial confidence and reward variance, and rescales advantages by the allocation ratio to balance gradients across variable-size groups.AI-ResearcherUses moving averages of success and reward variance to estimate learning potential, with modified advantage normalization for unequal rollout counts.Baselines: learned or gradient-based value signalsSciAgentsTrains a lightweight difficulty predictor to drive a capped rollout scheduler, retaining within-question advantage normalization for stability.DeepScientistValues questions by the magnitude and cross-trajectory agreement of their policy gradients, then assigns both rollouts and loss weight accordingly; this requires per-question gradient statistics.The human paper jointly resamples and loss-weights questions using confidence and difficulty. Base adds a standard UCB exploration bonus to whole-question allocation, while most baselines schedule from success, variance, entropy, or a learned difficulty signal; DeepScientist is the exception, using gradient magnitude and agreement. Trained alone measures whether a promising partial trajectory can be completed, which directly targets the hard-but-learnable versus unreachable distinction.

I.3Capability Grounded Subagent Abstention

Table 15:Case study on capability grounded subagent abstention. Every method receives a user request and information about what the subagent can do. The methods differ in how they decide whether to answer or abstain. The same example applies throughout. A user asks for customer groups by age and gender, but the subagent cannot access those fields.ComponentDescriptionProblem definitionA coordinator sends a user requestqqto a specialized sub-agent. The sub-agent receives the request, its task context, and information about its available tools, data, and permissions. It must return either a valid task resultyy, such as a customer segmentation query, or a structured abstentionaa. The abstention states the failure category and the specific reason the task cannot be completed, such as a missing age field, a privacy restriction, or an unclear request. The goal is to avoid unsupported answers while continuing to complete requests the sub-agent can handle.ChallengeSpecialized sub-agents sometimes give plausible answers to requests they cannot complete because the request is unclear or requires unavailable tools, data, or permissions. The coordinator may pass those answers to users without knowing they are unsupported. The challenge is to help sub-agents recognize these limits and report them clearly enough for the coordinator to clarify, reroute, or stop the request.Human paperCore idea — EARS.Teach abstention as part of the sub-agent’s ordinary response policy, covering missing capabilities, ambiguous requests, and requests sent to the wrong specialist.Mechanism.Fine-tune on production requests paired with either valid results or typed abstentions; humans label a seed set and calibrated LLM judges scale the annotations. At inference, the sub-agent decides directly whether to answer.Distinction / limit.Unlike registry gates, EARS can learn limits that are difficult to formalize and needs no separate checker on every request. Its decisions are not mechanically grounded in a live inventory and depend on training-data coverage.IdeaScientist-BaseCore idea.Verify thereasonfor abstention, not only the answer/abstain decision, so the coordinator receives a minimal, causally relevant explanation of the failure.Mechanism.Train on examples of checker feedback and corrected explanations; at inference, an LLM translates its proposed reason into inventory conditions and revises it until a logic checker confirms both necessity and relevance.Distinction / limit.Baseline gates mostly stop at detecting infeasibility; Base adds iterative proof-checking of the stated reason. Its reliability still rests on an accurate inventory and faithful translation from language to logic.IdeaScientist-TrainedCore idea.Prevent unsupported generation by decomposing the request into required tools, fields, permissions, and operations, then checking every requirement before the sub-agent runs.Mechanism.Compare the LLM-extracted requirements with a live structured inventory; execute only if all checks pass, otherwise return a typed abstention naming the missing or prohibited capability. The basic version needs no training.Distinction / limit.This is an explicit, auditable gate rather than EARS’s learned decision, but it is methodologically close to five baselines below. Its main value is the concrete requirement decomposition; errors in parsing or stale inventories remain failure points.Baselines: explicit pre-execution capability gatesDr. ClawChecks the request against a declared capability list and returns a typed failure reason that the coordinator can use to reroute, clarify, or stop.ResearchAgentMatches request requirements to a registry of tools and constraints before execution, returning a structured failure category and explanation on mismatch.CycleResearcherPlaces a thresholded feasibility assessor over a capability schema before the task agent, choosing execution or structured abstention.SciAgentsVerifies required tools, data, and permissions before execution and reports a categorized reason for coordinator rerouting or escalation.DeepScientistChecks each request against explicit tools, constraints, and scope boundaries before generation, producing typed abstentions for downstream recovery.Baselines: confidence-aware and hybrid gatesAutoResearchClawMaps a calibrated pre-execution feasibility score to risk, then also audits outputs so the coordinator can decompose, reroute, or escalate failures.AI-ResearcherCombines executable hard constraints with semantic capability matching and an uncertainty gate, returning recovery guidance when the check fails.The human paper fine-tunes abstention as part of the sub-agent’s response policy, allowing it to learn ambiguous or misrouted cases but tying reliability to data coverage. Base instead checks and revises the stated reason for failure, emphasizing a precise explanation. Trained decomposes a request into required capabilities and checks them against a live inventory before generation. Five baselines use essentially the same registry-gate pattern; AutoResearchClaw adds output auditing, and AI-Researcher combines executable constraints with semantic matching and uncertainty.

I.4Adaptive Multimodal Speculative Decoding

Table 16:Case study on adaptive drafting for multimodal speculative decoding. Every method receives an image, a conversation, and the tokens already accepted by a target model. The methods differ in how they adapt drafting when a conversation shifts between visually grounded and text only requests.ComponentDescriptionProblem definitionA large vision and language model is thetargetwhose answer we want to produce; a cheaper model is thedrafterthat proposes several tokens at a time. Given the image, the text prompt, and the tokens already accepted, a method chooses how to draft the next block of tokens. The target checks that block in one verification pass. Accepted tokens become part of the answer; when a token is rejected, the verification procedure supplies a corrected token. The objective is to reduce total generation time while preserving the target model’s output distribution through the standard acceptance and correction procedure. The method produces draft probabilities as an intermediate result and an accepted token sequence as its final result(Leviathan et al., 2023).ChallengeA draft model that predicts well for image questions may predict poorly when the conversation shifts to a question that needs only text. Poor predictions make the large model reject more draft tokens, while extra drafting work can erase any speed gain. The challenge is to adapt drafting to the current request so that thetime to produce the final answerimproves across different kinds of conversation.Human paperCore idea — TABED.Keep visual and text drafting active as an ensemble, and continuously shift their mixture weights toward whichever method has recently predicted the target model better.Mechanism.Combine the drafters’ next-token distributions; after each target verification, compare their saved predictions with the target distribution and update the weights for the next block. No new router training is required.Distinction / limit.TABED uniquely learns online frompast target verificationwhile retaining multiple drafters, rather than making a hard routing decision. This robustness costs extra drafting work even with shared parameters and batching.IdeaScientist-BaseCore idea.Use the divergence between a drafter’s text-only and vision-conditioned next-token distributions as a step-level estimate of how much information the image contributes.Mechanism.Compute both draft distributions, turn their KL divergence into a gate on visual-feature injection, and close the visual path when it is redundant while retaining standard target verification. No new training is required.Distinction / limit.Base proposes continuous visual gating rather than selecting a fixed specialist, making it finer-grained than the routing baselines. Running both draft paths adds cost, and the written pseudocode does not clearly connect the gate to the final draft-token distribution.IdeaScientist-TrainedCore idea.Route each conversational turn using the target model’s own prefill-stage language-to-visual attention mass as a zero-shot estimate of visual relevance.Mechanism.Normalize the already-computed attention signal, choose either a vision-conditioned or text-only specialist for the turn, and pass that drafter’s blocks through ordinary target verification; the router needs no training.Distinction / limit.Unlike heuristic modality classifiers, Trained reads the model whose distribution the drafter must approximate; unlike TABED, it commits to one drafter rather than maintaining an ensemble. Attention is still only a proxy, and target prefill is not literally zero-latency overhead.Baselines: hard routing among specialized draftersAutoResearchClawClassifies request modality, selects a specialized drafter, prunes visual tokens, and falls back to ordinary decoding when draft confidence is low.Dr. ClawUses a lightweight, training-free context classifier to switch between visual and text-reasoning drafters as the request type changes.SciAgentsUses attention entropy and token diversity to label each phase visual or textual, then sends the chosen specialist’s block to the target for verification.DeepScientistDetects visual, mixed, and textual phases from token proportions and positions, switching specialist drafters only at block boundaries.Baselines: multi-signal routing and adaptive controlResearchAgentUses estimated modality balance and reasoning complexity to select or blend drafters, with visual-token pruning and a syntax-aware text drafter.CycleResearcherRoutes from attention entropy and key–value-cache gradients, while adapting draft length to maintain the target acceptance rate.Baseline: online soft ensembleAI-ResearcherMixes visual, textual, and hybrid draft distributions, adjusts the visual-token budget, and updates mixture weights from recent verification outcomes without gradient training.The human paper keeps several drafting methods active and updates their mixture weights from past target verification. Base instead computes text-only and vision-conditioned draft distributions at each step and uses their KL divergence to gate visual-feature injection; its pseudocode does not clearly connect that gate to the final draft-token distribution. Trained makes a hard per-turn choice from the target model’s own prefill-stage visual attention, avoiding a learned or heuristic context classifier and the human method’s continuing ensemble cost. The baselines mostly route from modality or phase indicators; ResearchAgent and CycleResearcher add richer control signals, while AI-Researcher is the closest online ensemble.