Many people are now talking about Research Agents. The most intuitive picture is: give it a stack of papers and let it come up with a new idea on its own. But this paper reminded me of a crucial issue: the quality of an idea often depends on whether it has been placed back into the real literature context. The paper proposes a system called ResearchStudio-Idea. Its goal is to organize common idea-generation methods in ML research into reusable skills. It mainly consists of three parts: Paper-Search is responsible for finding relevant papers, Scoop-Check is responsible for checking whether a novelty claim conflicts with prior art, and IdeaSpark is responsible for actually generating research ideas. The core is IdeaSpark. The authors analyzed 1,947 papers from ICLR, ICML, and NeurIPS between 2021 and 2025, extracted 31 ideation sub-patterns from them, and finally organized them into 15 reusable research ideation patterns. These patterns are more like compressed research experience: which scenarios they are suitable for, what bottlenecks they typically solve, how to differentiate from existing methods, and which past papers have used similar ideas. The experimental results are also straightforward. On 100 ICLR 2026 Oral seeds, IdeaSpark’s idea quality scored 3.87/4, and it ranked first on 88 seeds. In contrast, generating ideas directly with a bare model performed much worse. There is also a very interesting phenomenon in the paper: GPT-5.5 bare has the highest novelty score but the lowest quality. This illustrates a problem: an idea that looks “new” is not necessarily a good idea. Often it’s simply because it is too broad and too vague, so it is less likely to directly collide with existing work. The most interesting part of this paper is that it shifts the focus of Research Agents from “generating more ideas” to “turning idea generation into a set of reusable research skills.” In the future, truly useful Research Agents may increasingly rely on this kind of skill library. They will need to read more papers, and also consider evidence, bottlenecks, existing methods, and risks within the same framework. The next step for AI research may move from “being better at fantasizing” to “being better at asking questions along the real research trajectory.”

arxiv: https://arxiv.org/abs/2607.04439

An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Source: https://arxiv.org/html/2607.04439 Qihao Zhao¹,Yangyu Huang^2‡,Yalun Dai^1,2,Lingao Xiao^2,3,Jianjun Gao¹,Xin Zhang²,Wenshan Wu² Scarlett Li²,Yang He^3,4,Yan Lu²,Yap Kim Hui^1‡
1Nanyang Technological University²Microsoft Research³National University of Singapore ⁴CFAR, A*STAR

ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Qihao Zhao¹,Yangyu Huang^2‡,Yalun Dai^1,2,Lingao Xiao^2,3,Jianjun Gao¹,Xin Zhang²,Wenshan Wu² Scarlett Li²,Yang He^3,4,Yan Lu²,Yap Kim Hui^1‡
1Nanyang Technological University²Microsoft Research³National University of Singapore ⁴CFAR, A*STAR

Large language models have made research ideation increasingly accessible, yet effective idea development requires more than generating candidate directions. Researchers must ground a problem in current literature, identify meaningful bottlenecks, differentiate from existing solutions, and evaluate risks before committing to implementation. We present ResearchStudio-Idea as a reusable skill suite for this first mile of research ideation. The suite includes Paper-Search, a standalone multi-source literature search skill; Scoop-Check, a standalone prior-art collision checker for novelty claims; andIdeaSpark, the end-to-end skill that composes evidence grounding, pattern-guided generation, collision retrieval, audit, and idea-card rendering into one workflow. IdeaSpark is constructed from a corpus of 1,947 machine learning conference papers collected from ICLR, ICML, and NeurIPS between 2021 and 2025, including Oral papers, a separately tracked high-citation subset, and rejected submissions. Analysis of these outcomes reveals 31 recurring ideation sub-patterns, consolidated into 15 reusable ideation patterns. Each pattern is operationalized as a structured card containing research contexts, bottleneck types, differentiation strategies, supporting precedents, and common failure modes. Given a research problem and an evidence bundle, IdeaSpark evaluates evidence readiness, reconstructs the surrounding research context, identifies unresolved bottlenecks, selects relevant patterns, instantiates one candidate direction, retrieves potentially conflicting prior work, and performs outcome-informed auditing. This workflow transforms reusable ideation patterns into traceable research proposals. Blind automated-judge evaluations show that IdeaSpark consistently produces stronger research proposals than no-skill and generic-skill baselines while maintaining competitive novelty. These results suggest that large-scale conference outcomes contain reusable signals about how impactful research directions are formulated, differentiated, and evaluated, and that such signals can be operationalized as practical skills for evidence-grounded research ideation.
Project:https://aka.ms/ResearchStudio

Refer to caption Refer to caption Figure 1:IdeaSpark improves idea quality while maintaining competitive novelty in blind automated-judge evaluations. Left: quality–novelty trade-off over 100 ICLR-2026-Oral seeds, each judged in 3 blind rounds against three baselines: Opus-4.8 bare, Opus-4.8 self-generated, and GPT-5.5 bare. IdeaSpark occupies the high-quality, competitively novel region, whereas GPT-5.5 illustrates a novel-but-empty failure mode: high apparent novelty but substantially lower quality. Right: mean idea-quality across 21 ICLR primary-area domains; IdeaSpark is highest in every domain, suggesting the gain is broad rather than domain-specific.^††footnotetext:^‡Corresponding author:[email protected] (mailto:[email protected]),[email protected] (mailto:[email protected])

Contents

1. 1Introduction (https://arxiv.org/html/2607.04439#S1)1. 1.1Problem and scope (https://arxiv.org/html/2607.04439#S1.SS1) 2. 1.2Motivation (https://arxiv.org/html/2607.04439#S1.SS2) 3. 1.3Approach overview (https://arxiv.org/html/2607.04439#S1.SS3) 4. 1.4Main empirical findings (https://arxiv.org/html/2607.04439#S1.SS4) 5. 1.5Contributions (https://arxiv.org/html/2607.04439#S1.SS5) 6. 1.6Report organization (https://arxiv.org/html/2607.04439#S1.SS6) 2. 2Related Work (https://arxiv.org/html/2607.04439#S2)1. 2.1End-to-end “AI scientist” systems (https://arxiv.org/html/2607.04439#S2.SS1) 2. 2.2Multi-agent and search-based ideation (https://arxiv.org/html/2607.04439#S2.SS2) 3. 2.3Pattern induction from conference outcomes (https://arxiv.org/html/2607.04439#S2.SS3) 4. 2.4Novelty, evaluation, and benchmarks (https://arxiv.org/html/2607.04439#S2.SS4) 5. 2.5Surveys and skill-engineering practice (https://arxiv.org/html/2607.04439#S2.SS5) 6. 2.6How IdeaSpark differs from current idea-generation methods (https://arxiv.org/html/2607.04439#S2.SS6) 3. 3Dataset Construction (https://arxiv.org/html/2607.04439#S3)1. 3.1Scope and labeling (https://arxiv.org/html/2607.04439#S3.SS1) 2. 3.2Metadata coverage (https://arxiv.org/html/2607.04439#S3.SS2) 3. 3.3Convention for in-paper paper references (https://arxiv.org/html/2607.04439#S3.SS3) 4. 4Two-Stage Innovation-Signature Extraction (https://arxiv.org/html/2607.04439#S4)1. 4.1Stage 1: Eight base fields (https://arxiv.org/html/2607.04439#S4.SS1) 2. 4.2Stage 2: Four domain-agnostic rewrites (https://arxiv.org/html/2607.04439#S4.SS2) 5. 5Unsupervised Pattern Discovery (https://arxiv.org/html/2607.04439#S5)1. 5.1Embedding (https://arxiv.org/html/2607.04439#S5.SS1) 2. 5.2Clustering (https://arxiv.org/html/2607.04439#S5.SS2) 3. 5.3Cluster inventory (https://arxiv.org/html/2607.04439#S5.SS3) 4. 5.4Per-cluster acceptance composition (https://arxiv.org/html/2607.04439#S5.SS4) 6. 6Ideation-Pattern Induction (https://arxiv.org/html/2607.04439#S6)1. 6.1From definition to operational card (https://arxiv.org/html/2607.04439#S6.SS1) 2. 6.2Coverage and granularity (https://arxiv.org/html/2607.04439#S6.SS2) 3. 6.3Multi-strategy assignments (https://arxiv.org/html/2607.04439#S6.SS3) 4. 6.4Ideation-pattern adjacency in embedding space (https://arxiv.org/html/2607.04439#S6.SS4) 5. 6.5Paper-level multi-label ideation pattern assignment (https://arxiv.org/html/2607.04439#S6.SS5) 7. 7Acceptance and Impact Analysis (https://arxiv.org/html/2607.04439#S7) 8. 8Domain Distribution and Ideation-Pattern Breadth (https://arxiv.org/html/2607.04439#S8)1. 8.1Domain induction (https://arxiv.org/html/2607.04439#S8.SS1) 2. 8.2Domain×\timesideation pattern heatmap (https://arxiv.org/html/2607.04439#S8.SS2) 3. 8.3Ideation-pattern breadth: how domain-agnostic is each pattern? (https://arxiv.org/html/2607.04439#S8.SS3) 9. 9Temporal and Conference Structure (https://arxiv.org/html/2607.04439#S9)1. 9.1Temporal trends (https://arxiv.org/html/2607.04439#S9.SS1) 2. 9.2Conference preferences (https://arxiv.org/html/2607.04439#S9.SS2) 10. 10Do Rejected Papers Inhabit a Different Strategy Space? (https://arxiv.org/html/2607.04439#S10)1. 10.1Reject-only clustering and mapping (https://arxiv.org/html/2607.04439#S10.SS1) 2. 10.2Interpretation (https://arxiv.org/html/2607.04439#S10.SS2) 11. 11Ablation: Embedding Model and Abstraction Stage (https://arxiv.org/html/2607.04439#S11)1. 11.1Embedding model: OpenAI vs SPECTER2 (https://arxiv.org/html/2607.04439#S11.SS1) 2. 11.2Abstraction stage: silhouette is not enough (https://arxiv.org/html/2607.04439#S11.SS2) 12. 12Discussion (https://arxiv.org/html/2607.04439#S12)1. 12.1Empirical takeaways (https://arxiv.org/html/2607.04439#S12.SS1) 13. 13IdeaSpark Design (https://arxiv.org/html/2607.04439#S13)1. 13.1Positioning and two-tier architecture (https://arxiv.org/html/2607.04439#S13.SS1) 2. 13.2Design principles (https://arxiv.org/html/2607.04439#S13.SS2) 3. 13.3Workflow and phase contracts (https://arxiv.org/html/2607.04439#S13.SS3)1. 13.3.1Phase 0: Literature grounding (https://arxiv.org/html/2607.04439#S13.SS3.SSS1) 2. 13.3.2Phase 1: Bottleneck identification (https://arxiv.org/html/2607.04439#S13.SS3.SSS2) 3. 13.3.3Phase 2: Pattern-guided ideation (https://arxiv.org/html/2607.04439#S13.SS3.SSS3) 4. 13.3.4Phase 3: Quality gauntlet (https://arxiv.org/html/2607.04439#S13.SS3.SSS4) 5. 13.3.5Phase 4: Expansion, implementability audit, rendering, and validation (https://arxiv.org/html/2607.04439#S13.SS3.SSS5) 4. 13.4Output surface (https://arxiv.org/html/2607.04439#S13.SS4) 5. 13.5Faithfulness: resisting hallucination (https://arxiv.org/html/2607.04439#S13.SS5) 14. 14Evaluation: Generated-Idea Quality and Novelty (https://arxiv.org/html/2607.04439#S14)1. 14.1Systems compared (https://arxiv.org/html/2607.04439#S14.SS1) 2. 14.2Problem seeds: 100 method-agnostic directions from ICLR 2026 (https://arxiv.org/html/2607.04439#S14.SS2) 3. 14.3Output normalization (controlling confounds) (https://arxiv.org/html/2607.04439#S14.SS3) 4. 14.4Two automated judges (https://arxiv.org/html/2607.04439#S14.SS4) 5. 14.5Protocol and metrics (https://arxiv.org/html/2607.04439#S14.SS5) 6. 14.6Results (https://arxiv.org/html/2607.04439#S14.SS6) 7. 14.7Analysis (https://arxiv.org/html/2607.04439#S14.SS7) 15. 15Limitations (https://arxiv.org/html/2607.04439#S15) 16. 16Artifact Cards (https://arxiv.org/html/2607.04439#S16)1. 16.1Data card (https://arxiv.org/html/2607.04439#S16.SS1) 2. 16.2Model and backend card (https://arxiv.org/html/2607.04439#S16.SS2) 3. 16.3Skill card (https://arxiv.org/html/2607.04439#S16.SS3) 17. 17Responsible Use (https://arxiv.org/html/2607.04439#S17) 18. 18Conclusion (https://arxiv.org/html/2607.04439#S18) 19. Appendix (https://arxiv.org/html/2607.04439#Ax1) 20. AEnd-to-end Generated Idea Card (https://arxiv.org/html/2607.04439#A1) 21. BKill-switch and Falsification