@Xudong07452910: 现在很多人聊 Research Agent，最容易想象的画面是：给它一堆论文，让它自己想一个新 idea。但这篇论文提醒我一个很关键的问题： idea 的质量，很多时候取决于它有没有被放回真实的文献脉络里。论文提出了一个系统，叫 R…

X AI KOLs Timeline 2026/07/11 03:32 论文

research-agent idea-generation ml-patterns evidence-grounded skill-suite conference-papers

摘要

本文介绍 ResearchStudio-Idea，一个基于 ML 会议论文分析的可复用研究构思技能套件，包含 Paper-Search、Scoop-Check 和 IdeaSpark，通过从 1,947 篇论文中提取的 15 个模式来生成有证据支撑的研究想法。

现在很多人聊 Research Agent，最容易想象的画面是：给它一堆论文，让它自己想一个新 idea。但这篇论文提醒我一个很关键的问题： idea 的质量，很多时候取决于它有没有被放回真实的文献脉络里。论文提出了一个系统，叫 ResearchStudio-Idea。它的目标是把 ML 研究里常见的 idea 生成方式，整理成可以复用的 skill。它主要包含三个部分：Paper-Search 负责找相关论文，Scoop-Check 负责检查 novelty claim 有没有撞车，IdeaSpark 负责真正生成研究想法。最核心的是 IdeaSpark。作者分析了 2021–2025 年 ICLR、ICML、NeurIPS 的 1,947 篇论文，从里面抽出 31 个 ideation sub-pattern，最后整理成 15 个可复用的 research ideation patterns。这些 pattern 更像是研究经验的压缩：什么场景适合用，通常解决什么 bottleneck，和已有方法怎么拉开差异，过去哪些论文用过类似思路。实验结果也比较直接。在 100 个 ICLR 2026 Oral seeds 上，IdeaSpark 的 idea quality 得到 3.87/4，并且在 88 个 seed 上排名第一。相比之下，裸模型直接生成 idea 的效果明显弱很多。论文里还有一个很有意思的现象： GPT-5.5 bare 的 novelty 分数最高，但 quality 最低。这说明一个问题：看起来「新」的 idea，不一定是好的 idea。很多时候只是因为它太泛、太空，所以不容易和已有工作直接撞上。这篇最有意思的地方在于，它把 Research Agent 的重点从「多生成几个想法」，推进到了「把想法生成变成一套可复用的研究技能」。未来真正有用的 Research Agent，可能会越来越依赖这种 skill library。它需要读更多论文，也需要把证据、瓶颈、已有方法和风险放在同一个框架里考虑。 AI 科研的下一步，可能会从「更会幻想」走向「更会沿着真实研究脉络提出问题」。 arxiv: https://arxiv.org/abs/2607.04439

查看原文

查看缓存全文

缓存时间: 2026/07/11 09:21

现在很多人聊 Research Agent，最容易想象的画面是：

给它一堆论文，让它自己想一个新 idea。

但这篇论文提醒我一个很关键的问题：

idea 的质量，很多时候取决于它有没有被放回真实的文献脉络里。

论文提出了一个系统，叫 ResearchStudio-Idea。它的目标是把 ML 研究里常见的 idea 生成方式，整理成可以复用的 skill。

它主要包含三个部分：Paper-Search 负责找相关论文，Scoop-Check 负责检查 novelty claim 有没有撞车，IdeaSpark 负责真正生成研究想法。

最核心的是 IdeaSpark。

作者分析了 2021–2025 年 ICLR、ICML、NeurIPS 的 1,947 篇论文，从里面抽出 31 个 ideation sub-pattern，最后整理成 15 个可复用的 research ideation patterns。

这些 pattern 更像是研究经验的压缩：什么场景适合用，通常解决什么 bottleneck，和已有方法怎么拉开差异，过去哪些论文用过类似思路。

实验结果也比较直接。

在 100 个 ICLR 2026 Oral seeds 上，IdeaSpark 的 idea quality 得到 3.87/4，并且在 88 个 seed 上排名第一。相比之下，裸模型直接生成 idea 的效果明显弱很多。

论文里还有一个很有意思的现象：

GPT-5.5 bare 的 novelty 分数最高，但 quality 最低。

这说明一个问题：看起来「新」的 idea，不一定是好的 idea。很多时候只是因为它太泛、太空，所以不容易和已有工作直接撞上。

这篇最有意思的地方在于，它把 Research Agent 的重点从「多生成几个想法」，推进到了「把想法生成变成一套可复用的研究技能」。

未来真正有用的 Research Agent，可能会越来越依赖这种 skill library。它需要读更多论文，也需要把证据、瓶颈、已有方法和风险放在同一个框架里考虑。

AI 科研的下一步，可能会从「更会幻想」走向「更会沿着真实研究脉络提出问题」。

arxiv: https://arxiv.org/abs/2607.04439

An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Source: https://arxiv.org/html/2607.04439 Qihao Zhao1,Yangyu Huang2‡,Yalun Dai1,2,Lingao Xiao2,3,Jianjun Gao1,Xin Zhang2,Wenshan Wu2 Scarlett Li2,Yang He3,4,Yan Lu2,Yap Kim Hui1‡

(1Nanyang Technological University2Microsoft Research3National University of Singapore 4CFAR, A*STAR)

ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Qihao Zhao1,Yangyu Huang2‡,Yalun Dai1,2,Lingao Xiao2,3,Jianjun Gao1,Xin Zhang2,Wenshan Wu2

Scarlett Li2,Yang He3,4,Yan Lu2,Yap Kim Hui1‡

1Nanyang Technological University2Microsoft Research3National University of Singapore 4CFAR, A*STAR

Large language models have made research ideation increasingly accessible, yet effective idea development requires more than generating candidate directions. Researchers must ground a problem in current literature, identify meaningful bottlenecks, differentiate from existing solutions, and evaluate risks before committing to implementation. We present ResearchStudio-Idea as a reusable skill suite for this first mile of research ideation. The suite includes Paper-Search, a standalone multi-source literature search skill; Scoop-Check, a standalone prior-art collision checker for novelty claims; andIdeaSpark, the end-to-end skill that composes evidence grounding, pattern-guided generation, collision retrieval, audit, and idea-card rendering into one workflow. IdeaSpark is constructed from a corpus of 1,947 machine learning conference papers collected from ICLR, ICML, and NeurIPS between 2021 and 2025, including Oral papers, a separately tracked high-citation subset, and rejected submissions. Analysis of these outcomes reveals 31 recurring ideation sub-patterns, consolidated into 15 reusable ideation patterns. Each pattern is operationalized as a structured card containing research contexts, bottleneck types, differentiation strategies, supporting precedents, and common failure modes. Given a research problem and an evidence bundle, IdeaSpark evaluates evidence readiness, reconstructs the surrounding research context, identifies unresolved bottlenecks, selects relevant patterns, instantiates one candidate direction, retrieves potentially conflicting prior work, and performs outcome-informed auditing. This workflow transforms reusable ideation patterns into traceable research proposals. Blind automated-judge evaluations show that IdeaSpark consistently produces stronger research proposals than no-skill and generic-skill baselines while maintaining competitive novelty. These results suggest that large-scale conference outcomes contain reusable signals about how impactful research directions are formulated, differentiated, and evaluated, and that such signals can be operationalized as practical skills for evidence-grounded research ideation.Project:https://aka.ms/ResearchStudio [Uncaptioned image]

Refer to caption

Figure 1:IdeaSpark improves idea quality while maintaining competitive novelty in blind automated-judge evaluations. Left: quality–novelty trade-off over 100 ICLR-2026-Oral seeds, each judged in 3 blind rounds against three baselines: Opus-4.8 bare, Opus-4.8 self-generated, and GPT-5.5 bare. IdeaSpark occupies the high-quality, competitively novel region, whereas GPT-5.5 illustrates a novel-but-empty failure mode: high apparent novelty but substantially lower quality. Right: mean idea-quality across 21 ICLR primary-area domains; IdeaSpark is highest in every domain, suggesting the gain is broad rather than domain-specific.††footnotetext:‡Corresponding author:[email protected],[email protected]###### Contents

1Introduction

1.1Problem and scope

LLM-based research agents have made scientific ideation easier to scale. Recent systems can retrieve papers, propose hypotheses, coordinate specialist agents, plan experiments, write code, and draft reports[29,40,49,18,11,25]. Search-based methods expand the candidate space. Novelty tools make prior-art checking more explicit. This progress shifts the bottleneck. The hard question is no longer whether an LLM can produce a plausible proposal. It is whether early-stage research work can be organized into reusable skills that ground evidence, generate one defensible direction, and audit its distance from prior art before experiments begin.

ResearchStudio-Idea addresses this first-mile problem as a suite of three open skills.Paper-Searchprovides reusable literature grounding across arXiv, DBLP, OpenAlex, OpenReview, Semantic Scholar, and Crossref.Scoop-Checkperforms claim-level prior-art collision checking by decomposing a proposed novelty into problem framing, core mechanism, key insight, and application domain, then comparing those axes against retrieved prior work.IdeaSparkis the end-to-end skill: it composes literature grounding, bottleneck diagnosis, pattern-guided candidate construction, collision retrieval, audit, and idea-card packaging into one research-problem-to-idea-card workflow. Paper-Search and Scoop-Check are therefore not hidden implementation details; they are standalone skills that also supply reusable search and review functions inside the larger ideation loop.

The empirical center of this report is IdeaSpark because it is the point where ResearchStudio-Idea’s search, generation, and review functions must work together. We study it asoutcome-grounded skill inductionfor LLM research ideation. The object we induce is an inference-time skill, not an acceptance predictor, novelty score, or paper-writing agent. The skill consists of ideation pattern cards, workflow prompts, schemas, retrieval hooks, and validators that an LLM loads while generating a structured idea. We ground it in public ML conference outcomes because those outcomes contain traces of research practice. Oral papers show cases that program committees elevated. High-citation papers show what the community reused. Rejected submissions show how similar attempts fail. The goal is to turn those traces into operational guidance, while keeping acceptance and execution outside the claim.

This report asks two questions. First, can conference outcomes be mined into a compact but usable map of ideation patterns? Second, can that map be packaged as a model-agnostic skill suite whose workflow generates and audits one research idea? We answer the first question with a 1,947-paper analysis of ICLR, ICML, and NeurIPS outcomes. We answer the second by releasing ResearchStudio-Idea’s three skills and evaluating IdeaSpark-generated ideas with blind automated judges. The evaluation is scoped to the idea stage. It does not test implementation success, human peer review, or program-committee selection.

1.2Motivation

Existing systems address important pieces of research ideation, but they leave a missing skill layer between outcome data and generation. End-to-end systems automate long research workflows[29,40,49,61]. Multi-agent and search systems explore larger proposal spaces[60,17,53,38,4]. Retrieval-augmented methods ground hypotheses in adjacent literature[23,14,56,8]. Novelty tools, benchmarks, and pattern-induction methods check prior-art collision or name recurring research moves[55,59,7,43,41,26,13,22,44]. These components are complementary. What remains under-specified is how to turn them into interoperable skills, and how to turn conference outcomes into an executable ideation workflow: a reusable procedural object that helps an LLM organize a gap, instantiate a research strategy, distinguish it from neighboring work, and audit the proposal before implementation.

Research assistants are increasingly capable of retrieving literature, synthesizing evidence, generating candidate ideas, and iteratively refining proposals. The transition from retrieved evidence to actionable research opportunities, however, remains weakly structured. Researchers still need to decide which limitations constitute meaningful bottlenecks, which opportunities remain unresolved, and which directions are distinct enough from prior work to justify further investment.

Current evaluations of LLM-generated ideas make this gap concrete. A large blind study finds that LLM-generated NLP ideas can be judged more novel than expert ideas while being less feasible[46]. A follow-up execution study reports that the gap becomes sharper once ideas are carried out[45]. HindSight similarly finds that ideas rated as more novel by an LLM are less likely to match real future work[19]. These results do not imply that an idea-stage skill can solve execution. They do show that surface novelty, proposal fluency, and post-hoc novelty scoring are weak proxies for useful ideation.

A useful skill should instead expose how real papers connect a gap to a method. It should show how they assemble differentiation evidence, and how similar attempts fail before experiments begin. We call these recurring procedural objectsideation patterns. A pattern is not merely a label such as “decompose the problem” or “add supervision.” It must state what kind of gap it applies to, how the method is constructed, what distinguishes it from nearby prior work, what evidence makes it credible, and where similar attempts fail.

The missing middle is therefore an outcome-grounded skill layer that connects research evidence, identified bottlenecks, reusable ideation patterns, and candidate research directions. Retrieved papers expose observations, limitations, reviewer concerns, and unresolved questions. Generated proposals require strategic moves that connect those observations to plausible directions. The missing layer is the mechanism that transforms evidence into opportunities and opportunities into auditable research proposals.

We need to know how these patterns recur across ML papers, how they compose inside contributions, and how they differ across accepted, high-citation, and rejected work. We also need to know whether they can be operationalized as reusable inference-time skills. ResearchStudio-Idea addresses this missing middle as a suite, and IdeaSpark tests the strongest version of the claim: whether search, pattern-guided generation, collision checking, and audit can be composed into one faithful ideation workflow rather than another generic brainstorming prompt or novelty scorer. Section2.6gives the detailed comparison to current idea-generation methods.

1.3Approach overview

ResearchStudio-Idea packages early-stage ideation into three reusable skills. Paper-Search supplies the literature-grounding primitive. Scoop-Check supplies the claim-level prior-art review primitive. IdeaSpark induces ideation patterns from conference outcomes and packages them as the structured end-to-end ideation skill. Given a research problem and an evidence bundle, IdeaSpark performs evidence assessment, bottleneck diagnosis, innovation-pattern retrieval, candidate construction, differentiation analysis, collision retrieval, and outcome-informed auditing. The output is not only a candidate research direction, but also the supporting rationale, historical precedents, and diagnostic evidence needed to evaluate it. Figure2summarizes this data-to-skill flow from corpus construction to runtime use.

Refer to caption Figure 2:IdeaSpark data-to-skill workflow. The upper band constructs reusable ideation assets from the 1,947-paper ICLR / ICML / NeurIPS corpus: papers are outcome-labeled, normalized into strategy signatures, mined into 31 sub-patterns, and induced into 15 operational pattern cards. The lower band shows how the skill uses those cards at inference time: evidence grounding and full-text retrieval feed a staged reasoning loop for bottleneck diagnosis, pattern-guided candidate generation, and collision/audit verdicts, followed by expansion, validation, and idea-card deliverables.The empirical pipeline has four stages. First, we collect papers from ICLR, ICML, and NeurIPS using public OpenReview metadata and Semantic Scholar citation information[34,42]. Each paper is labeled as Oral, High-Cited, Reject, or a combination when labels overlap. Second, we extract strategy-level innovation fields and rewrite them into domain-agnostic descriptions. This separates the methodological operation from topic vocabulary. As a result, clustering groups papers by research strategy rather than by application area. Third, we embed and cluster these strategy signatures, using a workflow inspired by scientific document representations and density-based topic discovery[6,47,1,12,31,30,33]. Fourth, we induce higher-level ideation patterns from the fine-grained clusters and convert each pattern into an operational card.

The resulting cards are not taxonomy labels. Each card records the research situation the pattern addresses, the structure distilled from accepted examples, the differentiating mechanism relative to nearby prior work, the evidence expectations that make the claim inspectable, and the failure modes abstracted from rejected submissions. This contrast between success and failure offers insights beyond accepted-only inductions. In the runtime skill, the cards serve two roles. They guide structured idea generation, and they provide an audit reference for checking whether a candidate preserves the intended structure, avoids known failure modes, and names plausible prior-art threats. Generation and audit share the same corpus-derived object, while the paper’s claim remains at the idea stage.

The IdeaSpark layer follows a model-agnostic, two-tier design. The runtime tier includes lean skill specification, phase prompts, schemas, retrieval hooks, and deterministic validators. The evidence tier consists of 15 ideation-pattern cards, 31 sub-pattern cards, a domain-by-pattern matrix, saturation records, and a corpus-derived failure-mode inventory. This separation follows recent skill-authoring guidance: keep the runtime lightweight, use progressive disclosure for detailed evidence, and validate phase boundaries with deterministic checks where possible[3,39,54,58]. Consequently, the skill is faithful by construction rather than by instruction. Every significant claim either traces to a retrieved record or is explicitly marked as model-supplied. Retrieval gates and deterministic validators enforce this contract rather than relying on prompt compliance (§13.5).

1.4Main empirical findings

This report supports five findings:

First, the induced pattern space is compact but nontrivial.From 31 fine-grained ideation sub-patterns, the pipeline induces 15 higher-level ideation patterns. The number 15 is not a canonical ontology of ML innovation. It is the operational granularity produced by the current extraction, abstraction, embedding, clustering, and induction pipeline. Because it comes from a single Opus 4.7 induction call, an inter-prompt, inter-seed, and inter-model stability study remains future work; we expect a comparable run to land at 12–18 categories with substantial overlap.

Second, rejected and accepted papers often share the same high-level pattern space.Reject-only re-clustering maps every rejected-paper cluster back onto the existing 15-pattern vocabulary, with no out-of-taxonomy bucket. This suggests that rejected submissions are most useful as contrastive evidence about weak instantiations, failure modes, and boundary cases, not as a separate negative strategy class.

Third, multi-pattern composition is the norm.A separate paper-level multi-label pass shows that papers commonly combine several ideation patterns rather than executing a single isolated move. In this corpus,k=2k=2is the modal composition size across Oral, HC, and Reject papers, with a33.6%33.6\%tail atk≥3k\geq 3. This motivates the skill design choice to generate one to three pattern roles rather than selecting a single pattern as a recipe.

Fourth, ideation patterns are broadly domain-covering but domain-conditional in effect.The largest patterns appear across many induced research domains, which supports the use of domain-agnostic pattern cards. Acceptance and impact profiles, however, vary by domain-pattern cell. The skill therefore treats domain statistics as audit context, not as a deterministic generation prior.

Fifth, endpoint idea quality is supported by an automated-judge study, but not by human review.An automated-judge endpoint evaluation (Section14) indicates that IdeaSpark improves generated-idea quality over no-skill and generic-skill baselines. Acceptance-level claims still require a blind human study.

1.5Contributions

This report makes three contributions.

Outcome-grounded pattern map.We provide a 1,947-paper corpus analysis spanning Oral, high-citation, and rejected papers from major ML conferences. The analysis induces 15 ideation patterns, 31 sub-patterns, and 28 research domains. It then measures pattern-level acceptance, citation, domain, conference, temporal, and multi-pattern composition profiles.

Contrastive, failure-aware pattern cards.We turn the accepted-versus-rejected contrast into card content. Each card pairs success conditions distilled from accepted work with failure modes distilled from rejected work. Accepted-only induction cannot supply this object. The empirical warrant is the reject-only mapping of §10: rejected papers do not occupy a separate strategy space, so their value is as contrastive evidence about weak instantiations and boundary cases.

ResearchStudio-Idea skill suite and endpoint evaluation.We package the work as three reusable skills: Paper-Search for literature grounding, Scoop-Check for prior-art collision checking, and IdeaSpark for end-to-end idea generation and audit. We also report a blind automated evaluation of IdeaSpark-generated ideas along quality and novelty axes.

1.6Report organization

Section2positions ResearchStudio-Idea and IdeaSpark against recent work on LLM-based scientific discovery, ideation, novelty assessment, benchmarks, and pattern induction. Sections3–6describe the dataset, extraction procedure, clustering pipeline, and ideation-pattern induction. Sections7–9analyze acceptance profiles, domain structure, temporal trends, and conference-level variation. Section10validates the taxonomy against rejected papers. Section11reports embedding and abstraction ablations. Section12synthesizes the empirical takeaways. Section13specifies the IdeaSpark runtime and audit workflow within the ResearchStudio-Idea suite, and Section14reports the automated-judge evaluation of generated ideas. Section15consolidates limitations and threats to validity across the corpus, skill, and evaluation. Section16presents the released data, model, and skill-card artifacts.

2Related Work

The 2024–2026 literature on LLM research-idea generation has expanded rapidly. We group it into the families most relevant to IdeaSpark and survey each below. To keep the comparison coherent, we defer direct head-to-head differentiation to §2.6.

2.1End-to-end “AI scientist” systems

End-to-end systems automate the full research lifecycle. AI Scientist[29]drafts, implements, evaluates, and writes up research ideas in a single loop, although its output was often judged incremental. AI-Researcher[49]extends this direction with structured architectures and decoupled training pipelines for autonomous innovation. Agent Laboratory[40]starts from a human-supplied research idea and produces a research report and code, with variable human-in-the-loop control. Idea2Plan[18]bridges idea generation and research planning through ReAct-style scaffolding with arXiv tools. The Sakana AI Scientist v2[57]adds evolutionary-search refinement, while Google AI co-scientist[11]uses a Gemini-based multi-agent architecture for hypothesis and proposal generation. Auto Research[25]spans literature review, ideation, innovation pattern, experimentation, paper writing, and rebuttal in a multi-agent framework.

A parallel critical literature documents why full-lifecycle autonomy remains fragile. Trehan et al.[51]distill six recurring failure modes from four autonomous ML-paper attempts: training-default bias, implementation drift under execution pressure, long-horizon context degradation, premature success declarations, thin domain knowledge, and weak experimental taste. Bisht et al.[5]argue that current agentic scientists are not yet built for autonomy. They emphasize problem-selection bias, missing tacit laboratory knowledge, output homogenization from preference optimization, and the absence of experimental feedback loops. This line of critique motivates IdeaSpark’s narrower scope: improve the idea before downstream execution begins.

2.2Multi-agent and search-based ideation

Multi-agent and search-based systems focus more directly on the ideation step. VirSci[48]and IRIS[9]model scientific teamwork through role-separated agents, such as ideators, reviewers, and retrievers. Deep Ideation[60]introduces an explore-expand-evolve workflow over a scientific concept network, with a critic engine trained on real reviewer feedback. Nova[17]adds an iterative planning-and-search loop that retrieves external knowledge to enrich each candidate and improve novelty and diversity.

Other systems make search itself the core ideation mechanism. FlowPIE[53]casts idea generation as a test-time evolution process driven by flow-guided Monte Carlo Tree Search over the literature. This design lets exploration and ideation co-adapt rather than run as separate stages. The multi-workflow benchmark[38]compares reflection-based refinement, Sakana-style evolution, Google Co-Scientist multi-agent reasoning, GPT Deep Research recursive decomposition, and Gemini multimodal long-context workflows. It finds that decomposition-based and long-context workflows reach a mean novelty of 4.17/5. Alien Science[4]formalizes a complementary creativity gap,cognitive availability, and learns to sample coherent but cognitively unavailable directions from clustered “idea atoms.”

2.3Pattern induction from conference outcomes

The closest line of work to ours induces innovation patterns from top-conference papers. Liu et al.[26]annotate Oral papers against a predefined 12-category taxonomy. MoRI[13]extractsmotivation→\toinnovation patternpairs from ICLR 2024–2025 accepted papers. It then trains a reasoning policy through supervised fine-tuning and controllable reinforcement learning on feasibility, novelty, and effectiveness. MotivGraph-SoIQ[22]builds a motivational knowledge graph and pairs it with Socratic dialogue for ideation. Navigating Ideation Space[44]decomposes scientific ideas into conceptual representations for positioning new ideas.

A parallel paradigm grounds idea generation in knowledge graphs mined from large literature corpora. SciMuse[14]builds a knowledge graph from 58 million research papers. In a 100-research-group-leader evaluation, its cross-disciplinary graph paths help AI-generated ideas be perceived as novel and impactful. KG-grounded hypothesis generation[56]retrieves relevant graph sub-structures to constrain LLM hypothesis output and reports improved novelty and plausibility on biomedical benchmarks. Graphs of Research[8]turns citation structure into a training signal. It extracts a per-seed-paper citation-evolution directed acyclic graph from the 2-hop reference neighborhood and fine-tunes an LLM to generate ideas that plausibly extend it.

2.4Novelty, evaluation, and benchmarks

A separate family judges ideas rather than producing them. The first line focuses on novelty collision. NovBench[55]provides a large-scale benchmark for novelty evaluation, with 1,684 paper-review pairs and a four-dimensional framework covering relevance, correctness, coverage, and clarity. RINoBench[41]complements it with 1,381 expert-judged ideas focused on novelty judgment. It finds that LLM novelty verdicts can diverge substantially from expert gold, even when the model gives human-like rationales. We treat this as motivation for grounding novelty checks in retrieved evidence rather than model opinion. OpenNovelty[59]produces transparent evidence-based novelty reports for 500+ ICLR 2026 submissions. GraphMind[7]ties paper key elements to retrieved literature for interactive novelty assessment. Literature-Grounded Novelty Assessment / Idea Novelty Checker[43]uses RAG with two-stage retrieve-then-rerank and literature-grounded rationales. Scideator[36]combines paper facets and novelty assessment modules for human-LLM ideation. Augmenting Research Ideation with Data[27]integrates metadata and automated preliminary validation into ideation, reporting gains in feasibility and quality on social-science tasks.

The second line evaluates idea quality directly. Si et al.[46]ran a 100+-NLP-researcher blind study showing that LLM-generated ideas are ratedmore novelbutless feasiblethan expert ideas. Their follow-up execution study[45]reports that AI ideas score significantly lower than human ideas after execution. The gap is therefore operational, not merely stylistic or novelty-related. AI Can Learn Scientific Taste[50]trains a taste model on accepted/rejected pairs. HindSight[19]evaluates LLM-generated ideas via future-citation impact. Learning to Predict Future-Aligned Research Proposals[52]predicts whether a candidate proposal aligns with future research directions. MIRAI[24]forecasts future paper impact from title and abstract, then uses that predictor to steer idea generation toward higher-impact directions.

The third line packages these judgments as standing benchmarks. IdeaBench[15]evaluates research idea generation against referenced-work baselines. AI Idea Bench 2025[35]curates 3,495 papers from ICLR top-2% and CVPR highlights as ground truth. ResearchBench[28]decomposes scientific discovery into inspiration retrieval, hypothesis composition, and hypothesis ranking. NewtonBench[62]evaluates scientific-law discovery on 324 physics tasks with counterfactual law shifts.

2.5Surveys and skill-engineering practice

Two recent surveys map the broader landscape. “From Automation to Autonomy”[61]organizes systems through a Tool / Analyst / Scientist taxonomy. “Towards Scientific Intelligence”[37]surveys architectures, benchmarks, and ethical questions. On the engineering side, Anthropic’s skill-authoring practices[3]shape IdeaSpark’s two-tier design: progressive disclosure, a lean top-level specification, deterministic routines for repetitive steps, and schema-validated phase boundaries. The embedding and clustering substrate is described where it is used, in §6and §11.

2.6How IdeaSpark differs from current idea-generation methods

Current idea-generation methods fall into three families. End-to-end “AI scientist” systems automate the full research lifecycle. Search-based systems generate many candidates and filter them with learned novelty, taste, or impact signals. Pattern- and knowledge-graph-induction methods mine accepted papers or large literature graphs for reusable structure. IdeaSpark draws ingredients from all three families: pattern induction, an accept/reject contrast, and retrieval-grounded generation. Its distinguishing move is to bind these ingredients onto one object.

That object is the non-parametric, data-induced pattern card. The same card that helps select a move for a bottleneck is also used to audit the generated candidate. This differs from searching a candidate space and ranking ideas by a learned signal. It also differs from fitting a parametric motivation→\tomethod policy, where generation and checking can become separate procedures.

Three properties follow from this design. First, the cards are induced from a three-way Oral / HC / Reject contrast, so they carry an explicitfailuresignal that accepted-only induction and concept-level knowledge graphs do not. Second, the skill stops at one reviewer-facing idea card rather than running the full research lifecycle. This choice reflects the premise that idea quality is an upstream bottleneck for many downstream failures. Third, the audit checks the candidate against the corpus-derived failure modes of the pattern that produced it, together with retrieved prior art. The knowledge that generates an idea and the knowledge that checks it are therefore one object, not two disconnected models. Execution-level gaps are surfaced rather than masked by surface novelty.

3Dataset Construction

3.1Scope and labeling

We collect papers from ICLR, ICML, and NeurIPS for 2021–2025. Collection uses the OpenReview API[34](v1 for ICLR 2021–2023; v2 for ICLR 2024+, ICML 2023+, and NeurIPS 2023+) and Semantic Scholar citation metadata[42]. ICLR 2020 is excluded from this release because its review-data schema differs from later years. For each paper, we retain the title, abstract, author list, OpenReview id, decision string, available review fields, available meta-review, and Semantic Scholar citation count. The review fields include rating, confidence, soundness, contribution, summary, strengths, weaknesses, and questions.

The three labels operationalize the contrast at the heart of this work. Oral marks what program committees most value. High-Cited marks what the community adopts. Reject supplies the failure signal that accepted-only induction cannot provide. Each paper receives one or more labels, and labels may overlap. In this corpus, 49 papers are simultaneously Oral and High-Cited.

Oral(1,014 papers): papers whose venue decision is Oral, marking program-committee preference.

High-Cited (HC)(260 papers,HCallHC_{\mathrm{all}}): top-30 most-cited papers per venue-year by Semantic Scholar citation count, representing community adoption. For 2025, we use the top-10 most-cited papers because citation counts are still accumulating. The high-citation pool overlaps Oral:HCall=260HC_{\mathrm{all}}=260, Oral∩\capHC=49, andHCdisjoint=211HC_{\mathrm{disjoint}}=211. We useHCallHC_{\mathrm{all}}for citation-label coverage andHCdisjointHC_{\mathrm{disjoint}}when analyses need mutually exclusive Oral, HC, and Reject classes.

Reject(722 papers): papers with an explicit Reject decision and a non-trivial review set, representing concrete failure signal.

Table1shows the venue-year breakdown. The per-venue counts reflect two independent factors.

First,venue openness: ICLR is OpenReview-native and publishes both accept and reject decisions with full review threads. Its Reject pool is therefore the deepest. ICML and NeurIPS expose decisions on OpenReview only from 2023 onward. ICML 2023–2024 in particular released very few rejected submissions, which is why those rows show 0–3 in the Reject column.

Second,review accessibility: even when a decision is published, the meta-review and reviewer comments may be withheld or formatted such that automatic parsing fails. This is why metadata coverage (Section3.2) shows only 340 of the 1,947 papers carrying a meta-review, even though all 1,947 have decisions.

Finally, NeurIPS 2025 contributes no HC papers. As the most recent venue (December 2025), its citation counts had not accumulated enough at collection time for even a top-10 cut to be meaningful. These asymmetries do not affect the clustering stage, but we treat them carefully in the quantitative analysis (Section7).

Table 1:Dataset composition. Each column counts a paper if it carries that label; the same paper can appear in multiple columns. HC denotesHCallHC_{\mathrm{all}}. The label-inclusive Total is1,9961{,}996, giving1,9471{,}947unique papers after the4949Oral∩\capHC overlaps. Per-class disjoint pools used in Section7are1,0141{,}014Oral,211211HCdisjointHC_{\mathrm{disjoint}}, and722722Reject. ICLR 2020 is excluded.

3.2Metadata coverage

After merging and deduplicating by OpenReview submission id, 1,947 unique papers remain. All 1,947 carry the eight base fields. The four abstracted fields (Section4) are populated with non-empty content for 1,891 papers. The remaining 56 papers are dropped from the embedding pool (§5).

Coverage of the initial merged metadata varies by field. In total, 1,074 papers carry the full abstract, 716 carry parsed reviews, 682 carry an extracted introduction, and 340 carry a meta-review. For card generation, we run a separate review-recovery pass over public OpenReview threads. The resulting card-level review-evidence coverage is reported in §12. Papers without recoverable review text are still included in signature extraction. Their review-derived fields, however, are treated as low-evidence auxiliary fields rather than evidence grounded in reviewer text.

3.3Convention for in-paper paper references

Throughout this paper, we reference corpus papers by an internal identifier of the form<VENUE>_<YEAR>_<NNNN>. Examples includeICLR_2022_0094andNeurIPS_2023_0753. Each identifier resolves uniquely to an OpenReview submission via theopenreview_idfield indata/dataset/all_records_full.json. After consulting the dataset, a paper can be looked up athttps://openreview.net/forum?id=<openreview_id>.

Of the 1,947 corpus papers, approximately 1,088 are referenced at least once in this report. They usually appear as evidence under a methodology card’s success-condition or failure-mode list. We do not include all corpus papers as bibliography entries. A complete corpus bibliography would obscure the second-order related-work citations that constitute Section2. The prepared dataset is therefore the authoritative bibliographic resolution layer for anypaper_idappearing in this report.

When a specific corpus paper is discussed substantively in body text, we provide a proper bibliography entry. Examples include the diffusion-ELBO paper[20]=NeurIPS_2023_0753, LP-FT[21]=ICLR_2022_0094, the DP continual-observation paper[16]=ICML_2023_0551, EigenGame[10]=ICLR_2021_0018, and SAM-as-Bayes[32]=ICLR_2023_0331.

4Two-Stage Innovation-Signature Extraction

To convert each paper into a form suitable for strategy-level clustering, we extract a twelve-fieldinnovation signature. Eight base fields capture the paper in its own terms. Four describe the innovation mechanism, and four record review and outcome signals. A second stage rewrites the four mechanism fields into domain-agnostic form. Only these four rewritten fields are embedded for clustering. The eight base fields are retained for extraction and later card evidence.

4.1Stage 1: Eight base fields

For Stage 1, we feed Claude Sonnet 4.6[2]the paper’s title, available abstract, available introduction, available reviews, and available meta-review. Introductions are extracted from the PDF when recoverable. The model then returns the following structured fields:

1.innovation_approach: one sentence describinghowthis paper innovates as a reasoning strategy.
2.key_step: the single most critical reasoning step from diagnosis to solution.
3.why_non_obvious: the cognitive barrier that blocked others from this approach.
4.trigger_condition: domain-agnostic situation where this strategy applies, phrased as “When X, apply Y to achieve Z.”
5.reviewer_praise: three specific points the reviewers praised.
6.reviewer_concern: three specific reviewer concerns.
7.acceptance_signal: one sentence explaining what made the paper Oral / cited / rejected.
8.contribution_type: one of {theoretical, methodological, empirical, benchmark, system}.

The first four fields are the most strategy-bearing, so Stage 2 rewrites only those. The last four supply review and outcome evidence used later in card construction. Extraction runs under a JSON schema that enforces non-empty lists of length 3 for the two review-derived fields. All 1,947 papers pass Stage 1 extraction.

4.2Stage 2: Four domain-agnostic rewrites

A direct embedding of paper descriptions tends to latch onto domain nouns, such as “Transformer”, “diffusion”, “image”, or “molecule”. This produces topic clusters rather than strategy clusters. We address this failure mode with a second Claude Sonnet 4.6 pass. It rewrites the first four base fields into strategy-only equivalents. The prompt instructs the model to replace domain nouns with generic placeholders, adopt imperative phrasing (“Audit …”, “Transfer …”, “Design …”), preserve the mechanism, and drop application detail. The resulting four fields are:

1.abstract_strategy←\leftarrowinnovation_approach
2.abstract_key_step←\leftarrowkey_step
3.abstract_why_non_obvious←\leftarrowwhy_non_obvious
4.abstract_trigger_condition←\leftarrowtrigger_condition

A representative contrast illustrates the shift. For the paperHigh Fidelity Speech Synthesis with Adversarial Networks,innovation_approachreads “They solved a domain transfer problem by borrowing the PatchGAN multi-scale discriminator strategy from image GANs and re-instantiating it at multiple temporal windows in audio, adding conditioning signals to close the fidelity gap”. The correspondingabstract_strategyreads “Transfer a multi-scale discrimination strategy from one data modality to another by re-instantiating it at the appropriate domain-specific granularity, then add conditioning signals to close the remaining fidelity gap.” The generic version is what we feed into the embedding model.

The contribution of this abstraction step to strategy-level clustering is quantified in the ablation of §11. All four abstract fields are returned for the full corpus. Of the 1,947 papers,1,8911{,}891carry non-empty content in all four fields. The remaining5656have at least one empty field and are dropped from the embedding pool in §5.

5Unsupervised Pattern Discovery

5.1Embedding

For each paper, we concatenate the four abstracted strategy fields:abstract_strategy,abstract_key_step,abstract_why_non_obvious, andabstract_trigger_condition. Each field is prefixed asfield: content. The average concatenated length is 1,185 characters. After the Stage 2 rewrite of §4, 1,891 of 1,947 papers carry all four fields. We embed these records with OpenAItext-embedding-3-large(3,072 dimensions) and L2-normalize the vectors.

5.2Clustering

UMAP[31]reduces the 3,072-dimensional cosine-normalized space to 10 dimensions. We usen_neighbors=15n\_\text{neighbors}{=}15,min⁡_dist=0\min\_\text{dist}{=}0, and seed 42. HDBSCAN[30]then clusters the reduced space withmin⁡_samples=⌈min⁡_cluster_size/3⌉\min\_\text{samples}{=}\lceil\min\_\text{cluster\_size}/3\rceilandcluster_selection_method=eom. We sweepmin⁡_cluster_size∈{10,15,20,25,30,40}\min\_\text{cluster\_size}\in\{10,15,20,25,30,40\}; Table2reports the full sweep.

Table 2:HDBSCAN sweep on the 1,891-paper embedding. Selected row bolded.We selectmin⁡_cluster_size=10\min\_\text{cluster\_size}{=}10. This setting identifies 31 ideation sub-patterns and gives the strongest silhouette in the sweep (0.5840.584, computed over clustered points). The tradeoff is that902/1,891=47.7%902/1{,}891=47.7\%of papers remain unclustered. Coarseningmin⁡_cluster_size\min\_\text{cluster\_size}does not recover this fraction; it stays in the4040–49%49\%range across the sweep. The unclustered points are therefore far from all density peaks, not fine-granularity artifacts.

What “unclustered” actually means (and why we don’t call it “noise”).

HDBSCAN normally labels points outside any density peak as “noise”. We instead use “unclustered” because the label is geometric, not substantive. It marks where a paper sits in the embedding; it does not mean the strategy is weak, idiosyncratic, or outside the framework.

The flagged47.7%47.7\%arenotstrategy-empty. All1,8911{,}891embedded papers carry full content in the fourabstract_*fields, and the unclustered and clustered groups have comparable median lengths. The label reflects embedding geometry at the chosenmcsthreshold. A paper lands “unclustered” when its strategy embedding falls between two or three modal clusters, often because it executes several ideation patterns in roughly equal measure. In short, “unclustered” meansbetween modal clusters, notoutside the strategy space. The 15-pattern framework works at the paper level through independent multi-label tagging (§6.5), so it covers unclustered and clustered papers equally.

5.3Cluster inventory

Cluster sizes (excluding unclustered) range from 13 (the smallest valid size) to 86, with mean 31.9 and median 22. The five largest clusters are:

•C02(n=86n{=}86):Controlled Diagnostic Construction with One-Variable-Isolated Items, underDesign a Confound-Isolating Diagnostic.
•C18(n=82n{=}82):Substrate Adaptation Across Modalities for a Mature Recipe, underUnify Heterogeneous Inputs into One Space.
•C06(n=59n{=}59):Probabilistic-Objective Unification via Equivalence Proof, underProve Equivalence to Unify.
•C14(n=50n{=}50):All-Pairs Attention Operator Replacement, underSubstitute the Operator or Representation.
•C05(n=50n{=}50):Identifiability-Theorem Pivot on the Limiting Assumption, underAudit and Pivot an Assumption.

Each cluster is automatically labeled by an Opus 4.7 pass. The pass reads the cluster’s eight centroid-nearest representative papers and produces a 2–6 word label plus a one-sentence description. A second Opus 4.7 pass then produces a level-2disambiguation cardfor each cluster. The resulting tactic-level patterns are used in §6.3. Each disambiguation card has six panels:

1.tactical_pattern: the cluster’s specific reasoning move at the granularity below the level-1 ideation pattern.
2.Step-by-Step: a 5-step abstract structural-move recipe distilled from the cluster’s accepted examples, with reject-derived boundaries embedded.
3.differentiation_within_parent: what distinguishes this cluster from its siblings under the same parent.
4.when_to_pick_this_one: the situational trigger conditions under which this sub-pattern is the right choice.
5.tactical_failure_mode: the cluster-specific failure shape that recurs in its Reject papers.
6.Examples: a set ofpaper-agnosticOral and Reject lessons. No per-paper citations are used. The lessons are abstract distillations of the cluster’s success and failure shapes, so that Phase 2.2 generation applies the abstract pattern to a new gap rather than mimicking specific papers.

We use the 31-cluster inventory as raw material for ideation pattern induction in the next section.

5.4Per-cluster acceptance composition

Although the embedding interleaves Oral, HC, and Reject papers across most clusters (Fig.4b), the cluster-level proportions are non-uniform. Figure3shows each cluster’s Oral / HC / Reject composition, sorted by Oral rate. Six clusters clear the Oral-safe threshold (pO≥65%p_{O}{\geq}65\%amongO+RO{+}R,nO≥5n_{O}{\geq}5). One cluster crosses the Reject-warn threshold (pO≤35%p_{O}{\leq}35\%amongO+RO{+}R,nR≥5n_{R}{\geq}5). Each cluster below is written as Oral/HC/Reject counts and Oral share amongO+RO{+}R. The flagging is informational. It is surfaced to human artifact readers and the audit layer, but is never used to pre-filter retrieval or generation:

•Oral-safe:6 clusters. These tactics, when properly executed, land at the top of the acceptance distribution. The five sharpest by Oral share amongO+RO{+}Rare C21Sequential-Control Re-CastingunderReframe as a Solvable Object(13/0/3, 81%), C09Low-Sensitivity Operator SubstitutionunderSubstitute the Operator or Representation(17/0/4, 81%), C16Symmetry-Group ConstructionunderEncode Structure by Construction(19/3/7, 73%), C27Intractable-Problem Re-CastingunderReframe as a Solvable Object(12/0/5, 71%), and C30Curvature-Object SubstitutionunderSubstitute the Operator or Representation(14/1/6, 70%).
•Reject-warn:1 cluster. C13Relational-Topology Encoded as StructureunderEncode Structure by Construction(4/1/12, 25%) is the only cluster whose reject share dominates by thenR≥5n_{R}{\geq}5threshold. Reviewers consistently flag this cluster’s papers as proposing a topology-as-prior architecture without empirically demonstrating that the topology is doing identifiable work over standard alternatives.
•Mixed: the remaining 24 clusters. These tactics produce both Oral and Reject papers in comparable shares. They are the bulk of the working space, and acceptance hinges on the per-paper success conditions and failure modes characterized in the methodology cards (Section6.1).

The per-cluster labels and full counts are inclustering/extra/cluster_composition.csv.

These flags are risk signals, not recommendation labels.

Oral-safedoes not mean “preferred”;Reject-warndoes not mean “forbidden”. They are descriptive risk statistics with limited sample sizes. They are also sensitive to year, conference, and topic saturation. A Reject-warn cluster’s rejected papers may have failed on execution rather than tactic choice. We therefore expose these flags to human artifact readers and to the audit layer as informational risk signals. We do not gate retrieval, generation, or recommendation on them. A user whose problem genuinely matches a Reject-warn cluster’s tactic can still proceed, but the sub-pattern card informs them of the documented failure modes.

Refer to caption Figure 3:Acceptance composition of the 31 clusters, sorted by Oral rate amongO+RO{+}R. Six clusters clear the 65% Oral threshold; one clears the 65% Reject threshold. The remaining 24 are mixed. Cluster labels are colored by risk flag (green = Oral-safe, red = Reject-warn, gray = mixed; deliberately off the bar palette); the threshold usespOp_{O}amongO+RO{+}R, so it is shown via label color rather than anxx-axis line, because the bars areO/HC/RO/HC/Rshares that include HC. Refer to caption (a)31 clusters, colored; unclustered in gray. (b)Colored by label (Oral / HC / Reject).

Figure 4:2D UMAP projection of the 1,891-paper embedding. Oral, HC, and Reject papers are interleaved across essentially every cluster: the embedding captures strategy, not acceptance.

6Ideation-Pattern Induction

Fine-grained ideation sub-patterns answer “what specific move did this paper make?” They do not answer the higher-level question: “what kind of move is that?” To induce that level, we ask Claude Opus 4.7 to build a Level-1 ideation-pattern taxonomy from the 31 clusters. The prompt imposes four constraints. The taxonomy must contain 6–18 entries. Each entry must describe a reusable reasoning strategy rather than a domain or venue. Each entry must include a definition, an operational signature, and a when-to-apply clause. Finally, every cluster must map to one primary ideation pattern, with an optional secondary pattern.

Opus returns a 15-ideation pattern taxonomy (Table3) and a 31-cluster mapping in a single structured call. We make no edits to either. The 15 ideation patterns are:

Table 3:The 15 induced ideation patterns.ncln_{\text{cl}}=fine-grained clusters mapped as primary; Oral / HC / Reject counts are paper-level primary-mapping counts by cluster-level inheritance (989989of the1,8911{,}891embedded papers; the rest are HDBSCAN-unclustered and receive their pattern via the multi-label pass of §6.5).nnis the unique paper count under cluster-level primary; because4949papers are both Oral and High-Cited, a row’s Oral+{+}HC+{+}Reject can exceednn. Figure5sizes each wedge by the Oral+{+}HC+{+}Reject sum.Each ideation pattern is specified at the level needed for use in the skill, with its full definition, operational signature, and when-to-apply clause given in the per-pattern card.

Non-mutually-exclusive operators.

The 15 ideation patterns arenotmutually exclusive categories. Several pairs naturally overlap.Audit and Pivot an Assumptioncan overlap withProve Equivalence to Unify, because an equivalence proof often relaxes a load-bearing assumption.Substitute the Operator or Representationcan overlap withEncode Structure by Construction, because a structured operator both substitutes and encodes.Manufacture the Supervisory Signalcan overlap withDesign a Property-Targeting Pretext Objective, because the latter is a structured form of the former.

The cluster-level mapping forces one primary pattern per cluster for organization. The papers themselves often execute several operators in composition, and the paper-level multi-label tagging in §6.5confirms this empirically. The right reading is therefore compositional: the 15 patterns are ideation operators, and a paper is represented as a composition of operators rather than assigned to one exclusive class.

A note on small-sample ideation patterns.

Three ideation patterns have small per-paper-primary populations under cluster-level inheritance:Adapt by Conditioning, Not Retraining(n=18n{=}18),Characterize a Limit, Then Surpass It(n=15n{=}15), andDesign a Property-Targeting Pretext Objective(n=15n{=}15). These are valid operators because the underlying clusters are statistically distinct in embedding space. They are alsolow-support operators: their per-pattern acceptance statistics rest on thin evidence and should not be used as strong recommendation priors. The corresponding analytical cards therefore carry an internalconfidence: lowflag.

The runtime prompt cards do not expose raw counts or confidence flags to the model, because doing so induced distribution-bias homogenization toward high-nnpatterns in earlier designs. Low-support patterns remain available when the user’s problem genuinely fits them. They are neither privileged nor down-weighted by the skill’s selection step. The corpus-side flag is retained here for analytical transparency.

Plain-language aliases.

The ideation pattern names use compact technical phrasing for analytical precision: for example,Audit and Pivot an AssumptionandSubstitute the Operator or Representation. Each pattern also has aplain-language aliasfor user-facing skill output. This keeps the framework legible to non-specialists. Examples includeAudit and Pivot an Assumption→\to“Audit the load-bearing assumption and pivot”,Reframe as a Solvable Object→\to“Reformulate the unsolved as a solvable object”, andSubstitute the Operator or Representation→\to“Substitute the operator or representation”.

6.1From definition to operational card

A definition and operational signature tell a researcherwhatan ideation pattern is. They do not explainwhen it lands and when it fails. To capture that distinction, we run a second Opus 4.7 pass for each ideation pattern. The input includes all Oral, HC, and Reject papers assigned to the pattern, their four abstracted fields, and the available review and meta-review evidence. Review evidence is recovered for 83% of the corpus overall; details are in §12. The pass returns a structured card with seven panels:

1.success_conditions: cross-Oral patterns of what makes the pattern land.
2.failure_modes: cross-Reject patterns of how it fails.
3.oral_reject_gap: the prose contrast between the two pools.
4.oral_hc_gap: the prose contrast between PC-favored and community-favored execution.
5.reviewer_expectations: bullets tagged by provenance ([oral_reviews],[reject_reviews],[both]).
6.cognitive_barriers: the field-level blind spots that make the pattern non-obvious.
7.representative_examples: 6–8 paper-anchored exemplars (Oral and Reject), each with a one-line lesson.

Evidence-bearing fields must cite paper IDs. This rule applies tosuccess_conditions,failure_modes,oral_reject_gap,oral_hc_gap,reviewer_expectations, andrepresentative_examples. The model is also instructed to setconfidence=lowwhen sample sizes or review coverage fall below thresholds, and to refuse the Oral-vs-HC contrast whennHC<5n_{HC}<5.

The 15 resulting cards are the operational substrate of the skill. Definitions, operational signatures, and when-to-apply clauses support Phase 2.1 pattern fit. Sub-pattern tactical recipes support Phase 2.2 instantiation. Failure modes, Oral-vs-Reject gaps, representative examples, and reviewer expectations support Phase 3 audit and Phase 4 explanation. They are not used as acceptance-rate priors or ranking signals.

The cards’Examplespanel surfacespaper-agnostic Oral and Reject lessons: bullets distilled from the cluster’s acceptance pool, without per-paper citations. This design keeps Phase 2.2 focused on applying the abstract pattern to a new gap rather than mimicking any single corpus paper. App.Dreproduces theAudit and Pivot an Assumptioncard—paired with one of its sub-pattern cards—as a high-coverage corpus-to-card example.

6.2Coverage and granularity

The 15 ideation patterns have two coverage figures, and we report both because they answer different questions:

•Cluster-level coverage: 989 of 989 clustered papers (100%100\%of clustered;52%52\%of the 1,891-paper embedding pool) receive a primary ideation pattern by inheritance from their ideation sub-pattern. Every cluster maps to exactly one of the 15 primary patterns; there is no out-of-taxonomy residual.
•Paper-level multi-label coverage: the independent multi-label tagging pass (§6.5) assigns one or more ideation pattern labels toallpapers including the902902HDBSCAN-unclustered ones. The 15-ideation pattern framework therefore covers the corpus completely at the paper level, even though only52%52\%of papers are assignable via cluster-level inheritance.

The distribution has a long tail. By cluster-level primary count, the top three ideation patterns account for 384 papers:Audit and Pivot an Assumption(n=181n{=}181),Substitute the Operator or Representation(n=109n{=}109), andLiberate a Fixed Generative Component(n=94n{=}94). The bottom three account for 48 papers:Adapt by Conditioning(n=18n{=}18),Characterize a Limit(n=15n{=}15), andDesign a Property-Targeting Pretext Objective(n=15n{=}15). This is the long-tail signature of an inductively derived taxonomy. Many innovations use a small number of deeply worked strategies; a smaller tail uses rare but still coherent operators.

Figure5visualizes the hierarchy with three concentric rings. The inner ring shows the 15 ideation patterns, sized by paper count. The middle ring subdivides each pattern into its constituent sub-clusters, 31 in total, tinted with the parent pattern’s color. The outer band encodes each pattern’s Oral acceptance rate on a light→\rightarrowdark gray scale. Two empirical observations follow.

(i) Methodology fragmentation is uneven across patterns, but the taxonomy is not over-split.Audit and Pivot an Assumptiondecomposes into 6 sub-clusters.Substitute the Operator or Representationdecomposes into 4.Reframe as a Solvable Object,Manufacture the Supervisory Signal,Encode Structure by Construction, andLiberate a Fixed Generative Componentdecompose into 3 each. The remaining 9 patterns are single-cluster patterns, which indicates tactically coherent moves that the abstraction does not split further. The 6-cluster decomposition underAudit and Pivotreflects substantively different audit-pivot moves, such as identifiability-theorem pivots, distributional-assumption pivots, and sufficiency-bound pivots.

(ii) Cluster sizes are tight and well-supported.Cluster sizes range from 13 to 86 (mean 31.9, median 22). Of the 31 clusters, only 2 are small (n<15n<15); the corresponding tactical cards are auto-flagged asconfidence: low. The tight size distribution corresponds to a high silhouette (0.5840.584).

Refer to caption Figure 5:Ideation-pattern hierarchy across 989 clustered papers.Three concentric rings share one angular layout, ordered by size clockwise from 12 o’clock; categorical colors identify the 15 patterns and are reused across all rings, with white gaps separating segments.Inner ring: the 15 induced Level-1 ideation patterns, each wedge sized by its clustered-paper count (the count is printed inside the larger wedges; full pattern names are in the right-hand legend).Middle ring: each pattern’s arc is subdivided into its constituent sub-clusters (31 in total), shaded as a lighter tint of the parent pattern; thenumberof segments within a wedge shows how finely that pattern fragments, while the segmentangleis the equal share within the pattern, not the per-sub-cluster paper count.Outer ring: a thin heat band encoding each pattern’s Oral acceptance ratepO=nO/(nO+nHC+nR)p_{O}=n_{O}/(n_{O}+n_{HC}+n_{R})on the light-to-dark gray scale shown at left (light = low, dark = high), so reviewer outcome can be read against methodology at a glance. Totals: 15 patterns, 31 sub-clusters, 989 clustered papers (of 1,891 embedded).

6.3Multi-strategy assignments

This subsection reports the cluster-level view of multi-strategy composition. This view is deliberately limited: the cluster-level mapping caps each paper atprimary + secondary. The deeper paper-level multi-label view in §6.5recovers the true modal composition, including the 902 HDBSCAN-unclustered papers excluded here. It should be read as the authoritative composition analysis.

Paper-level coverage of the secondary slot.The cluster-level(primary, secondary)assignment propagates to papers. Of the 989 clustered papers,729729(73.7%73.7\%) carry both a primary and a non-trivial secondary, while260260(26.3%26.3\%) carry only a primary. The902902HDBSCAN-unclustered papers are excluded from these cluster-level statistics and assigned later by multi-label tagging in §6.5. Multi-strategy assignment is therefore the norm among coherently clustered papers.

Two cluster-level signals are worth stating because both fall outside the Oral-enriched paper-level ranking in §6.5(Table6). First, the clearest reject-enriched combination isarchitectural operator substitution++heterogeneous decomposition(−17.9-17.9pp from the58.4%58.4\%baseline), followed byalgebraic equivalence unification++operator substitution(−8.4-8.4pp). The likely failure mode is stacked unresolved tradeoffs: a more expressive operator is asked to absorb multi-population heterogeneity, so reviewers read the operator as the architecture rather than as a controlled inductive bias. Second, composition itself tracks acceptance. The729729paired papers reachpO=58.2%p_{O}=58.2\%(366/263366/263), essentially the58.4%58.4\%baseline. The260260single-pattern papers reach only52.7%52.7\%(116/104116/104,−5.7-5.7pp). A single-primary cluster tends to be a lone tactical move, whose acceptance signal is weaker than a deliberately composed pair.

6.4Ideation-pattern adjacency in embedding space

Why do certain ideation pattern pairs combine often? The OpenAItext-embedding-3-largeembedding gives a direct geometric measure. Each ideation pattern has a centroid, computed as the mean of its papers’ normalized embeddings. Pairwise centroid cosine then quantifies how close two patterns are in strategy space. Figure6shows the full15×1515\times 15matrix, and Table4gives each pattern’s nearest neighbor.

Refer to caption Figure 6:Centroid-cosine similarity between the 15 ideation patterns. Rows/columns ordered by mean similarity (most “central” ideation patterns first). All off-diagonal entries fall in[0.79,0.96][0.79,0.96]: ideation patterns are not orthogonal in this embedding; they form a dense neighborhood on a low-dimensional manifold.Table 4:Nearest-neighbor ideation pattern and centroid cosine. Sorted by descending cosine. The two most isolated patterns areDesign a Confound-Isolating Diagnostic(0.9040.904) andCharacterize a Limit, Then Surpass It(0.9170.917); they do not share a neighborhood and form their own pockets in the embedding.Three observations:

(i) Adjacency predicts the Oral-enriched combinations.The top three cluster-level Oral-enriched pairs are all high-cosine neighbors:Audit and Pivot++Structural Prior Encoding(+6.1+6.1pp),Operator Substitution++Audit and Pivot(+5.5+5.5pp), andAudit and Pivot++Controlled Diagnostic Design(+3.7+3.7pp).Audit and Pivot an Assumptionsits at cosine0.9490.949fromOperator Substitution, and connects toStructural Prior Encodingat0.9460.946. Geometric proximity is therefore useful evidence for which secondary pattern is plausible to compose with a given primary.

(ii) The two most isolated patterns are not simply the smallest.Design a Confound-Isolating Diagnostic(n=86n{=}86) has the lowest nearest-neighbor cosine (0.9040.904).Characterize a Limit, Then Surpass It(n=15n{=}15) has the second-lowest (0.9170.917). Isolation therefore does not track support alone. Thecontrolled diagnosticpattern is geometrically distinct because its operational signature is to build a confound-isolating measurement instrument, which does not compose with the other 14 patterns in a standard way. This makes it a high-novelty leg candidate.

(iii) Three patterns share hub status.Audit and Pivot an Assumption,Reframe as a Solvable Object, andSubstitute the Operator or Representationeach appear as the nearest neighbor for 3 other patterns. The hub set is distributed across diagnosis, framing, and substitution moves. Phase 2.1 therefore has three balanced default secondaries to consider when the bottleneck does not already imply a specific composition.

6.5Paper-level multi-label ideation pattern assignment

The cluster-level mapping in §6caps each cluster, and therefore each clustered paper, atprimary + secondary. To test whether 2 is the right cap, and to surface 3+ pattern combinations directly, we run a separate Sonnet 4.6 pass on every paper independently. Given the 15 ideation-pattern definitions, the paper title, the abstract, and the fourabstract∗fields, the model emits a JSON list of patterns the paper actuallyexecutes, not merely mentions. Each label includes a quoted evidence span and aprimary/supportingrole. There is no upper cap onk=|applied_patterns|k=|\text{applied\_patterns}|; the minimum is 1.

All 1,891 papers with the four abstracted fields received a non-empty multi-label assignment. Rather than report a Bayesian acceptance rate (which requires constructing a denominator that mixes the OpenReview Oral/Reject decision with the citation-based HC class), we report theclass-conditional distributionP(k∣decision)P(k\mid\text{decision})directly. Table5shows, within each acceptance class, what fraction of that class’s papers executekkideation patterns.

Table 5:Class-conditionalkkdistribution. Each column sums to 100% within its class.k¯\bar{k}is the per-class mean.Three observations:

(i)k=2k{=}2is the empirical mode in every class.Overall, 59.2% of papers execute exactly two ideation patterns. The mode is shared across Oral, HC, and Reject. Only7.2%7.2\%of papers execute a single pattern, while a33.6%33.6\%tail executes three or more. The cluster-based view in §6.3, capped atk≤2k\leq 2by construction, therefore under-counts the 3-way tail. Paper-level multi-label tagging is the right granularity for capturing that tail.

(ii) Class means cluster tightly aroundk¯=2.3\bar{k}=2.3.The class meansk¯R=2.26≤k¯O=2.28≤k¯HC=2.32\bar{k}_{R}=2.26\leq\bar{k}_{O}=2.28\leq\bar{k}_{HC}=2.32are statistically indistinguishable at this sample size. Compositionsizedoes not separate acceptance classes; compositioncontent(which patterns) does (§7).

(iii) Cluster-pair assignment under-represents true ideation-pattern coverage.Cross-tabulating against the cluster-derived(primary, secondary)pair shows that multi-label tagging picks up signal beyond the dominant cluster. Most papers have at least one supporting pattern outside their cluster pair. The cluster pair captures the dominant axis, but not the routine 1–2 supporting moves that multi-label tagging surfaces.

Refer to caption Figure 7:Multi-label ideation pattern assignment. (a)kkdistribution as grouped counts by acceptance class. (b) Class-conditional distributionP(k∣decision)P(k\mid\text{decision}): each curve is a probability mass function summing to 100%.k=2k{=}2is the mode for all three classes.##### Top combinations under multi-label tagging.

With paper-level multi-label assignments, we can rank 2-way and 3-way combinations directly (Table6). We use the same definition ofpO=nO/(nO+nR)p_{O}=n_{O}/(n_{O}+n_{R})as in the cluster-level table, so the cross-table comparison is clean; the dataset baseline ispO=58.4%p_{O}=58.4\%. Three combinations stand out:Algebraic Equivalence Unification+Generative Process Redesign(nO+nR=20n_{O}+n_{R}=20,pO=85.0%p_{O}=85.0\%,+26.6+26.6pp),Algebraic Equivalence Unification+Heterogeneous Decomposition(nO+nR=22n_{O}+n_{R}=22,pO=81.8%p_{O}=81.8\%,+23.4+23.4pp), andAudit and Pivot+Characterize a Limit, Then Surpass It(nO+nR=60n_{O}+n_{R}=60,pO=80.0%p_{O}=80.0\%,+21.6+21.6pp). The third is the highest-volume strongly Oral-enriched combination in the dataset. Its empirical signature is: identify the load-bearing assumption, then prove a strict improvement over the limit it imposes. The Oral-enriched side of the 2-way table is consistent: pair a diagnostic or unification move with either a constructive substrate or a sharp analytical commitment.

Table 6:Top ideation pattern combinations under multi-label tagging, ranked bypO=nO/(nO+nR)p_{O}=n_{O}/(n_{O}+n_{R})for direct comparability with the cluster-level combinations of §6.3. 2-way thresholdnO+nR≥20n_{O}+n_{R}\geq 20; 3-way thresholdnO+nR≥10n_{O}+n_{R}\geq 10. BaselinepO=58.4%p_{O}=58.4\%(1014 Oral, 722 Reject).CombinationnOn_{O}nHCn_{HC}nRn_{R}pOp_{O}ΔpO\Delta p_{O}Top 2-way (nO+nR≥20n_{O}+n_{R}\geq 20)algebraic equivalence unification+generative process redesign170385.0%+26.6+26.6algebraic equivalence unification+heterogeneous decomposition182481.8%+23.4+23.4assumption audit & pivot+characterize limit then surpass4821280.0%+21.6+21.6generative process redesign+reframe as solvable object221775.9%+17.5+17.5algebraic equivalence unification+structural prior encoding150768.2%+9.8+9.8generative process redesign+self-supervised signal engineering197967.9%+9.5+9.5decompose and delegate+reframe as solvable object191967.9%+9.5+9.5controlled diagnostic design+heterogeneous decomposition2081066.7%+8.3+8.3Top 3-way (nO+nR≥10n_{O}+n_{R}\geq 10)audit & pivot+decompose & delegate+reframe as solvable object80280.0%+21.6+21.6audit & pivot+characterize limit then surpass+reframe as solvable object110378.6%+20.2+20.2audit & pivot+controlled diagnostic design+heterogeneous decomposition82372.7%+14.3+14.3audit & pivot+generative process redesign+heterogeneous decomposition83561.5%+3.1+3.1audit & pivot+reframe as solvable object+structural prior encoding120957.1%−1.3-1.3 Refer to caption Figure 8:Top combinations under multi-label tagging. (a) 2-way combinations (n≥20n\geq 20); (b) 3-way combinations (n≥10n\geq 10). Dotted line: dataset-wide baselinepO=58.4%p_{O}=58.4\%.

What multi-label tagging implies for the skill.

The skill treats ideation-pattern composition as 1–3 entries per generated idea, withk=2k=2as the empirically supported default. For each composed pattern, it draws success and failure conditions from the ideation-pattern cards in §6.1. The combination statistics in Table6are surfaced only as context through the candidate’sinnovation_pattern_landscapeblock. That block shows the area’s pattern usage alongside the candidate’s chosen composition, so the user can see where the idea sits in the empirical distribution.

Pattern selection itself happens upstream by structural fit to the bottleneck in Phase 2.1, not by composition-rate priors. Loading priors at generation time was tested in an earlier design and removed because it induced distribution-bias homogenization across runs. The Oral-enriched combinations in Table6are therefore empirical observations, not skill target templates.

7Acceptance and Impact Analysis

We define acceptance bias asΔOR(m)=poral(m)−preject(m)\Delta_{\text{OR}}(m)=p_{\text{oral}}(m)-p_{\text{reject}}(m). Here,poral(m)p_{\text{oral}}(m)is the fraction ofcluster-level Oral primarieswhose primary pattern ismm, andpreject(m)p_{\text{reject}}(m)is the analogous fraction for Reject papers. PositiveΔOR\Delta_{\text{OR}}means the pattern appears more often in accepted work than in rejected work, relative to class totals. NegativeΔOR\Delta_{\text{OR}}means the opposite.

This analysis is cluster-level and therefore partial. A cluster-level primary is available for482482of1,0141{,}014Oral papers and367367of722722Reject papers. The denominators ofpX(m)p_{X}(m)are the cluster-level totals (NO=482N_{O}=482,NHC=172N_{HC}=172,NR=367N_{R}=367), not the full class populations. The remaining HDBSCAN-unclustered papers are covered by the multi-label tagging pass in §6.5. Figure9plots the bias for all 15 patterns; Table7reports all values.

Table 7:Per-ideation pattern acceptance bias (cluster-level).pX(m)p_{X}(m)is the fraction of the cluster-level-primary papers in classXXwhose primary is patternmm. Denominators are the cluster-level-primary totals:NO=482N_{O}=482Oral,NHC=172N_{HC}=172HC,NR=367N_{R}=367Reject (these are the subsets of the full1,014/260/7221{,}014/260/722class populations that received a cluster-level primary; the remaining HDBSCAN-unclustered papers are covered by the multi-label tagging of §6.5).ΔOR=poral−preject\Delta_{\text{OR}}{=}p_{\text{oral}}-p_{\text{reject}}in percentage points.ΔOH=poral−phc\Delta_{\text{OH}}{=}p_{\text{oral}}-p_{\text{hc}}(positive = PC-favored over community); reported only whennHC≥5n_{\text{HC}}\geq 5. Refer to caption Figure 9:Per-ideation-pattern acceptance bias. Bars showΔOR\Delta_{\mathrm{OR}}; positive bars are Oral-enriched and negative bars are Reject-enriched. Diamonds showΔOH\Delta_{\mathrm{OH}}wherenHC≥5n_{HC}\geq 5.Two axes structure the table. The Oral-vs-Reject axisΔOR\Delta_{\text{OR}}is tight, with a spread of only±2.9\pm 2.9pp. At the main-pattern level, strategy choice contributes little to acceptance. The orthogonal PC-vs-community axisΔOH\Delta_{\text{OH}}is much wider, reaching roughly±13\pm 13pp, and is the stronger discriminating signal. Four observations follow.

(O1) TheΔOR\Delta_{\text{OR}}axis is tight.The range is±2.9\pm 2.9pp:Reframe as a Solvable Objectis at+2.9+2.9, whileDecompose for Differentiated Treatmentis at−2.9-2.9. This narrow spread suggests that the main pattern alone explains little acceptance variance. Execution at the sub-pattern level, meaning the cluster rather than the Level-1 pattern, is what discriminates.

(O2)Audit and Pivotis high-volume, high-variance, slightly Reject-leaning, and heavily PC-favored.It carries19.5%19.5\%of Oral papers, the largest cluster-level Oral share, and21.5%21.5\%of Reject papers, also the largest. This givesΔOR=−2.0\Delta_{\text{OR}}{=}{-}2.0pp. ItsΔOH=+13.1\Delta_{\text{OH}}{=}{+}13.1pp is the largest positive PC bias in the taxonomy: PCs disproportionately Oral audit/pivot papers, while the community cites them less than expected. Execution quality, not strategy choice, separates this pattern’s9494Oral from7979Reject papers.

(O3)Unify Heterogeneous Inputs into One Spaceis the dataset’s clearest community-favored pattern.ItsΔOH=−11.2\Delta_{\text{OH}}{=}{-}11.2pp is driven by a17.4%17.4\%HC share against a6.2%6.2\%Oral share, proportionally almost three times more HC than Oral. Cross-modal-alignment papers are reproduced and cited heavily, but PCs do not elevate them to Oral at the same rate. This is the inverse ofAudit and Pivot.

(O4)Reframe as a Solvable Objectis the cleanest Oral signal.BothΔOR=+2.9\Delta_{\text{OR}}{=}{+}2.9pp andΔOH=+7.1\Delta_{\text{OH}}{=}{+}7.1pp are positive. PCs reward the pattern, and the community does not over-cite it relative to Oral frequency. Combined with its high cosine toAudit and Pivot(0.9560.956) in the adjacency analysis, this supports a concrete Oral recipe: recast an intractable native formulation as a solvable object, then audit and pivot the assumption that the recast makes load-bearing.

The PC-vs-community axis (ΔOH\Delta_{\text{OH}}).

TheΔOH=poral−phc\Delta_{\text{OH}}{=}p_{\text{oral}}-p_{\text{hc}}column isolates a second axis: whether program committees and the citing community agree on what counts. Observations O2–O4 show that PCs tend to reward structural-insight moves, while the community tends to reward usable-infrastructure moves. The two rankings are nearly orthogonal. The audit layer therefore surfacesΔOH\Delta_{\text{OH}}as a distinct signal, separate from the acceptance-prediction axis.

Use in the skill.

IdeaSpark does not use the acceptance or PC-vs-community statistics as selection priors at any phase. Pattern selection is driven by structural fit to the bottleneck. The statistics enter only as context for audit.

8Domain Distribution and Ideation-Pattern Breadth

The ideation-pattern cards (§6.1) and the per-pattern bias profile (§7) treat the 1,947 papers as one population. ML, however, is not a monolithic field. A strategy that lands an Oral inlearning theorymay not land invision-language pretraining. IdeaSpark therefore needs to distinguish broadly transferable patterns from domain-conditional ones. This section quantifies that distinction.

8.1Domain induction

We do not preset a domain taxonomy. Instead, we induce one from the data using the same architecture that produced the 15 ideation patterns:

1.Sonnet 4.6 extracts 1–3 normalized research-subarea tags per paper from its title, abstract, and (when available,77.9%77.9\%of the dataset) author-supplied OpenReview keywords.
2.The 3,909 unique free-form tags are embedded with OpenAItext-embedding-3-largeand clustered with UMAP+HDBSCAN (k=148k{=}148tag-clusters atmcs=7\textit{mcs}{=}7, unclustered25.5%25.5\%).
3.Sonnet 4.6 labels each tag-cluster.
4.Opus 4.7 induces a Level-1 domain taxonomy ofN∈[8,30]N\in[8,30]entries and maps each tag-cluster to a domain label.

The model returnsN=28N{=}28domains. We make no edits.

The per-paper assignment is the union of domains over each paper’s tags. Coverage is high:98.5%\mathbf{98.5\%}of papers (1,918 / 1,947) receive at least one domain. The average paper receives1.811.81domains. The 28 domains range fromn=256n{=}256(Transfer, Continual & Meta-Learning) ton=16n{=}16(Online & Bandit Learning). The full ordered list and machine-readable taxonomy are inclustering/domains/domain_taxonomy.json.

8.2Domain×\timesideation pattern heatmap

We cross the 28 induced domains with the 15 ideation patterns using paper-level multi-label tagging (§6.5). Each paper contributes one count to every (domain, pattern) cell it belongs to. The grid has28×15=42028\times 15=420cells. Of these,392392(93%93\%) contain at least one paper, and292292(70%70\%) havenO+nR≥5n_{O}+n_{R}\geq 5. Figure10shows two views. Panel (a) reports raw paper count per cell. Panel (b) reports the cell-level Oral ratepO=nO/(nO+nR)p_{O}=n_{O}/(n_{O}+n_{R}), masked wherenO+nR<5n_{O}+n_{R}<5.

Refer to caption Figure 10:(a) Paper counts per (domain, ideation pattern) cell, with domains and ideation patterns sorted by total. (b) Per-cell Oral ratepOp_{O}(deeper green is higher; gray cells havenO+nR<5n_{O}+n_{R}<5, too few papers to estimate a rate; the colorbar marks the58.4%58.4\%dataset average). The colored signal in (b) is heterogeneous: the same ideation pattern lands at very different Oral rates across domains.The cell-level view adds domain specificity to §7. The same ideation pattern can land at very different Oral rates across domains. One pattern is the main exception:Characterize a Limit, Then Surpass Itgives the cleanest cross-domain Oral signal in the corpus, occupying four of the five strongest cells. The five strongest cells, each withpO≥90%p_{O}\geq 90\%onnO+nR≥8n_{O}+n_{R}\geq 8, are:

•Trustworthy & Responsible ML×\timesCharacterize a Limit, Then Surpass It:𝟏𝟎𝟎%\mathbf{100\%}Oral (nO=12n_{O}{=}12,nR=0n_{R}{=}0), the dataset’s cleanest acceptance cell.
•Learning Theory & Generalization×\timesCharacterize a Limit, Then Surpass It:100%100\%(11/011/0).
•Reinforcement Learning×\timesDesign a Confound-Isolating Diagnostic:93%93\%(13/113/1).
•Representation & Self-Supervised Learning×\timesCharacterize a Limit, Then Surpass It:91%91\%(10/110/1).
•Graph Learning×\timesCharacterize a Limit, Then Surpass It:91%91\%(10/110/1).

The five weakest dense cells (pO≤25%p_{O}\leq 25\%) are:

•Distributed & Federated Learning×\timesReframe as a Solvable Object:9%9\%.
•Graph Learning×\timesSubstitute the Operator or Representation:21%21\%.
•Visual Perception & Recognition×\timesDecompose for Differentiated Treatment:22%22\%.
•Model Efficiency & Compression×\timesAdapt by Conditioning, Not Retraining:25%25\%.
•Graph Learning×\timesLiberate a Fixed Generative Component:25%25\%.

The weakest dense cells show the other side of the same result. A pattern that wins in one domain can under-perform sharply in another. This supports the cards’ guidance that success conditions do not transfer mechanically across domains.

The cross-domain consistency ofCharacterize a Limit, Then Surpass Ithas a clear operational basis. When a paper identifies a formal limit, such as an information-theoretic, sample-complexity, expressivity, or generalization limit, and proves a strict improvement over it, the reviewer-defensible signature transfers across otherwise different domains.

8.3Ideation-pattern breadth: how domain-agnostic is each pattern?

For the skill to judge whether a pattern is plausible in a user’s area, it needs each pattern’s domain footprint. Figure11plots, for each ideation pattern, the number of domains it touches and the share of its papers concentrated in its largest domain.

Refer to caption Figure 11:Ideation-pattern breadth: number of distinct domains each ideation pattern has≥5\geq 5papers in (out of 28). The annotation gives each pattern’s multi-label paper count: a paper is counted under every pattern it carries, so these counts overlap across patterns and sum to more than the1,8911{,}891-paper corpus. Most patterns touch≥18\geq 18domains, supporting the skill’s domain-agnostic positioning.Three observations:

(i) Diagnostic and framing patterns reach the entire field.Audit and Pivot an AssumptionandReframe as a Solvable Objecteach have≥5\geq 5papers inall 28 domains.Decompose for Differentiated Treatmentreaches 27 domains. These three are also the highest-volume patterns in the multi-label distribution, with10081008,527527, and529529papers respectively under tagging. They also anchor the Oral-enriched multi-label combinations in Table6. Breadth, volume, and acceptance signal therefore co-occur.

(ii) Construction-level patterns are still broad, but more concentrated.Encode Structure by Constructionreaches2525domains,Liberate a Fixed Generative Componentreaches2525,Manufacture the Supervisory Signalreaches2323, andSubstitute the Operator or Representationreaches2222. These patterns depend on a specific architectural choice or generative inductive bias. They appear across most of the field, but with clearer domain-modal anchors. Generative redesign concentrates inDiffusion, Flow & Score Models, and operator substitution concentrates inSequence Modeling & Architectures.

(iii) Domain-narrow patterns are not necessarily Reject-prone.Relax Discrete Search to Continuoustouches99domains,Unify Heterogeneous Inputs into One Spacetouches1111, andCharacterize a Limit, Then Surpass Ittouches1212. The first two are narrow because their operational substrates exist only in some subfields: discrete-search-amenable problems and cross-modal alignment scaffolds.Characterize a Limit, Then Surpass Itis also narrow, but it is the cleanest cross-domain Oral signal we observe (§8.2). Its narrowness reflects a prerequisite. The field must already have produced a formal limit characterization, which is not true everywhere.

We do not report a per-domain temporal slice. Conditioning on (domain×\timesyear×\timespattern) leaves only 1–3 papers per cell for smaller domains, which is too thin for a quantitative claim. The skill also does not rely on such aggregate slices. Its Phase 2.1 saturation read is computed live from each run’s Phase 0 retrieval (§9), so domain-specific saturation is read from the actual retrieved literature rather than from coarse dataset bins.

9Temporal and Conference Structure

9.1Temporal trends

Refer to caption Figure 12:Per-pattern share of papers by year under multi-label tagging (% of that year’s papers carrying the pattern in any role; rows sum tok¯≈230%\bar{k}\approx 230\%). Left: top-8 by total volume. Right: remaining seven. Lines that rise through the window are the patterns currently ascending; those that fall are losing share.We summarize temporal change across the 2021–2025 window by comparing an early period, defined as the average over 2021–2022, with a late period, defined as the average over 2024–2025. The clearest movers are:

•Risers.Decompose and Delegate to Solvers(0.0%0.0\%early→\to6.3%6.3\%late,Δ=+6.3\Delta=+6.3pp) is the dataset’s fastest-growing pattern. It tracks the emergence of LLM-driven multi-agent and tool-orchestrated solver pipelines from 2023 onward.Design a Confound-Isolating Diagnostic(11.3%→15.6%11.3\%\to 15.6\%,Δ=+4.3\Delta=+4.3pp) reflects the maturation of controlled-perturbation evaluation as a first-class research move.Unify Heterogeneous Inputs into One Space(Δ=+2.6\Delta=+2.6pp) is the third clearly rising pattern.
•Fallers.Decompose for Differentiated Treatment(34.7%→26.2%34.7\%\to 26.2\%,Δ=−8.5\Delta=-8.5pp) loses the most ground. The “split into sub-populations and treat differently” move appears to have been absorbed into more specialized patterns, including operator substitution and structural priors.Audit and Pivot an Assumption(58.7%→50.9%58.7\%\to 50.9\%,Δ=−7.8\Delta=-7.8pp) remains the highest-volume pattern, but gradually gives back share.Liberate a Fixed Generative Component(Δ=−5.7\Delta=-5.7pp) is the third faller, consistent with saturation of the generative-redesign wave that dominated 2021–2023.

Sample-size caveat for early years.

The corpus is unbalanced across years. The 2021 slice has 77 multi-labeled papers and 2022 has 73, while each year from 2023 to 2025 has537537–606606. The early-period share is therefore aggregated over a∼\sim150-paper base. Changes at the lower end of theΔ\Deltatable (|Δ|<2|\Delta|<2pp) are not statistically robust, so we do not call them out. The risers and fallers above all carryΔ≥2.5\Delta\geq 2.5pp, corresponding to absolute paper-count moves of more than∼15\sim 15papers per direction.

How the skill uses these trend signals.

These corpus-wide temporal verdicts are descriptive context only. The skill never loads them as generation priors. Instead, Phase 1 reads pattern saturation live from each run’s own retrieved literature (§13).

9.2Conference preferences

Figure13reports each pattern’s share within each venue, measured as the percentage of that venue’s papers carrying the pattern under multi-label tagging. Three venue signatures emerge.

Refer to caption Figure 13:Pattern usage by venue under multi-label tagging. Each bar reports the share of papers in a venue that carry the pattern in any role. Patterns are sorted by total volume.ICML over-indexes on assumption-audit and limit-characterization.Audit and Pivot an Assumptionreaches56.2%56.2\%share at ICML versus52.4%52.4\%elsewhere, a+3.8+3.8pp lift.Characterize a Limit, Then Surpass Itreaches5.7%5.7\%versus4.3%4.3\%, a+1.4+1.4pp lift. Both patterns are diagnostic or theoretical, consistent with ICML’s preference for sharply bounded analytical contributions.

ICLR over-indexes on construction-side moves.Encode Structure by Constructionreaches20.6%20.6\%versus16.2%16.2\%elsewhere (+4.4+4.4pp).Liberate a Fixed Generative Componentreaches16.2%16.2\%versus13.3%13.3\%(+3.0+3.0pp). ICLR also under-indexes onProve Equivalence to Unify(−2.7-2.7pp) andAudit and Pivot an Assumption(−4.5-4.5pp). The venue signature favors architectural and representational construction over algebraic unification.

NeurIPS over-indexes on heterogeneity-driven decomposition and supervisory-signal engineering.Decompose for Differentiated Treatmentreaches31.4%31.4\%versus26.8%26.8\%elsewhere (+4.7+4.7pp).Manufacture the Supervisory Signalreaches18.0%18.0\%versus14.3%14.3\%(+3.7+3.7pp). NeurIPS under-indexes onDesign a Confound-Isolating Diagnostic(−4.0-4.0pp). The venue signature is broadly systems-level composition plus invented supervision, consistent with its empirical bias toward large-experiment papers.

These venue signatures are quantified per pattern. The skill surfaces them only as audit context for the Phase 4 idea card, so the user can see the venue prior alongside the candidate’s chosen pattern composition.

10Do Rejected Papers Inhabit a Different Strategy Space?

A natural concern is that clustering Oral, HC, and Reject papers in one space may force Reject papers into clusters defined by accepted work. If rejected papers useddifferentstrategies, the induced ideation patterns would under-represent them. We test this directly. The result is negative: rejected work clusters within the same 15-pattern taxonomy rather than in a strategy space of its own.

10.1Reject-only clustering and mapping

We re-run the embedding+clustering pipeline on the 711 Reject papers that carry fullabstract_*fields. Of the 722 Reject papers, 11 lack one of the four fields. We keep the UMAP hyperparameters fixed (n_components=5n\_\text{components}{=}5,n_neighbors=10n\_\text{neighbors}{=}10, cosine metric, seed 42). We also keep HDBSCAN fixed atmin⁡_cluster_size=10\min\_\text{cluster\_size}{=}10withcluster_selection_method=leaf. This configuration produces 13 Reject-only clusters, with an unclustered fraction of40.5%40.5\%(288/711288/711).

We then map each Reject-only cluster to its nearest main-pattern centroid in the original3,0723{,}072-dimensionaltext-embedding-3-largespace (§6). All 13 clusters land within0.8970.897–0.9860.986cosine of their assigned primary. In total, 11 of the 15 main patterns receive at least one Reject-only cluster. No Reject-only cluster requires an out-of-taxonomy label.

Four main patterns receivenoReject-only mapping:Adapt by Conditioning, Not Retraining,Characterize a Limit, Then Surpass It,Design a Property-Targeting Pretext Objective, andEncode Structure by Construction. Three are small-sample patterns (n≤18n\leq 18each), and one is a constructively encoded pattern. The empirical reading is that Reject papers do not form a separate strategic space. They map into the existing 15-pattern taxonomy at high cosine.

Table8compares the resulting Reject distribution against the main-taxonomy Reject distribution.

Table 8:Reject distribution. TheReject-onlycolumn counts papers in each main pattern via Reject-only-clustering→\tonearest-main-centroid mapping; theReject-in-fullcolumn counts the same papers via their main-cluster primary inheritance.Δ\Deltais the divergence (≥0\geq 0: Reject-only assigns more papers here than the main taxonomy does).∑\sumReject-only=423=423(rest unclustered);∑\sumReject-in-full=367=367(rest fall in non-mapped clusters).∑|Δ|=198\sum|\Delta|=198.

10.2Interpretation

Ten of the 15 patterns yield comparable Reject counts across the two computations (|Δ|≤10|\Delta|\leq 10). The Reject pool’s strategic shape is therefore stable whether we cluster it alongside accepted papers or separately.

The notable divergences are concentrated in two directions. Reject-only clustering assigns substantially more papers toUnify Heterogeneous Inputs into One Space(+52+52) andAudit and Pivot an Assumption(+19+19) than the in-full mapping does. Conversely, it assigns fewer papers toEncode Structure by Construction(−23-23),Reframe as a Solvable Object(−15-15), andManufacture the Supervisory Signal(−10-10).

The pattern is consistent: when Reject papers cluster among themselves, they movetowardthe audit/unify lanes andawayfrom constructive lanes. The likely interpretation is that failed constructive moves can appear structurally like audit/unify moves once stripped of the construction that made them constructive.

Refer to caption Figure 14:Left: Reject-only clustering (n=711n{=}711, 13 clusters,40.5%40.5\%unclustered). Right: Reject-only-clustering→\tomain mapping vs. main taxonomy primary distribution over the same Reject papers. The two columns largely agree; the divergences are concentrated onUnify Heterogeneous Inputs into One SpaceandAudit and Pivot an Assumption(over-attracted) vs.Encode Structure by Construction(absent from the Reject-only clustering).Positive.The 15-pattern taxonomy covers the Reject population without modification. Every Reject-only cluster maps into an existing pattern, with no out-of-taxonomy bucket. Rejection is therefore not explained by use of a different kind of strategy.

Negative.The Reject-only divergence (∑|Δ|=198\sum|\Delta|=198) shows that the fine-grained surface form of rejected work drifts toward more general audit/unify patterns and away from constructive patterns. This has a direct consequence for the skill. When Phase 2.1 selects a constructive primary pattern, such asEncode Structure by Construction,Reframe as a Solvable Object, orManufacture the Supervisory Signal, Phase 3.2 should check whether the candidate’s executed surface form has collapsed toward the corresponding audit/unify lane.

11Ablation: Embedding Model and Abstraction Stage

Two design choices are central to the induced taxonomy. First, the production pipeline uses a general-purpose paraphrase-strong encoder rather than SPECTER2. Second, it embeds thedomain-agnosticrewrites rather than the four original base fields. We ablate both choices on the same 1,891-paper subset used by the production pipeline. UMAP parameters are held fixed, and HDBSCAN is swept overmin⁡_cluster_size∈{10,15,20,25}\min\_\text{cluster\_size}\in\{10,15,20,25\}. Figure15and Table9report the comparison.

Refer to caption Figure 15:Clustering quality across three embedding-input choices, asmin⁡_cluster_size\min\_\text{cluster\_size}varies. Solid line: silhouette. Dashed line: number of clusterskk. Production (left) is stable. OpenAI + base fields (middle) collapses tok=2k=2degenerate mega-clusters atmin⁡_cluster_size=15\min\_\text{cluster\_size}=15–2020. SPECTER2 (right) stays at lower silhouette across the sweep.Table 9:Clustering ablation. All three variants use UMAP (10-dim,n_neighbors=15n\_\text{neighbors}{=}15,min⁡_dist=0\min\_\text{dist}{=}0, cosine) and an HDBSCAN sweep overmin⁡_cluster_size\min\_\text{cluster\_size}. The two OpenAI rows differ only in which text is embedded. The production configuration is bolded.### 11.1Embedding model: OpenAI vs SPECTER2

With identical input text and identical downstream processing, OpenAItext-embedding-3-largeproduces more clusters than SPECTER2 atmin⁡_cluster_size=10\min\_\text{cluster\_size}=10(3131vs.2323). It also achieves a substantially higher silhouette (0.5840.584vs.0.4380.438, a+0.15+0.15gap). SPECTER2 degrades across the sweep, reaching0.3330.333atmin⁡_cluster_size=25\min\_\text{cluster\_size}=25. Its unclustered fraction remains in the4141–54%54\%range.

The qualitative difference matches the training objectives. SPECTER2 is trained on citation-informed contrastive pairs and is strong at placing topically adjacent papers near each other. Our target, however, is strategy similarity rather than topic similarity. Two papers should cluster together if they both “re-instantiate a proven mechanism at the appropriate domain-specific granularity”, even if they work on different modalities. The general-purpose paraphrase encoder better preserves this strategy-equivalence relation and produces a tighter strategy partition.

11.2Abstraction stage: silhouette is not enough

The base-fields ablation gives a more subtle result. Atmin⁡_cluster_size=10\min\_\text{cluster\_size}=10, its silhouette is nearly identical to production (0.5860.586vs.0.5840.584). Its unclustered fraction is also lower (35.7%35.7\%vs.47.7%47.7\%). On the raw clustering metric, the base-fields variant therefore looks slightly better.

The same configuration is unstable across the sweep. Atmin⁡_cluster_size=15\min\_\text{cluster\_size}=15–2020, it collapses tok=2k=2mega-clusters with silhouette0.8840.884and0%0\%unclustered. Inspection shows that these are not strategy clusters. One aggregates papers about engineering choices in deep learning. The other aggregates papers about formal or analytic ML. They are macro-topic clusters, not reasoning-move clusters.

This degenerate collapse indicates that topic cohesion has overpowered strategy cohesion. The base fields contain domain-specific vocabulary, including architecture names, dataset names, and field-specific mathematical objects. At coarser granularities, topic proximity then dominates the similarity structure. Two diffusion papers can cluster together because both are about diffusion, regardless of what they actuallydowith diffusion.

The Stage-2 rewrite moves abstraction into the text before embedding. It removes topic anchors and leaves strategy-level signal. The production configuration remains well behaved across coarser settings:kkstays between 12 and 19, and silhouette stays in the0.530.53–0.580.58range. It does not collapse into mega-clusters.

Ablation takeaways.

We select the production configuration for construct validity and stability, not because it maximizes silhouette. The two ablations point in the same direction. The general-purpose paraphrase encoder beats SPECTER2 (Δ\Deltasilhouette=+0.15=+0.15) because SPECTER2’s citation prior anchors on topic rather than on the strategy similarity needed here. The domain-agnostic rewrites tie the base fields on silhouette at the chosen setting, but avoid collapse at adjacent settings.

The base-field variant pays for cohesion in the wrong currency: it can achieve high silhouette by forming topic clusters. The Stage-2 rewrite removes those topic anchors and preserves robustness across the granularity sweep without giving up cluster quality. Production is therefore the only configuration whose clusters support the ideation-pattern taxonomy of §6while remaining stable across the HDBSCAN sweep.

12Discussion

12.1Empirical takeaways

The analyses above support four claims. A fifth claim is supported only at the idea-stage endpoint and remains open for human validation.

(i) The strategy space is shared, not stratified.Re-clustering the Reject pool in isolation (§10) reproduced the same 15-pattern vocabulary rather than a separate space. The signal separating accepted from rejected work therefore lies less in strategy choice than in execution. The per-pattern success conditions and failure modes carry that distinction. The one systematic divergence is that isolated Reject clusters drift toward audit/unify lanes and away from constructive lanes. This suggests that failed constructive moves can look like audit/unify moves once stripped of the construction that defines them.

(ii) The Oral-vs-Reject axis is flat at the main-pattern level; PC-vs-community is the stronger discriminator.Across granularities (§7), “which patterns” contributes only a small main effect to acceptance. “How well they compose” contributes a larger one. This is the gap targeted by the skill’s audit step. The nearly orthogonalΔOH\Delta_{OH}axis, contrasting PC-favored and community-favored patterns, is wider and acts as a distinct second signal. The audit surfaces this signal independently of acceptance prediction. This is the softer of the two axes, however: HC papers come from a separately curated high-citation set rather than from OpenReview decisions, so we reportΔOH\Delta_{OH}only atnHC≥5n_{HC}\geq 5and discount it elsewhere, and reconstructing HC from decision-aligned citation percentiles would let it carry the same weight as Oral-vs-Reject.

(iii) Methodologies are broad in coverage but concentrated in effect.Most patterns reach the majority of the 28 induced domains (§8.3). Yet the same pattern lands at very different Oral rates across domains (Fig.10b). The cleanest cross-domain Oral signal isCharacterize a Limit, Then Surpass It. For the skill, this means domain-cell rates should be surfaced as audit context, not loaded as generation priors.

(iv) Composition is the norm, withk=2k{=}2as the empirical mode.Paper-level multi-label tagging (§6.5) shows that compositionsizedoes not separate acceptance classes:k¯≈2.3\bar{k}\approx 2.3in all three. Compositioncontentmatters more. The skill therefore composes one to three patterns selected by bottleneck fit, defaulting to two, rather than ranking patterns by acceptance priors.

(v) Partially closed: execution validity at the skill endpoint.The construct-validity analysis in §11shows that the cluster–pattern–card chain is internally coherent and stable across the HDBSCAN sweep. The reject-only analysis shows that every Reject-only cluster maps into the taxonomy. Section14adds an endpoint evaluation: when invoked at inference time and judged blind by two automated skills, IdeaSpark ideas receive the highest quality scores among four sources. What remains open is human-grounded validation against expert reviewers, implementation outcomes, and acceptance outcomes.

From taxonomy to system.

The taxonomy is the prerequisite for the skill system, but it is also a stand-alone deliverable. It has two immediate uses. A researcher can compare a candidate idea against the 15 ideation patterns and ask which success conditions the draft satisfies. A reviewer can also name a specific failure mode, for example “combination indistinguishable from sum of parts.” The skill system in §13operationalizes both uses inside an LLM scaffold.

13IdeaSpark Design

We buildIdeaSparkto turn the empirical substrate in Sections6–10into a runnable research-ideation workflow. This section describes its runtime boundaries, evidence flow, phase contracts, audit logic, output surface, and mechanical validators. Endpoint idea quality is evaluated separately by automated judges in Section14. Human-rated quality and held-out rediscovery remain future work.

13.1Positioning and two-tier architecture

The skill converts an under-specified research direction into one reviewer-facing proposal. A successful run returns an artifact bundle centered on a compact idea card. The card contains the motivation, method flow, core claim, falsification prediction, and feasibility validation. One user input produces one of three outcomes: an idea-card bundle, ado_not_generatediagnosis when the direction cannot be grounded, or aphase_3_failedreport when the audit abandons a candidate. The skill never asks the user mid-flow. Missing context is inferred when reasonable; unrecoverable ambiguity becomes an explicit stop condition.

The skill is not an ideation-pattern recommender, experiment planner, or paper writer. It does not propose datasets, baselines, ablation matrices, expected figures, or a calendar plan. Those choices depend on user context the skill cannot access. It produces an idea plus an honest feasibility judgment; empirical engineering remains the user’s responsibility.

IdeaSpark is implemented as a two-tier skill. The runtime tier contains compact instructions, orchestration logic, retrieval hooks, phase prompts, rendering scripts, and deterministic validators. The evidence tier contains the 15 ideation-pattern cards, 31 sub-pattern cards, domain-by-pattern matrix, saturation records, and corpus-derived failure-mode inventory. Evidence is read progressively. Phase 2 reads only the pattern definitions and sub-pattern tactics needed to instantiate a candidate. Phase 3 reads failure modes, anti-pattern context, and collision evidence for audit. Phase 4 reads the audit findings to explain the final card. Raw counts and confidence flags remain analytical metadata for readers. They are not used as generation priors.

13.2Design principles

The skill’s core stance is that a publishable research idea isconstructed, not invented from scratch. The reasoning chain has three steps. First, read recent literature in the user’s area and identify a structural gap no retrieved paper closes: thebottleneck. Second, use the induced ideation-pattern vocabulary to ask what kind of research move could close that gap. Third, instantiate the chosen move as a concrete mechanism. The 15-pattern and 31-sub-pattern substrate supplies the vocabulary for the second step. It keeps generation at the level of pattern-applied-to-gap rather than unconstrained brainstorming.

Principle 1: Literature first, vocabulary second, idea third.

Bottleneck identification must be grounded in retrieved papers, not model recall. Pattern selection reads the shape of the unresolved gap and asks which operational move would close it. Idea instantiation then uses the selected sub-pattern as a tactical guide. It applies the abstract move to the specific gap rather than copying a paper-specific template.

Principle 2: Substantive contribution over methodological labels.

A candidate cannot defend itself as “we use ideation pattern X while prior work used pattern Y.” Method choice is process. The contribution is what the candidatederives, constructs, measures, or tightensthat prior work did not. This is enforced in the differentiation deltas and downstream audit checks. The 15 patterns are diagnostic vocabulary, never the contribution claim itself.

Principle 3: Generation freedom, audit anchoring.

Earlier iterations placed too many corpus-derived constraints inside generation. That biased candidates toward frequent corpus moves and away from the user’s bottleneck. The current design keeps positive pattern vocabulary in Phase 2, but moves negative and frequency-based signals to audit and explanation. Phase 3 checks whether the candidate survives known failure modes. It does not reject a candidate merely because its pattern is common.

These choices follow from the empirical substrate. Rejected papers often use the same strategy operators as accepted papers, so the taxonomy serves as thinking and audit vocabulary rather than as a hard generator filter. Becausek=2k=2is the modal paper-level composition, with a33.6%33.6\%tail atk≥3k\geq 3, Phase 2 allows one to three gap-closure entries. Because some two-way combinations are reject-enriched unless the downstream artifact is substantively delivered, Phase 3 includes an anti-pattern substantive-verification check. Because acceptance is domain-conditional, saturation and domain-landscape information are surfaced as audit and explanation context rather than as recommendation priors.

13.3Workflow and phase contracts

The skill treats the 15 induced ideation patterns as diagnostic vocabulary, not as labels a candidate must classify into. The advance path combines model-reasoning phases with deterministic retrieval, rendering, and validation steps. The revise path adds one bounded repair step. That repair is constrained by audit targets and by byte-level preservation of the candidate’s kill-switch commitments.

13.3.1Phase 0: Literature grounding

Phase 0 builds the broad literature-grounding bundle. It is the only phase that retrievesarea-levelcontext. The later Phase 3.1 retrieval is narrower: it checks for mechanism-level collision with the generated candidate.

The system turns the user direction into several search queries, checks which connectors are available, and queries four sources. Each source is used for the role it covers best. arXiv supplies recent preprints. OpenReview supplies forward-looking, in-review signal. OpenAlex supplies the broad published-paper graph. Semantic Scholar supplies CS-focused metadata and short summaries. arXiv and OpenReview cover the recent 0–6 month window. OpenAlex and Semantic Scholar cover the 6–24 month window.

This split is deliberate. No single source provides all four strengths: preprint recency, in-review signal, breadth, and clean cross-identifiers for deduplication. Each source is queried where it is strongest. The two time windows are kept non-overlapping. A paper that appears both as a recent preprint and as an in-review submission is counted once, avoiding inflated pattern-saturation counts in Phase 1.

The retrieved papers are then deduplicated across sources and lightly tagged with ideation patterns for later analysis. Immediately after the literature table is written, Phase 0+ fetches a small full-text cache for the most relevant references. Phase 1 hard-gates on this cache, so the pipeline cannot silently degrade to abstract-only bottleneck reasoning.

Two refinements harden the retrieval.For recall, keyword matching is complemented by a meaning-based pass that finds conceptually close papers with little surface-word overlap. This pass is gated to the home field of the keyword results, so broader matching does not pull in off-field papers. Terminology drift would otherwise hide close competitors.For reading fidelity, the system fetches open-access full text where available. Bottleneck and differentiation judgments can then rest on method sections, not abstracts alone.

The overall goal is that every downstream check (bottleneck, citation, collision) traces to a retrieved record rather than to model memory.

13.3.2Phase 1: Bottleneck identification

Phase 1 first checks that Phase 0 produced a valid grounding bundle and that Phase 0+ produced the full-text cache. It then reads the user direction, retrieved records, abstracts, and cached full-text snippets. Its main output is one literature-grounded bottleneck: a concrete structural gap, not a topic label. It also names the closest adjacent papers, explains what each still leaves open, lists gaps Phase 0 did not close, and summarizes pattern usage across on-topic papers. These usage bands are descriptive, not temporal. A pattern issaturatedwhen it dominates the neighborhood,untestedwhen it appears at most once, andmid-frequencyotherwise. The summary is carried forward as audit and explanation context only.

To compare gaps on a common footing, Phase 1 arranges the neighborhood into amethod-lineage tree. Each retrieved method is a node linked to the earlier method it refines or replaces. The tree runs from foundational ancestors to current leaves. When the recent retrieval window misses an older ancestor, the model may add it as anawareness-onlynode, explicitly marked non-citable. This division is deliberate. The window is recent because frontier grounding matters most there, and model memory is least reliable there. Older foundational methods are more stable. Because awareness-only nodes make no novelty claim, they serve as regression guards rather than evidence. An anti-fabrication rule keeps the tree shallow on genuinely young lines instead of padding it with invented ancestors.

The tree exposes three things a flat abstract list cannot.Additivegaps sit at a leaf: an unmet need no current method achieves.Subtractivegaps are read from a shared ancestor: the load-bearing assumption every leaf inherits. These gaps matter because strong papers often remove an assumed-necessary mechanism rather than add another component. Third, the tree blocksregression. A proposed fix that an ancestor already implemented is treated as superseded, not novel. Throughout, the reading is evidence-gated. A subtractive opening may name only an assumption the retrieved methods actually show.

The route out of Phase 1 is binary: proceed, or stop with a diagnostic. Phase 1 never asks the user for missing information. It infers reasonable defaults, for example a single-researcher cloud compute budget when none is stated. It stops only when the direction is too broad, has no literature anchor, or the retrieved pool is too sparse to ground a bottleneck. The two structural routes cover an estimated 75–85% of Oral types in the corpus (theory-plus-reframing, scaling-law, surgical-fix-plus-theory, evaluation-validity-audit, and most empirical-reveal Orals); benchmark-construction and pure system or infrastructure directions are not yet separate triggers and currently resolve through one of the two routes, with the subtype named only as diagnostic explanation. Adding dedicated triggers for them is a localized extension that would raise coverage without changing the architecture.

13.3.3Phase 2: Pattern-guided ideation

Phase 2 separates pattern selection from candidate generation. This separation is important: the first step asks which corpus-derived research move can close the diagnosed gap, while the second step instantiates that move as a concrete candidate.

*Pattern selection.*The system chooses one anchor gap from Phase 1 and keeps only the sibling gaps that can plausibly form a single paper with it. For each retained gap, it reads the 15 main ideation-pattern cards and selects the pattern whose operational signature structurally closes that gap. Pattern frequency and saturation are recorded as audit context, but they do not act as generation priors.

*Candidate generation.*For each selected main pattern, the system chooses one sub-pattern and reads its tactical panels as a guide. The output is not just a title or theme. It contains the core mechanism, per-gap closure rationale, differentiation from the closest adjacent work, a compute budget, and a falsification prediction naming what should change if the mechanism is real.

Finally, the candidate passes through a deterministic sub-pattern citation gate. The gate verifies that every cited sub-pattern exists under the stated parent pattern before any expensive collision retrieval or audit is run.

13.3.4Phase 3: Quality gauntlet

Phase 3 has three steps: a focused literature search, a corpus-anchored audit, and an optional revision.

First, the system re-queries the literature sources using terms drawn from the candidate’s mechanism and claim. This search is separate from Phase 0 because it looks forrecent mechanism overlap, not broad area context.

Second, the audit reads the candidate, selected gaps, relevant sub-pattern cards, the Phase 0 evidence pool, the new collision hits, and the anti-pattern inventory. It runs four checks: gap-closure reject lessons, recipe application, anti-pattern substantive verification, and paper-pointed threat. It returns one of three verdicts, advance, revise, or abandon, plus concrete revision targets. The audit only judges; it does not edit the candidate.

Third, and only when revision is possible, a separate step applies the named edits or swaps in a same-parent sub-pattern. It does not re-judge the verdict, change the selected gaps, or touch the kill-switch commitments. Splitting judging from fixing keeps attack-finding and attack-resolution separate, and keeps every failure mode tied to corpus evidence.

Phase 3 audit details.

The audit is a single judgment step. It performs four corpus-anchored checks, applies a two-layer verdict, and emits revision targets when a bounded repair is possible. Each check requires concrete evidence from the retrieved literature, sub-pattern cards, or anti-pattern inventory. This makes the audit inspectable and resistant to rubber-stamping. The audit judges whether the candidate survives known failure modes. It is not a guarantee of novelty, acceptance, or endpoint idea quality.

Check 1: gap-closure-scoped reject scan.

What it catches: candidates that look novel from the outside but fall into corpus-documented reject patterns specific to the chosen tactic. For each proposed gap closure, the audit reads the relevant sub-pattern’s failure-mode panel and rejected-paper lessons. It evaluates whether the candidate clearly avoids the lesson, clearly matches it, or sits in a borderline state, and it must cite the candidate component that triggers the concern.

Check 2: recipe application.

What it catches: candidates that cite a real sub-pattern but only execute its parent pattern’s generic idea rather than the cluster’s distinctive move, the leading cause of incremental output. Against the cited sub-pattern card’s signature-move panel, the audit asks whether the core mechanism actually performs that move or merely gestures at the parent; an absent move marks the candidatebypassed. This is the semantic backstop the deterministic citation gate (§13.3) cannot provide: the gate proves only that the citation resolves to a real cluster under the right parent, never that the mechanism enacts the cluster’s tactic.

Check 3: anti-pattern substantive verification.

What it catches: keyword-stuffed mitigations, candidates whose core mechanism contains the right mitigation phrases without producing the artifact those phrases name. If the candidate uses one of the reject-enriched two-pattern compositions documented in Section6.3, the audit verifies that the required mitigation is substantively delivered in the mechanism, rather than through wording alone in the candidate.

Check 4: paper-pointed threat.

What it catches: candidates whose claim is already subsumed by a specific recent paper, whether retrieved by Phase 0’s broad-domain queries or by Phase 3’s mechanism-specific collision retrieval. If a paper that subsumes or competes with the candidate’s claim is found, the audit emits the threat paper, a one-paragraph subsumption argument citing the threat’s specific result, and a note on what would need to change. If no paper-pointed threat exists after honest search, the absence is a legitimate clearance signal; fabricating a generic threat to fill the audit is forbidden.

Why saturation is not gated here.

An earlier design added a fifth saturation-defense check. We removed it because it consulted no retrieved related work. It was an advisory prior, not a corpus-anchored fact, and therefore did not belong beside the four hard-floor checks above. Pattern saturation is still computed in Phase 1 and recorded in Phase 2. It now surfaces downstream as a Phase 4 reviewer concern rather than gating the audit. This keeps every Phase 3 check anchored on retrieved evidence.

Two-layer verdict.

The verdict has two layers.Layer 1is a hard floor the model cannot override. It abandons the candidate when a reject lesson clearly matches, when a reject-enriched composition has no insertable mitigation, or when a recent paper has exact-mechanism overlap.Layer 2is model judgment within the safe zone Layer 1 leaves. It chooses advance or revise based on whether borderline findings touch load-bearing structural properties. A recipe-application bypass always routes to revise: either swap to the sibling sub-pattern whose move the mechanism actually performs, or rework the mechanism to perform the cited move. If neither fits under the same parent, the run returns to Phase 2 regeneration rather than patching in place. The verdict must cite specific check findings; “all checks pass” without naming one is a process error.

13.3.5Phase 4: Expansion, implementability audit, rendering, and validation

Phase 4 turns the candidate into the final idea-card artifact bundle. It expands the canonical candidate, runs a default-on implementability audit, renders side artifacts, and then runs deterministic validators. The expansion derives the motivation, method flow, feasibility, reviewer concerns, literature differentiation, and domain-landscape explanation from the candidate plus the Phase 3 audit report.

The implementability audit is a single bounded LLM call. For each method step, it stateshowto build that step and flags anything left underspecified. It is written to a separate file. By construction, that file cannot carry the protected falsification and compute-budget commitments. The audit therefore adds buildability detail without changing the claim. If a run skips it, the implementability validators are not enabled.

Rendering is deterministic templating. The renderer builds the Markdown and LaTeX artifacts, and compiles a PDF when a local TeX engine is available. Because the templates are fixed, validators can compare protected commitments against the pre-render candidate and check that required sections are present before the user sees the card.

Phase 4 validation.

A suite of deterministic validators runs around the Phase 4 render path to check contracts asserted by earlier phases. They are mechanical. They detect violations through exact-text comparison, section presence, list length, schema coverage, readability checks, and citation consistency, not model judgment.

Kill-switch integrity (hard fail).

The candidate’s falsification prediction and compute budget must be preserved from initial candidate generation through final rendering. The validation chain depends on the verdict path: direct expansion when the audit advances, or revised candidate followed by expansion when the audit requests a bounded fix. The Phase 3 revision step is forbidden from modifying these two commitments. The validator catches any drift mechanically rather than relying on prompt compliance.

Expansion completeness (hard fail).

Phase 4 expansion must contain the structural sections the idea-card rendering depends on: motivation with multiple prior-work stopping points, method-flow steps, feasibility validation with component-level and overall verdicts, non-empty proposal summary, core claim, and sub-claims. The check is section presence and list length. Absence is a hard fail because missing sections would silently render as blank content in the final artifacts.

Additional contract checks.

Three further validators harden the pipeline contracts.subpattern_citation_consistencyruns immediately after Phase 2.2 and before any Phase 3 work. It checks that each cited sub-pattern resolves to a real cluster under the stated parent pattern.implementability_completenessconfirms that the audit covers every method step when Phase 4.1.5 ran.implementability_readabilityflags known readability regressions in the standard-register buildability text.

What the validators do not check.

They do not enforce semantic correctness inside a section, they do not check whether the candidate’s claim is actually novel, and they do not enforce cross-run determinism. The phase calls are LLM-reasoning steps with non-zero variance, so the same query can yield different but coherent candidates across runs; this is expected for an inference-time skill rather than a defect, and reporting distributions over repeated runs or seed-ensembled selection would make the variance explicit (§15).

13.4Output surface

The user-facing deliverable is deliberately slimmed to the three things a reader first judges: title, motivation, and method. An earlier, richer card printed the hook, claims, rebuttals, and provenance together. That version buried the idea under its own audit log. The current card is emitted in English and Chinese. Heavier material remains in the artifact bundle: the hook, problem-to-outcome summary, explicit claims, feasibility verdicts, literature-differentiation deltas, reviewer concerns and responses, and the literature-breakdown note. This lets an inspector trace why the candidate survived without turning the card into an audit dump.

A condensed generated cardheld-out ICLR 2026 seed: RL for long-context reasoningTitle.Phase-Stratified Credit: Verifier-Free Per-Phase Advantage for Long-Context Reasoning RL (method:PS-GRPO).Motivation (the bottleneck).GRPO collapses each long rollout into one group-normalized score and broadcasts it uniformly over every token. But only a few phases—where the model traces the key chain and grounds a fact—decide correctness, while most tokens scan distractor text. As context grows toward 128K, the decisive phases stay a near-constant budget while the filler grows, so the useful signal per decisive token shrinks like1/length1/\text{length}: the method collapses exactly in the long-context regime it was built for.Method (step 1 of 4).Build the strongest equal-compute GRPO baseline; record each rollout’s group-normalized advantageAg=(Rg−R¯q)/σqA_{g}=(R_{g}-\bar{R}_{q})/\sigma_{q}, broadcast uniformly to allTgT_{g}tokens, and its accuracy out to 128K—the reference curve.[Three further steps harvest a verifier-free per-phase credit density from the in-group correct/incorrect ordering; full card in App.A.]Falsification (kill-switch).Load-bearing variable:CC, the fraction of advantage magnitude on the decisive phases. Negative control: permute the phase labels so thesameCClands on random spans—if 128K accuracy survives, the gain came from generic variance reduction, not from localizing credit to the steps that determine correctness.(Full prediction with positive control in App.B.)

The falsification prediction is a single paragraph naming the minimal experiment and the metrics that should move. It gives qualitative direction but no pre-committed numerical thresholds. When the candidate makes a mechanism claim, the schema requires oneload-bearing variable: the quantity whose behavior carries the claim. It also requires a negative-control intervention whose predicted effect lands on the downstream task outcome, not on the variable’s own definitional transform. This blocks a recurring self-confirmation failure: “intervene onXXso thatXXbecomes zero.” That tests a definition, not a mechanism. Tying the control to an independent downstream effect makes the claim falsifiable. The distinguisher is required for mechanism candidates and optional for pure “does this work?” candidates. The output excludes experiment matrices, ablation plans, baseline tables, expected figures, and calendar projections. These belong to the user’s experimental engineering, not to the idea the skill produces.

13.5Faithfulness: resisting hallucination

The standing objection to LLM ideation is fabrication: invented citations, papers, numbers, or method detail asserted with false confidence. IdeaSpark distributes its defenses across the phases in §13.3. The enforced property is simple: every load-bearing claim either traces to a retrieved record or is explicitly marked as not doing so.

Grounding over memory.

Retrieval is not left to the model’s discretion. Phase 0 follows a fixed orchestration path rather than an in-context tool the model could replace with recall. Phase 1 refuses to start until the grounding bundle and full-text cache are present, so a bottleneck is not written from abstracts or memory alone. The method-lineage tree extends the same discipline backward in time. Ancestors supplied from model memory enter only as non-citable, awareness-only nodes. They make no novelty claim. A subtractive gap may name only an assumption the retrieved methods actually exhibit.

Citation faithfulness.

A deterministic gate checks every sub-pattern citation when the candidate is written. It halts the run on a hallucinated parent, a mis-filed sub-pattern, or an invented name before any costly Phase 3 retrieval is spent. The pattern cards carry paper-agnostic lessons with no per-paper identifiers. This removes, rather than merely polices, the surface on which apaper_idcould be hallucinated. A cleared audit may legitimately report no threat found; fabricating a generic threat to fill the slot is forbidden.

Claim faithfulness.

Two guards bind the candidate’s own assertions. Quantitative claims about predicted behavior, such as a rate, exponent, or scaling constant, must be cross-checked against same-setting papers in the full-text cache. The claim must then either align to that value or state the regime difference. This closes the recurring failure of pasting a number across incompatible regimes. The two kill-switch fields, falsification prediction and compute budget, are held byte-identical from generation through rendering by a hard validator. The model cannot silently soften the claim that makes the idea testable.

Honest abstention.

When the evidence does not support a proposal, the run stops with ado_not_generateorphase_3_failedreport rather than producing one anyway. When the implementability audit reaches a load-bearing detail it cannot supply without guessing, it records that detail as an open author decision. An honest gap is treated as more valuable than fluent filler.

Together, these mechanisms turn “do not hallucinate” from an instruction the model may ignore into a property enforced by retrieval gates, deterministic validators, and provenance discipline.

14Evaluation: Generated-Idea Quality and Novelty

This section reports anautomatedendpoint evaluation. It compares IdeaSpark against baselines on two axes,qualityandnovelty, using two auxiliary judging skills rather than human reviewers. The headline result is twofold. First, at matched length and format, judged blind, IdeaSpark’s idea cards receive the highest quality scores by a wide margin while remaining competitively novel. Second, novelty alone is misleading. The evaluation exposes a “novel-but-empty” failure mode, in which an idea appears highly novel because it is too vague to collide with prior art. We therefore treat the quality–novelty plane as the primary readout. As discussed in §15, both judges are LLM-based skills, so this is an automated-judge study, not a human acceptance study.

14.1Systems compared

We compare four idea sources on a shared set of problem directions (Table10). The comparison points form a ladder of controls around IdeaSpark.IdeaSparkis the system of this report.Opus-self-genis a generic ideation skill automatically authored by an Opus skill-creator, with no corpus grounding. Comparing it with IdeaSpark isolates the effect ofourcorpus-grounded skill from the effect of merely havingsomestructured skill.Opus-4.8 (bare)is a single prompt with no skill and no retrieval. IdeaSpark, Opus-self-gen, and Opus-4.8 (bare) all run on the same Opus 4.8-high backbone. Together they form a same-backbone ladder: bare model, generic auto-built skill, and corpus-grounded skill.GPT-5.5 (bare)is a second bare baseline from a different model family, included to separate backbone-specific effects from skill effects.

Table 10:The four idea sources compared. “Skill” = produced by a structured multi-phase skill; “Retrieval” = the generator read live literature while producing the idea.

14.2Problem seeds: 100 method-agnostic directions from ICLR 2026

Each system is asked to produce one idea perproblem seed. We construct 100 seeds from ICLR 2026 Oral acceptances retrieved through the OpenReview API. ICLR 2026 is chosen deliberately: it post-dates the induction corpus (2021–2025) and the assumed training data of the generators and judges. The seeds therefore act as a forward, held-out test rather than a rediscovery exercise.

To avoid leaking each paper’s own solution into the prompt, each seed is amethod-agnostic rewritegenerated from the title alone. The rewrite keeps the domain object and task or capability, but strips method suffixes and any abstract-derived bottleneck diagnosis. For example, a paper title containing a specific mechanism for physical realism in video becomes the seed “a novel ML research idea about physical realism in text-to-video models.” One seed is generated per paper and screened for leakage. The resulting seeds namewhat problemto work on without hintinghow.

14.3Output normalization (controlling confounds)

Quality and novelty judgments are sensitive to surface confounds, so all four sources are normalized to the same deliverable contract before judging. Every idea is converted to a three-section Markdown document: Title, Motivation, and Method. Motivation is held to[259,330][259,330]words and Method to[449,866][449,866]words, matching the length distribution of the source ICLR Oral papers. This removes the length and verbosity confound that otherwise rewards longer write-ups.

Every Method must also include formal equations with per-equation interpretation. This requirement matches the style of the source papers and puts vague prose and precise formalization on the same footing. The goal is to make judges score substance rather than surface technicality. We also strip protected “author-decision” annotations from IdeaSpark cards before scoring. These annotations are honest flags of unresolved design choices, but no other source emits them. Leaving them in would impose an asymmetric “honesty tax”; a controlled comparison in §14.7quantifies that effect.

14.4Two automated judges

Quality: theidea-qualityskill.

Quality is scored by a self-contained skill that judges an idea from first principles on three axes, each assessable without experiments. The axes are*(A) problem position*, whether the attacked gap is real, important, and non-obvious rather than a soft target;(B) method quality, decomposed into depth, soundness, and feasibility, with depth dominating; and*(C) problem-fit*, whether the method actually closes the stated gap. The skill requires a quoted phrase of evidence for every axis score. It judges substance rather than length or fluency, and applies an A/C gate so that a strong method on a trivial problem cannot rank highly. For comparison, it runs inlistwisemode, ranking several ideas for the same seed against one another while blind to source identity. Relative judgments are less noisy than absolute scores.

Novelty: thescoop-checkskill.

Novelty is scored by a prior-art collision skill. For each idea, the skill performs aliveliterature search, deep-reads the closest candidates, and matches the proposed contribution against each prior work on four axes: problem framing, core mechanism, key insight, and application domain. The number of matching axes maps to a five-level scale,level=5−(axes matching)\text{level}=5-(\text{axes matching}). ThusLevel 5 = no overlap (most novel)andLevel 1 = all four axes match (fully scooped). The verdict for an idea is the worst case, or minimum level, over its closest retrieved prior works. We take the worst case rather than the mean because one sufficiently close prior work can scoop an idea. Averaging would let unrelated retrieved papers dilute a decisive collision and inflate novelty. Higher is more novel, so novelty and quality point in the same direction.

14.5Protocol and metrics

Both axes use a blind, repeated protocol over all 100 seeds. For each seed, the four systems’ ideas are copied to neutral filenames under a fresh random label permutation. The shuffle differs by seed and by round to cancel position bias. Source identity is withheld from the judge.Qualityis a listwise rank: rank 1 scores44, rank 2 scores33, rank 3 scores22, and rank 4 scores11.Noveltyis the Level∈{1,…,5}\in\{1,\dots,5\}defined above.

Each seed is judged inthree independent roundswith fresh judge instances. For quality, we report each system’s mean over the 100 seeds after averaging the three rounds per seed. For novelty, we report the mean level over all 300 judgments. Quality and novelty are run as separate passes.

14.6Results

Tables11and12report the two axes. Figure1places the four systems in the quality–novelty plane. Figure16shows the full distribution of novelty levels.

Table 11:Quality(idea-qualitylistwise rank score,11–44, higher is better). Each of the 100 seeds is judged in 3 blind rounds; mean and std are taken over the 100 seeds, each averaged across its 3 rounds. “Wins” is the number of seeds the system ranked first on its 3-round average (out of 100).Table 12:Novelty(scoop-checklevel,11–55, higher is more novel). Each of the 100 seeds is judged in 3 blind rounds; mean, std, and the level distribution are taken over all 300 judgments (100 seeds×\times3 rounds). Right block: how the judgments distribute over levels L1 (fully scooped) to L5 (no overlap). Refer to caption Figure 16:Distribution of novelty levels per system (L1 = fully scooped→\rightarrowL5 = no overlap). The skill generators concentrate at L3 (medium overlap: shared framing/domain, distinct mechanism); the bare GPT-5.5 baseline piles up at L4, where its vagueness evades collision.

14.7Analysis

IdeaSpark wins on quality.

IdeaSpark attains the highest quality score (3.87/43.87/4) and ranks first on8888of the100100seeds by 3-round average. No other system exceeds2.62.6. The generic auto-authored skill, Opus-self-gen, scores2.572.57, essentially tied with the bare Opus-4.8 baseline at2.562.56. A structured skill alone therefore does not lift idea quality.

The same-backbone ladder isolates the effect of IdeaSpark’s specific design. IdeaSpark, Opus-self-gen, and Opus-4.8 (bare) share one backbone (§14.1). Opus-self-gen also has live retrieval. The observed gain is therefore not explained by model family or by retrieval alone. The lift comes from IdeaSpark’s corpus-grounded pattern cards together with its multi-phase, audit-anchored workflow.

Single-axis novelty is misleading.

On novelty alone, theworstsystem wins. The bare GPT-5.5 baseline scores highest on novelty (3.733.73, with214/300214/300judgments at L4), but is last on quality (1.001.00, zero wins). The judges’ rationales explain the mismatch. This baseline emits a near-identical topic-agnostic template, a generic “diagnostic-heads + contrastive-pairs + uncertainty-routing” scaffold that occasionally drifts off topic. Because the idea is vague, it hasno precise prior art to collide with, so it scores as “novel.”

This is the “novel-but-empty” regime. It is invisible to either axis alone. Only the quality×\timesnovelty plane (Figure1) separates genuine novelty, high on both axes, from vacuous novelty, high on novelty but low on quality. The two skill-based systems instead sit in the high-quality region while remaining at medium overlap (L3). Their ideas share problem framing and domain with the closest prior work but contribute a distinct mechanism, an honest and defensible novelty profile.

Calibration sub-study (honesty annotations).

On a55-seed pilot, leaving IdeaSpark’s self-flagged author-decision annotations in the card cost roughly one rank position, dropping it behind the bare Opus-4.8 baseline. Removing them restored its lead. These annotations are author-facing meta-notes rather than part of the idea, and no other source emits them. This motivates the normalization in §14.3: strip the annotations before scoring.

15Limitations

Each limitation below is a bounded property of the current study paired with a concrete next step. None changes the claims we make, which remain scoped to the idea stage. They mark where additional evidence would extend those claims.

Coverage is bounded by three ML conferences, which biases the induced distribution.

The corpus is built entirely from ICLR, ICML, and NeurIPS outcomes and their public reviews, and the evaluation seeds are all ICLR 2026 Orals. The induced domains, the ideation patterns, and the implicit sense of what counts as a strong direction therefore track the distribution of mainstream ML-conference research rather than science in general. Fields and venues under-represented in these proceedings are correspondingly under-covered, so generated ideas can inherit this bias and need not transfer to other communities. We write the patterns to be domain-agnostic to limit topic leakage, and the novelty pass checks overlap with live literature, but we do not enforce as-of backdating of the retrieval window. Cross-venue and cross-discipline corpora, backdated retrieval for a leakage-free forward-prediction variant, and cross-backbone judges are the direct ways to test and widen external validity.

The endpoint evaluation is automated, with a human study as the next stage.

Both axes are scored by LLM-based skills, so the results measure agreement with structured automated reviewers rather than human program-committee judgment, and the judges may share blind spots with the generators. An automated study is the appropriate first stage at the idea level; a blind expert study is the planned next step before any acceptance-level claim.

The judge metrics carry two known biases.

First,idea-qualityrewards a novelmechanism, so benchmark, systems, and pure-measurement contributions score lower on its method-depth axis by construction; we therefore read quality asrelativeamong generation systems, not as an absolute measure of research value. Second, novelty is meaningful only jointly with quality, since a sufficiently generic idea can evade prior-art collision and score as novel (the GPT-5.5 case), which is why we never rank on novelty alone but report it as one axis of a quality–novelty plane. Rubrics for resource and systems contributions, and a specificity-aware novelty metric that penalizes under-committed ideas, would reduce both biases.

16Artifact Cards

16.1Data card

The corpus (Section3) uses public metadata and public review material. Author identities are retained as paper metadata but are not used as model features for pattern induction or recommendation.

16.2Model and backend card

The current pipeline uses LLMs for extraction, taxonomy induction, card generation, and several skill phases, plus an embedding model for clustering. The principal models are Claude Sonnet 4.6 for signature extraction and paper-level tagging; Claude Opus 4.7 for taxonomy / domain induction and pattern-card generation; OpenAItext-embedding-3-largefor strategy-signature clustering; and Claude Opus 4.8 high for the skill runtime and endpoint evaluation in §14. GPT-5.5 is used only as the cross-family baseline. These choices are backend-specific, and cross-backend quality and format-validity measurement remains future work.

16.3Skill card

IdeaSpark is appropriate for literature-grounded ML research directions where the user can state a domain, adjacent literature, constraints, and a desired contribution type. It is not appropriate for confidential literature without connector permission, broad agenda-setting prompts with no adjacent papers, pure benchmark construction, complete paper writing, automated submission production, or claims requiring private peer-review data. It returns a structured idea card, ado_not_generatediagnosis, or aphase_3_failedreport, but it cannot guarantee novelty, acceptance, feasibility, or reviewer agreement. Users remain responsible for prior-art review, experimental design, feasibility assessment, ethical review, and downstream claims.

17Responsible Use

Research-ideation systems reduce the cost of producing plausible-looking proposals, which also increases misuse risks: low-quality proposal generation, reviewer burden, shallow arXiv submissions, and overconfident novelty claims. IdeaSpark is therefore scoped as an ideation scaffold, not a paper-writing agent. It outputs one idea card rather than a manuscript, includes failure paths, audits candidates against corpus-derived reject lessons and prior-art threats, and keeps falsification and feasibility checks visible.

Because the method draws onrejectedsubmissions and reviews, we treat that signal as sensitive. Rejected papers are used only as aggregate, pattern-level evidence about how strategic moves fail, never to rank, profile, or single out papers, authors, or reviewers. Author identities are retained only as metadata and are not model features (§3.2/ §16). Public deployment should include rate limits, artifact provenance, citation grounding, warnings that the system does not predict acceptance, and release rules for review-text redistribution and removal requests.

18Conclusion

We presented ResearchStudio-Idea, a three-skill suite for the first mile of literature-grounded ML research ideation. Paper-Search provides standalone multi-source literature grounding, Scoop-Check provides standalone prior-art collision checking, and IdeaSpark composes these functions with outcome-grounded pattern guidance to produce one auditable idea card. The empirical contribution is a 1,947-paper analysis of ICLR, ICML, and NeurIPS outcomes from 2021–2025, inducing 15 ideation patterns, 31 sub-patterns, and 28 research domains. The central system artifact is IdeaSpark: an operational card library and skill specification that make those patterns executable at inference time, with the same contrastive cards guiding bottleneck-to-move selection and candidate audit.

The main claim is deliberately bounded. ResearchStudio-Idea does not show that an LLM can generate Oral papers, predict acceptance, or replace experimental judgment. It shows that public conference outcomes contain reusable strategic operators, that rejected and accepted papers often share those operators, and that their success and failure conditions can be packaged into practical skills for search, generation, and review. The automated-judge evaluation in Section14provides initial endpoint evidence: at matched length and format, IdeaSpark improves judged idea quality over no-skill and generic-skill baselines while remaining competitively novel. The remaining gap is validation beyond the idea stage, including human reviewer preference, implementation success, and transfer outside the studied ML conference setting. Its value is to make pre-experimental ideation more structured, evidence-grounded, and auditable while leaving final prior-art review, experimental design, and scientific claims with the researcher.

Appendix

One generated card and its JSON evidence trail, a reject-path run, and a pattern / sub-pattern card — all real artifacts from §14, reproduced verbatim.

Appendix AEnd-to-end Generated Idea Card

IdeaSpark’s verbatimstd-register output for a held-out ICLR 2026 seed (RL for long-context reasoning), the system’s rank-1 card in §14; its JSON evidence trail (core claim, falsification, literature deltas) is in App.B.

[Uncaptioned image]

Appendix BKill-switch and Falsification Fields

These fields back the same card but are deliberatelynotprinted on it: they are the reviewer-defensible machinery of §6(mechanism-aware falsification with a named load-bearing variable plus a non-tautological negative control, and byte-identical kill-switch preservation across phases). Excerpted verbatim fromphase4_expansion.json.

{

“core_claim”:“RedistributingGRPO’ssingletrajectory-leveladvantageontoaverifier-free,in-group-harvestedper-phasecredit-density--withnothingadded--recoversthelong-contextlearningsignalthatuniformbroadcastdilutesascontextgrowsto128K.”,

“compute_budget”:“30A100-day(withinthe90A100-daydefault;comfortable).TwoGRPOtrainingrunsonasingle7-8Bpolicyover16K-tokenKeyChaindataplus128Kevaluationarethecost;legBaddsnoextraforwardpasses(itreusesthelog-probsalreadyproducedduringsampling),andthereisnoseparatelytrainedrewardmodelorcritictofund.”,

“falsification_prediction”:“Minimalexperiment:traintwocheckpointsfromthesamebaseLLMonidentical16KKeyChaindataandequalcompute--(i)vanillaGRPO(LoongRLsubstrate)and(ii)phase-stratifiedGRPO--andevaluatebothat16K/32K/64K/128KonRULER+extendedmulti-hopQA.Ifthecandidateworks,theaccuracygap(iiminusi)INCREASESwithcontextlength,i.e.stratifiedcreditisroughlyonparat16Kbutits128Kansweraccuracyismateriallyhigher,becausethepivotalphasesstayanear-constanttokenbudgetwhilethedistractorbulkgrows.Thesinglenamedload-bearingvariableistheper-phaseadvantageconcentrationratioC(fractionoftotaladvantagemagnitudeplacedonpivotal-phasetokens);themechanismclaimisthatgainsarecausedbyconcentratingcreditonthepivotalminority.Negativecontrol:permutethephaselabelssothedensitydlandsonrandomlychosenspansinsteadofthecontrast/self-score-identifiedpivotalphases(CisheldattheSAMEnumericlevel,onlyitsplacementisrandomized)--thepredictionisthatdownstream128Kansweraccuracycollapsesbacktothevanilla-GRPOcurve,NOTthatCchanges.ThisistherealPoppertest:ifaccuracysurvivesrandomplacement,thegaincamefromagenericvariance-reductionorextra-computeeffectratherthanfromlocalizingcredittothestepsthatdeterminecorrectness.Positivecontrol:astrippedmodelthatkeepsONLYtheminimal-paircontrastterm(dropself-score)shouldrecovermostofthe128Kaccuracygain,identifyingthein-groupcorrect/incorrectorderingastheload-bearingsourceoftheper-phasesignal.”,

“differentiation_from_lit[0]”:{

“paper_id”:“openalex:W4416246797”,

“delta”:“LoongRLleavestheGRPOadvantageA_guntouchedandbroadcastsituniformlyacrossthewhole16K-128Krollout,concentratingitsnoveltyinKeyChaindataconstruction;weconstructaper-phasestratifiedadvantageA_{g,s}thatredistributesthatsameA_gontothepivotal-phaseminority,anartifacttheirunmodified-optimizerdesignneverproduces.”

}

Appendix CReject-path Output (Phase 3 Abandon)

Not every direction yields a card. When the Phase 3 audit hits an unaddressable hard floor, the run emits aphase_3_failed.mdinstead of an idea (one of the three one-shot outcomes of §6). The example below is a real run on a seed whose direction was, essentially, the title of an existing 2025 paper; the corpus-anchored audit caught the exact-mechanism prior-art collision and abandoned rather than ship a scooped idea.††In the evaluation harness this run was subsequently re-routed onto a different Phase 2.1 gap to still produce a card for scoring; the abandon artifact reproduced here is the genuine output of the abandon branch.

Run:run_prompt85_tnsSeed direction:”A novel ML research idea about Non-Asymptotic analysis of (sticky) Track-and-Stop.”Candidate generated (Phase 2.2):”A Non-Asymptotic Sample-Complexity Certificate for Sticky Track-and-Stop”Verdict:abandonVerdict layer:hard_floor

Triggering checks

1. paper_pointed_threat — exact-mechanism collision (hard floor, unaddressable)

•Threat paper:semanticscholar:63d8201fbe32063e2ac336b0708f589e05cc3759— ”Non-Asymptotic Analysis of (Sticky) Track-and-Stop” (Poiani, Bernasconi & Celli, arXiv:2505.22475, 2025-05). Surfaced via Phase 3.1 collision retrieval.
•Subsumption:The threat abstract states verbatim that Track-and-Stop and Sticky Track-and-Stop ”are optimal in the asymptotic regime [but] their non-asymptotic guarantees remain unknown. In this work, we fill this gap and provide non-asymptotic guarantees for both algorithms.” This is byte-for-byte the candidate’s headline deliverable (a finite-confidence / all-delta guarantee for the genuine sticky-T&S sampling-plus-stopping rule, algorithm held fixed).
•Candidate’s sole differentiator fails:the named additive burn-in functional B(mu), built from the cumulative empirical-vs-oracle allocation deficit D_t, is an INTERNAL proof quantity. Any non-asymptotic bound on sticky-T&S must already control exactly this tracking-error/burn-in term, so the prior paper necessarily bounds it (named or not). Not a distinct external claim.
•addressable_via:null (cannot be repaired by a field edit; the headline deliverable already exists in print).

2. gap_closure_reject_check — triggered

•C19 lesson fires (candidate_match=yes):”A relaxation the field reads as a mechanical lift of a known argument will be judged incremental unless a genuinely novel obstacle is named and overcome.” The obstacle (non-asymptotic guarantees for sticky T&S) is already overcome in arXiv:2505.22475.
•C29 lesson fires (candidate_match=yes):”Re-deriving a rate that merely matches the baseline with near-identical algorithmic changes is read as an incremental extension.” The candidate holds the sticky-T&S algorithm fixed and re-derives the same object the prior paper already certifies.

Recipe application check = applied (both legs perform their cited cluster’s signature move) and anti_pattern check = no match — i.e. the failure axis is prior-work overlap, NOT recipe misuse or anti-pattern composition.

Root cause

Two independent hard-floor / triggered findings point at the same root cause: the user’s seed direction is, essentially, the literal title of an existing 2025 paper. The novelty residual the pipeline could find (a named B(mu) burn-in term) is an internal lemma object, not an externally differentiating claim.

User-side options

•Drop / replace the direction.Non-asymptotic analysis of (sticky) Track-and-Stop as a headline is already occupied by the 2025 paper arXiv:2505.22475.
•Re-frame to an unoccupied residual.Possible pivots NOT covered by the threat paper, e.g.: (a) the forced-exploration / ”sticky” schedule’s finite-confidence tradeoff (a deferred gap in Phase 2.1) — quantify how the forcing rate, set only for asymptotic consistency, governs the finite-delta overhead; (b) instance-dependent LOWER bounds on the burn-in (an impossibility result), rather than an upper bound that the prior paper already supplies; (c) a structured / non-exponential-family or contextual setting the prior non-asymptotic analysis does not reach. Each would require a fresh Phase 0 retrieval to confirm it is unoccupied.
•Re-run Phase 2.1 + 2.2 on a different selected gapfromphase1_output.json(4 gaps were found) with the threat paper now in scope, then re-critique.

Artifacts on disk

•phase0/ (lit_table.md, lit_results.json, fulltext_cache.json)
•phase1/phase1_output.json (state=proceed)
•phase2_select/phase2_select_output.json
•phase2_generate/phase2_generate_output.json (passed citation gate)
•phase3_collision/collision_hits.json (173 hits; critical: arXiv:2505.22475)
•phase3_critique/phase3_critique_output.json (verdict=abandon, hard_floor)
•phase_3_failed.md (this file)

No phase4/ produced. evaluation/opus4_8/85 NOT created (abandon path).

Appendix DExemplar Ideation Pattern and Sub-pattern Card

One of the 15 induced pattern cards (the high-coverageAudit and Pivot an Assumptionexemplar of §6), paired with one of its sub-pattern cards (the tactical recipe a Phase 2.2 candidate instantiates). Reproduced verbatim from the artifact deck; the full 15 patterns×\times31 sub-patterns are in the GitHub artifact release.

Audit and Pivot an Assumptionassumption_audit_and_pivotAudit the load-bearing assumption and pivotCluster-level primary: Oraln=94n=94— HCn=11n=11— Rejectn=79n=79— Totaln=181n=181Definition.Locate the load-bearing implicit assumption a result, guarantee, or defense rests on, then pivot on it: relax it to a weaker condition and re-prove (extending the guarantee), or violate it with a constructed counterexample/exploit (breaking the system or unlocking new behavior).Operational signature.identify the implicit assumption a result or defense rests on→\torelax it (weaker condition) or violate it (counterexample/exploit)→\tore-derive the guarantee or demonstrate the new behaviorWhen to apply.When a result’s strength or a system’s safety hinges on an assumption that real settings can weaken or that an adversary can violate.Step-by-Step:1.Surface the single implicit assumption the target result, guarantee, or defense silently rests on — the precondition the whole subfield has inherited as fixed (a quantity treated as immutable, a property treated as protective, a structure treated as required). Name it explicitly and show it is load-bearing: the existing result holds *because* of it. Do not pivot on an assumption whose removal leaves the achievable outcome unchanged — relaxing a non-binding assumption yields only an incremental generalization, the cluster’s most common rejection.2.Choose the pivot direction deliberately. Either relax the assumption to a strictly weaker condition and plan to re-derive the guarantee (extending it to settings previously thought impossible), or violate it with a constructed counterexample or exploit to break the system or unlock new behavior. The direction must change what is achievable — reach an outcome the assumption was believed to forbid — not merely restate the prior result under a cosmetic reframing.3.Establish that the pivoted condition is real, not a substitution of one strong assumption for another. For a relaxation, verify empirically — in natural data or the actual deployment regime — that the weaker condition genuinely holds, and confirm you have not smuggled in an equally restrictive or unverifiable replacement; the recurring failure is trading a known assumption for a distributional form or abstract regularity condition no real instance is shown to satisfy. For a violation, confirm the assumption is breakable by a realistic, resource-bounded actor rather than an idealized one.4.Re-derive rigorously and certify tightness. For a relaxation, prove the guarantee under the weaker condition and supply a matching lower bound, an exact equivalence, or an impossibility result, so the pivot is certified as the true barrier rather than an artifact of loose analysis. For a violation, demonstrate the exploit and trace its success to a structural mechanism of the pipeline, turning an isolated trick into a systematic, reusable template — empirical success with no identified mechanism is read as anecdote.5.Prove the pivot was necessary and differentiate it. Show the prior approach fails precisely because of the audited assumption via an explicit counterexample or failure-case characterization, and sharply distinguish your move from any prior work that pivoted similarly. Avoid bundling off-the-shelf components and crediting the bundle, and never name a tool or framework as the contribution without exercising it in the actual derivation — both are recurring rejection shapes in this cluster.Success conditions(from Oral):•The relaxed or violated condition is shown to hold in the real setting, not merely assumed — measured in natural data, or demonstrated reachable by a bounded adversary. A pivot is only valuable if the weaker precondition is actually satisfied where the method will be used; grounding the condition empirically converts a theoretical relaxation into a usable one and pre-empts the reviewer suspicion that one strong assumption was swapped for another.•A relaxed upper bound is paired with a matching lower bound, exact equivalence, or impossibility result that certifies the pivot as the genuine barrier. Tightness certificates separate a loose extension from a sharp one: they prove the audited assumption — not the analysis technique — was what limited prior results, which is the structural claim the contribution actually makes.•The audited assumption is replaced by a strictly weaker or structurally different one, often shifting the burden from the prior distribution to the geometry of the map or from pointwise to expectation-level conditions. Moving the constraint to a different and milder level of the problem is what unlocks new achievable results; a same-strength substitution re-imports the original barrier and leaves the frontier unchanged.•For exploits and violations, the success is traced to a structural property of the pipeline and packaged as a systematic design template rather than a one-off attack. A root-cause-level explanation generalizes across instances and gives the community a reusable lens, whereas an isolated demonstration is fragile and reads as engineering rather than insight.•Prior methods are shown to fail precisely because of the audited assumption, via explicit failure-case characterization presented before the positive result. Demonstrating that the assumption is the binding constraint — not an incidental simplification — is what makes the pivot necessary rather than optional, and reviewers consistently reward this clarity.Failure modes(from Reject):•Pivoting on an assumption whose relaxation produces no new achievable result, yielding a generalization that reviewers read as incremental. When the weaker condition leaves the attainable guarantee or rate unchanged, the contribution collapses to ’we also handle case X’ with no unlocking; the absence of a qualitatively new capability is the most cited reason these attempts fall short.•Replacing the audited assumption with an equally strong or unverifiable substitute — a restrictive distributional form, or an abstract regularity condition no known instance is shown to satisfy. The substitution re-imports the very barrier the paper claimed to remove, and reviewers flag that the new condition is untestable or unmotivated, so the claimed relaxation is not actually a relaxation.•Demonstrating an exploit or violation without validating it against an adaptive or informed defender. The central claim of a violation is that the property is hard to counter; showing only that an attack succeeds against a static target leaves that claim unproven, and the ’commonness implies safety’ or ’stealth’ arguments remain intuitive rather than substantiated.•Stacking several off-the-shelf components and attributing the gain to the bundle without isolating which pivot is load-bearing. Reviewers cannot identify a unifying principle and read the work as an engineering combination; without an ablation pinning the benefit to a specific structural move, the novelty above the constituent parts is unconvincing.•Naming a theoretical tool or framework as the contribution while never exercising it in the proofs, or shipping an error in the stated result. If the advertised machinery (a continuous-time analysis, a new perspective) is absent from the actual derivation, or a headline rate is wrong, confidence in the entire pivot evaporates regardless of the idea’s appeal.•Failing to sharply differentiate from prior work that already performed a near-identical prior pivot. In crowded areas a correct but under-differentiated move reads as redundant; reviewers point to the closest precedent and demand a decisive reason to prefer the new approach over that pivot, which an unfocused positioning cannot supply.Oral vs Reject gap.Accepted executions perform three observable moves that rejected ones skip. First, after naming the assumption they verify the pivoted condition concretely — measuring that the weaker condition holds in real or natural data, or that the violated property is reachable by a bounded adversary — whereas rejected papers relax an assumption and immediately substitute an equally strong or unverifiable one (a linear-Gaussian form, an abstract regularity condition with no exhibited satisfying function class). Second, accepted relaxations ship a tightness certificate — a matching lower bound, an exact algebraic equivalence, or an impossibility result — while rejected ones offer only a one-directional upper bound or a bare empirical demonstration, leaving open whether the pivot was the real barrier or just loose analysis. Third, accepted papers establish that prior methods fail precisely because of the audited assumption via an explicit counterexample or failure-case characterization, and for exploits they trace success to a structural pipeline mechanism that becomes a reusable template; rejected papers instead present the work as a generalization or an isolated trick, bundle off-the-shelf components without isolating the load-bearing pivot, or never exercise the theoretical tool they claim as their contribution.Oral vs HC gap.With a moderate HC sample (~10 papers, several of them attacks, benchmarks, or scaling studies), the pattern is that HC papers pivot on an assumption and characterize the consequence at scale or package it as urgently needed reusable infrastructure — a power-law fit for an attack, a measured safety/helpfulness trade-off, a large boundary-targeting benchmark, a broadly transferable defense — but typically stop at empirical characterization or a single bound. Oral papers add a tightness certificate on top of the same pivot: a matching lower bound, an exact equivalence, an identifiability proof, or an impossibility/optimality result that converts ’this works and here is how well’ into ’this is provably the right lever and here is the exact boundary.’ Oral papers also more often formalize the failure modes of prior approaches before presenting the positive result, so the pivot reads as a principled diagnosis rather than a strong empirical finding. Given the sample size this is a tendency rather than a strict law.Reviewer expectations:•[both]When a result rests on a relaxed assumption, supply a matching lower bound or tightness certificate confirming the analysis is not loose; the absence of a matching bound is repeatedly cited as a reason to hold back.•[reject_reviews]Characterize the new or weakened assumption — show which function classes, distributions, or real settings actually satisfy it — rather than introducing it as a convenient condition that merely lets the proof go through.•[reject_reviews]For attacks and violations, validate against adaptive or informed defenders, not just a static target, before claiming the violated property is hard to counter.•[both]Sharply differentiate the pivot from prior work that made a similar move, giving a decisive reason to prefer it; overlap with a close precedent (a prior drift-correction, a prior attack surface) must be addressed head-on.•[reject_reviews]Do not advertise a theoretical framework or tool as the contribution unless it is actually used in the derivation, and ensure headline rates and claims are error-free.•[oral_reviews]Establish and cleanly characterize the failure cases of prior approaches before presenting the positive result, so the pivot is seen as a principled diagnosis of the binding constraint.Cognitive barriers:•The audited assumption is invisible because the entire subfield inherited a framing that treats it as a fixed precondition rather than a controllable choice — a ’fixed’ local objective, a ’required’ supervision signal, a trigger that ’must’ live in the input. You cannot relax what you do not first see as a variable, so the hard part is denaturalizing a convention everyone reasons within.•A stated safety or design property is mistaken for an inherent guarantee: practitioners read ’reduced probability mass,’ ’norm-bounded imperceptibility,’ or ’scale dilutes adversaries’ as protection, which masks that the very property is the exploitable lever. Reframing a feature as a vulnerability requires inverting the intent the property was introduced to serve.•The pivot often hinges on importing a tool from an adjacent domain and recognizing it transfers — concentration tools from sequential testing, blocking from dependent-sequence analysis, scalarization from multi-objective optimization. Because the tool was built for a different purpose, its applicability to the audited assumption is not visible from within either field’s standard practice.•The relaxation frequently lives at a different level of the problem than the natural formulation suggests — reparameterizing the latent basis instead of the observed variables, constraining the geometry of the generative map instead of the prior distribution, measuring increments or contrasts instead of raw values. The obvious formulation hides the lever, so the move feels unreachable until one deliberately changes the level of analysis.Examples(paper-agnostic lessons):•[Oral lesson]When a guarantee is blocked by a quantity treated as fixed, ask whether that quantity is actually a controllable variable you can reshape cheaply each step — turning a hard constraint into a degree of freedom often dissolves the barrier.•[Oral lesson]Measure that the weaker condition you rely on actually holds in real data before claiming the relaxation is practical; a precondition verified in natural statistics is worth far more than one merely assumed.•[Oral lesson]Pair every relaxed upper bound with a matching lower bound, exact equivalence, or impossibility result, so the pivot is certified as the true barrier rather than an artifact of loose analysis.•[Oral lesson]Replace a distributional assumption with a structural or architectural one when the problem’s geometry already encodes the constraint — identifiability can rest on the shape of the generative map rather than on the prior over latents.•[Oral lesson]When building an exploit, trace its success to a structural property of the pipeline and convert it into a reusable design template rather than presenting an isolated trick.•[Oral lesson]Read every ’bounded,’ ’small,’ or ’imperceptible’ guarantee as a budget an adversary can neutralize — a stated protective property is frequently the exact lever to violate.•[Reject lesson]Relaxing an assumption that leaves the achievable result unchanged reduces the contribution to a generalization reviewers read as incremental.•[Reject lesson]Swapping the audited assumption for an equally strong or unverifiable one — a restrictive distributional form, or a regularity condition no known instance is shown to satisfy — re-imports the barrier you claimed to remove.•[Reject lesson]Demonstrating an exploit only against a static defender leaves the central claim, that the violation is hard to counter, unproven.•[Reject lesson]Bundling several off-the-shelf components and crediting the bundle, without isolating which pivot is load-bearing, invites a ’no unifying principle’ rejection.•[Reject lesson]Naming a theoretical framework as the contribution while never exercising it in the proofs — or shipping an error in the stated rate — collapses confidence in the whole result.•[Reject lesson]Failing to sharply distinguish the work from a prior near-identical pivot makes even a correct result read as redundant in a crowded area.

Representative sub-pattern under this pattern (the tactical recipe a Phase 2.2 candidate instantiates):

C01(O20/H9/R21)Tactical pattern.Across this cluster the audited assumption is always about the *representational locus* at which a security, detection, provenance, or safety property is presumed to be enforced — and the tactical move is to relocate the attack or defense to a different locus the field treated as irrelevant or invariant. Concretely: triggers are moved from the input image to the supervision/label co-occurrence space; the comparison signal for malicious updates is moved from parameter/gradient divergence to scalar-score distribution geometry; the detection feature is moved from a scalar confidence to local curvature, positional-depth profile, or the full sampled output distribution; provenance is moved from raw parameter comparison to transformation-invariant kernel alignment over intermediate activations; defense interception is moved from the output boundary to internal state trajectories. For attacks, the papers show the overlooked locus is exploitable at far lower cost or access than the assumed locus implied; for defenses, they show the overlooked locus carries an attack-agnostic, transformation-invariant signal the assumed locus lacks. The unifying verbs are ”relocate the signal,” ”re-anchor enforcement,” and ”demonstrate exploitability/robustness at the new locus.”Step-by-Step:1.Name the single representational locus that current attacks or defenses in the target security/detection setting all operate on (input perturbation, parameter/gradient divergence, scalar confidence score, deterministic mode output, externally-injected artifact), and state the tacit sufficiency-or-invariance claim bolted to it as a falsifiable proposition about where the load-bearing signal lives. The sharper the claim (’the trigger must be in the input’, ’scale dilutes contamination’, ’the scalar score is the signal’), the more leverage the pivot has.2.Identify an alternative locus — supervision records, intermediate-activation geometry, score-distribution shape, the full stochastic output distribution, behavioral input-output traces, internal-state trajectories — and argue structurally why the property of interest must manifest there too, ideally MORE robustly: invariant to the very transformations or access restrictions that defeat the assumed locus. If the alternative is only a cosmetic variant of the assumed locus, reviewers fold the contribution back into prior work, so the distinction must be load-bearing and explicit.3.Build the mechanism at the new locus: for an attack, the lowest-access, lowest-budget operation that exploits it; for a defense, a signal demonstrably agnostic to the attacker’s configuration or insertion point. Where possible, couple detection and action through the same locus signal so the move is one unified primitive rather than two bolted-on stages.4.Validate against the strongest adversary that actually knows the new locus — an adaptive attacker for a defense, or the realistic access/budget constraints for an attack — not merely a fixed, non-adaptive baseline. Quietly demonstrating the move only against the bypassed locus, or against outdated baselines, is the single most common reason this tactic fails to convince.5.Supply a mechanistic account of WHY the new locus carries the signal — an invariance proof, an impossibility-style bound, or a root-cause decomposition of the pipeline — not merely evidence THAT it does. A relocation shown to work but never explained reads as a stacked combination of existing components or an anecdote, and forfeits the conceptual-lens credit that elevates these papers.Differentiation within parent.All siblings share the parent’s audit-and-pivot signature, but they pivot mathematical objects inside a fixed locus, whereas this cluster pivots the locus itself in adversarial/empirical security and safety settings and validates with attack-success rates and detection performance rather than tighter analysis. Cluster 11 re-derives privacy-accounting bounds by reframing repeated selection as a resource-bounded mechanism — the assumption is relaxed but the quantity (a cumulative bound) and its locus (the accounting calculus) are unchanged; this cluster instead relocates the enforcement point entirely (e.g., from parameter-divergence to score-distribution geometry) and shows the old locus was the wrong place to look. Likewise cluster 29 proves a new property of a projection operator on a constraint set and cluster 5 derives an exact algebraic equivalence between model families — both are theorem-driven reformulations of analytic structure, while this cluster’s deliverable is a demonstrated attack surface or a transformation-invariant detection signal. The contrast is empirical-locus-shift-under-adversary, not closed-form quantity tightening.When to pick this one.Pick this instantiation when the gap sits in a security, safety, detection, or provenance setting and you can observe that every existing attack or defense is anchored at one representational locus (raw input, parameters/gradients, a scalar score, a deterministic output, or an injected artifact). The trigger to instantiate here is a suspicion that the real load-bearing signal lives at an overlooked locus — labels, activations, distribution shape, or behavior — that the field has tacitly assumed irrelevant or invariant.Tactical failure mode.The recurring failure is a successful relocation that is demonstrated but not adversarially closed: the paper shows its new-locus attack or defense beats a fixed, non-adaptive baseline, yet never stress-tests against an informed adversary who also operates at the new locus, so stealthiness, evasion, or robustness claims are asserted rather than proven. A second shape is a novelty-boundary collapse — the ”new” locus turns out to be one a prior paper already used (normalization statistics, activation-similarity alignment, pre-encoded semantic concepts), and the work fails to quantify what is genuinely different. A third is reintroducing the very assumption the relocation was meant to dissolve, by depending on a clean reference model, a known anomaly distribution, or other access the realistic threat model denies. A fourth is offering only a demonstration with no mechanistic account of why the locus carries the signal, which reads as a stacked pipeline of off-the-shelf parts. Self-check: can an adversary who knows your new locus defeat it as cheaply as the old one, is your locus provably distinct from published loci, and can you state structurally WHY the signal lives there?Examples(paper-agnostic lessons):•[Oral lesson]Moving a trigger or detection signal from the input space to the supervision/label space opens an attack surface the entire defense literature was built around the wrong locus to catch.•[Oral lesson]A detection or provenance signal anchored in intermediate-representation geometry survives exactly the transformations — fine-tuning, pruning, unit reordering, rescaling — that destroy parameter- and output-level signals.•[Oral lesson]Treating the shape of a learned distribution (its local curvature, positional-depth profile, or full sampled spread) as the signal, rather than a scalar summary, surfaces structure that point estimates provably discard.•[Oral lesson]Replacing a randomized, probabilistic guarantee with a deterministic combinatorial construction at the same structural locus can upgrade a soft bound into an exact certificate.•[Oral lesson]Showing that a stated security property itself — an imperceptibility norm bound, scale-induced dilution, or deterministic evaluation — is the exploitable handle converts a design parameter into a provable vulnerability.•[Oral lesson]Decomposing an empirically observed failure into structural root causes of the training pipeline turns ad hoc attack discovery into a systematic, predictive design process.•[Reject lesson]Claiming a relocated signal is ’stealthier’ or ’harder to remove’ without defeating an adaptive defender who searches the same new locus leaves the central claim unverified.•[Reject lesson]Relocating to a locus already exploited by published work — normalization statistics, activation-similarity losses, semantic concept triggers — collapses the novelty boundary unless the difference is made explicit and measured.•[Reject lesson]A defense whose relocation silently depends on a clean reference model or a known anomaly distribution reintroduces precisely the strong assumption the move was supposed to eliminate.•[Reject lesson]Demonstrating the new locus only on fixed, non-adaptive or outdated baselines while omitting any account of why it works reads as a stacked pipeline of existing components rather than a principled shift.•[Reject lesson]Wiring off-the-shelf generators or tools into a new attack setting without a structural reason for the locus change gets judged as engineering integration, not insight.

References

[1](2026)SPECTER2 model card.Note:Official SPECTER2 Hugging Face model card; checked May 10, 2026External Links:LinkCited by:§1.3.
[2]Anthropic(2026)Claude models overview.Note:Official Anthropic model documentation; checked May 10, 2026External Links:LinkCited by:§4.1.
[3]Anthropic(2026)Skill authoring best practices.Note:Official Anthropic documentation; checked May 10, 2026External Links:LinkCited by:§1.3,§2.5.
[4]A. H. Artiles, M. Weiss, L. Brinkmann, A. Goyal, and N. Rahaman(2026)Alien science: sampling coherent but cognitively unavailable research directions from idea atoms.Note:arXiv:2603.01092; checked May 10, 2026External Links:2603.01092,LinkCited by:§1.2,§2.2.
[5]H. Bishtet al.(2026)Agentic AI scientists are not built for autonomous scientific discovery.External Links:2605.08956Cited by:§2.1.
[6]A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. S. Weld(2020)SPECTER: document-level representation learning using citation-informed transformers.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp. 2270–2282.Note:ACL 2020; arXiv:2004.07180; checked May 10, 2026External Links:LinkCited by:§1.3.
[7]I. L. da Silva, H. Yan, L. Gui, and Y. He(2025)GraphMind: interactive novelty assessment system for accelerating scientific discovery.Note:arXiv:2510.15706; checked May 10, 2026External Links:2510.15706,LinkCited by:§1.2,§2.4.
[8]S. Gaoet al.(2026)Graphs of research: citation evolution graphs as supervision for research idea generation.External Links:2605.14790Cited by:§1.2,§2.3.
[9]A. Garikaparthi, M. Patwardhan, L. Vig, and A. Cohan(2025)IRIS: interactive research ideation system for accelerating scientific discovery.Note:arXiv:2504.16728; checked May 10, 2026External Links:2504.16728,LinkCited by:§2.2.
[10]I. Gemp, B. McWilliams, C. Vernade, and T. Graepel(2021)EigenGame: PCA as a nash equilibrium.InInternational Conference on Learning Representations,Note:ICLR 2021 Outstanding Paper; corpus paper_id: ICLR_2021_0018; checked May 10, 2026External Links:LinkCited by:§3.3.
[11]J. Gottweis, W. Weng, A. Daryin, T. Tu, A. Palepu,et al.(2025)Towards an AI co-scientist.Note:arXiv:2502.18864; checked May 10, 2026External Links:2502.18864,LinkCited by:§1.1,§2.1.
[12]M. Grootendorst(2022)BERTopic: neural topic modeling with a class-based tf-idf procedure.Note:arXiv:2203.05794; checked May 10, 2026External Links:2203.05794,LinkCited by:§1.3.
[13]C. Gu, J. Cheng, M. Zhang, P. Zheng, J. Zheng, and G. He(2026)MoRI: learning motivation-grounded reasoning for scientific ideation in large language models.Note:arXiv:2603.19044; checked May 10, 2026External Links:2603.19044,LinkCited by:§1.2,§2.3.
[14]X. Gu and M. Krenn(2024)Interesting scientific idea generation using knowledge graphs and LLMs: evaluations with 100 research group leaders.Note:arXiv:2405.17044; checked May 10, 2026External Links:2405.17044,LinkCited by:§1.2,§2.3.
[15]S. Guo, A. H. Shariatmadari, G. Xiong, A. Huang, E. Xie, S. Bekiranov, and A. Zhang(2024)IdeaBench: benchmarking large language models for research idea generation.Note:arXiv:2411.02429; checked May 10, 2026External Links:2411.02429,LinkCited by:§2.4.
[16]M. Henzinger, S. Sricharan, and T. Steiger(2023)The price of differential privacy under continual observation.InInternational Conference on Machine Learning,Note:Corpus paper_id: ICML_2023_0551; checked May 10, 2026External Links:LinkCited by:§3.3.
[17]X. Hu, H. Fu, J. Wang, Y. Wang, Z. Li, R. Xu, Y. Lu, Y. Jin, L. Pan, and Z. Lan(2024)Nova: an iterative planning and search approach to enhance novelty and diversity of LLM generated ideas.Note:arXiv:2410.14255; checked May 10, 2026External Links:2410.14255,LinkCited by:§1.2,§2.2.
[18]J. Huang, S. Cucerzan, S. K. Jauhar, and R. W. White(2025)Idea2Plan: exploring AI-powered research planning.Note:arXiv:2510.24891; checked May 10, 2026External Links:2510.24891,LinkCited by:§1.1,§2.1.
[19]B. Jiang(2026)HindSight: evaluating LLM-generated research ideas via future impact.Note:arXiv:2603.15164; checked May 10, 2026External Links:2603.15164,LinkCited by:§1.2,§2.4.
[20]D. P. Kingma and R. Gao(2023)Understanding diffusion objectives as the ELBO with simple data augmentation.InAdvances in Neural Information Processing Systems,Note:Corpus paper_id: NeurIPS_2023_0753; checked May 10, 2026External Links:LinkCited by:§3.3.
[21]A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang(2022)Fine-tuning can distort pretrained features and underperform out-of-distribution.InInternational Conference on Learning Representations,Note:Corpus paper_id: ICLR_2022_0094; checked May 10, 2026External Links:LinkCited by:§3.3.
[22]X. Lei, T. Zhou, Y. Chen, K. Liu, and J. Zhao(2025)MotivGraph-soiq: integrating motivational knowledge graphs and socratic dialogue for enhanced LLM ideation.Note:arXiv:2509.21978; checked May 10, 2026External Links:2509.21978,LinkCited by:§1.2,§2.3.
[23]P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, S. Riedel, and D. Kiela(2020)Retrieval-augmented generation for knowledge-intensive NLP tasks.InAdvances in Neural Information Processing Systems,Note:NeurIPS 2020; arXiv:2005.11401; checked May 10, 2026External Links:LinkCited by:§1.2.
[24]A. Liet al.(2026)MIRAI: prediction and generation of high-impact academic research.External Links:2606.05443Cited by:§2.4.
[25]C. Liu, C. Wang, J. Cao, J. Ge, K. Wang, L. Zhang, M. Cheng, P. Zhao, T. Li, X. Jia, X. Li, X. Li, Y. Liu, Y. Feng, Y. Huang, Y. Xu, Y. Sun, Z. Zhou, and Z. Xu(2025)A vision for auto research with LLM agents.Note:arXiv:2504.18765; checked May 10, 2026External Links:2504.18765,LinkCited by:§1.1,§2.1.
[26]J. Liu, M. Harmon, and Z. Zhang(2026)Sci-reasoning: a dataset decoding AI innovation patterns.Note:arXiv:2601.04577; checked May 10, 2026External Links:2601.04577,LinkCited by:§1.2,§2.3.
[27]X. Liu, X. Dong, X. Gao, Y. Feng, and X. Pang(2025)Augmenting research ideation with data: an empirical investigation in social science.Note:arXiv:2505.21396; checked May 10, 2026External Links:2505.21396,LinkCited by:§2.4.
[28]Y. Liu, Z. Yang, T. Xie, J. Ni, B. Gao, Y. Li, S. Tang, W. Ouyang, E. Cambria, and D. Zhou(2025)ResearchBench: benchmarking LLMs in scientific discovery via inspiration-based task decomposition.Note:arXiv:2503.21248; checked May 10, 2026External Links:2503.21248,LinkCited by:§2.4.
[29]C. Lu, C. Lu, R. T. Lange, Y. Yamada, S. Hu, J. Foerster, D. Ha, and J. Clune(2026)Towards end-to-end automation of AI research.Nature.Note:Nature 2026; originally arXiv:2408.06292; checked May 10, 2026External Links:Document,2408.06292,LinkCited by:§1.1,§1.2,§2.1.
[30]L. McInnes, J. Healy, and S. Astels(2017)Hdbscan: hierarchical density based clustering.Journal of Open Source Software2(11),pp. 205.Note:DOI:10.21105/joss.00205; checked May 10, 2026External Links:Document,LinkCited by:§1.3,§5.2.
[31]L. McInnes, J. Healy, and J. Melville(2018)UMAP: uniform manifold approximation and projection for dimension reduction.Note:arXiv:1802.03426; checked May 10, 2026External Links:1802.03426,LinkCited by:§1.3,§5.2.
[32]T. Möllenhoff and M. E. Khan(2023)SAM as an optimal relaxation of bayes.InInternational Conference on Learning Representations,Note:ICLR 2023 Spotlight; corpus paper_id: ICLR_2023_0331; checked May 10, 2026External Links:LinkCited by:§3.3.
[33]OpenAI(2026)Embeddings.Note:Official OpenAI API documentation; checked May 10, 2026External Links:LinkCited by:§1.3.
[34]OpenReview(2026)OpenReview documentation: using the api.Note:Official OpenReview documentation; checked May 10, 2026External Links:LinkCited by:§1.3,§3.1.
[35]Y. Qiu, H. Zhang, Z. Xu, M. Li, D. Song, Z. Wang, and K. Zhang(2025)AI idea bench 2025: AI research idea generation benchmark.Note:arXiv:2504.14191; checked May 10, 2026External Links:2504.14191,LinkCited by:§2.4.
[36]M. Radensky, S. Shahid, R. Fok, P. Siangliulue, T. Hope, and D. S. Weld(2024)Human-LLM compound system for scientific ideation through facet recombination and novelty evaluation.Note:Introduces Scideator; arXiv:2409.14634; checked May 10, 2026External Links:2409.14634,LinkCited by:§2.4.
[37]S. Ren, C. Xie, P. Jian, Z. Ren, C. Leng, and J. Zhang(2025)Towards scientific intelligence: a survey of LLM-based scientific agents.Note:arXiv:2503.24047; checked May 10, 2026External Links:2503.24047,LinkCited by:§2.5.
[38]D. Saraogi, R. Singhee, and D. Kumar(2025)Evaluating novelty in AI-generated research plans using multi-workflow LLM pipelines.Note:arXiv:2601.09714; checked May 10, 2026External Links:2601.09714,LinkCited by:§1.2,§2.2.
[39]T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom(2023)Toolformer: language models can teach themselves to use tools.InAdvances in Neural Information Processing Systems,Note:NeurIPS 2023; arXiv:2302.04761; checked May 10, 2026External Links:LinkCited by:§1.3.
[40]S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum(2025)Agent laboratory: using LLM agents as research assistants.Note:arXiv:2501.04227; checked May 10, 2026External Links:2501.04227,LinkCited by:§1.1,§1.2,§2.1.
[41]T. Schopfet al.(2026)Is this idea novel? an automated benchmark for judgment of research ideas.External Links:2603.10303Cited by:§1.2,§2.4.
[42]Semantic Scholar(2026)Semantic scholar graph api.Note:Official Semantic Scholar API documentation; checked May 10, 2026External Links:LinkCited by:§1.3,§3.1.
[43]S. Shahid, M. Radensky, R. Fok, P. Siangliulue, D. S. Weld, and T. Hope(2025)Literature-grounded novelty assessment of scientific ideas.Note:arXiv:2506.22026; checked May 10, 2026External Links:2506.22026,LinkCited by:§1.2,§2.4.
[44]Y. Shen, M. Liu, D. Zhou, and L. Huang(2026)Navigating ideation space: decomposed conceptual representations for positioning scientific ideas.Note:arXiv:2601.08901; checked May 10, 2026External Links:2601.08901,LinkCited by:§1.2,§2.3.
[45]C. Si, T. Hashimoto, and D. Yang(2025)The ideation-execution gap: execution outcomes of LLM-generated versus human research ideas.Note:arXiv:2506.20803; checked May 10, 2026External Links:2506.20803,LinkCited by:§1.2,§2.4.
[46]C. Si, D. Yang, and T. Hashimoto(2024)Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers.Note:arXiv:2409.04109; checked May 10, 2026External Links:2409.04109,LinkCited by:§1.2,§2.4.
[47]A. Singh, M. D’Arcy, A. Cohan, D. Downey, and S. Feldman(2022)SciRepEval: a multi-format benchmark for scientific document representations.Note:arXiv:2211.13308; includes SPECTER2 evaluation context; checked May 10, 2026External Links:2211.13308,LinkCited by:§1.3.
[48]H. Su, R. Chen, S. Tang, Z. Yin, X. Zheng, J. Li, B. Qi, Q. Wu, H. Li, W. Ouyang, P. Torr, B. Zhou, and N. Dong(2024)Many heads are better than one: improved scientific idea generation by a LLM-based multi-agent system.Note:arXiv:2410.09403; checked May 10, 2026External Links:2410.09403,LinkCited by:§2.2.
[49]J. Tang, L. Xia, Z. Li, and C. Huang(2025)AI-researcher: autonomous scientific innovation.Note:arXiv:2505.18705; checked May 10, 2026External Links:2505.18705,LinkCited by:§1.1,§1.2,§2.1.
[50]J. Tong, M. Li, H. Li, Y. Yang, Y. Mou, W. Ma, Z. Xi, H. Chen, X. Liu, Q. Cheng, M. Zhang, Q. Chen, W. Ge, Q. Guo, T. Ying, T. Sun, Y. Zheng, X. Chen, J. Zhao, N. Ding, X. Huang, Y. Jiang, and X. Qiu(2026)AI can learn scientific taste.Note:arXiv:2603.14473; checked May 10, 2026External Links:2603.14473,LinkCited by:§2.4.
[51]D. Trehanet al.(2026)Why LLMs aren’t scientists yet: lessons from four autonomous research attempts.External Links:2601.03315Cited by:§2.1.
[52]H. Wang, P. Jiang, J. Sun, Z. Shi, H. Yu, J. Han, and H. Ji(2026)Learning to predict future-aligned research proposals with language models.Note:arXiv:2603.27146; checked May 10, 2026External Links:2603.27146,LinkCited by:§2.4.
[53]Q. Wanget al.(2026)FlowPIE: test-time scientific idea evolution with flow-guided literature exploration.External Links:2603.29557Cited by:§1.2,§2.2.
[54]J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou(2022)Chain-of-thought prompting elicits reasoning in large language models.InAdvances in Neural Information Processing Systems,Note:NeurIPS 2022; arXiv:2201.11903; checked May 10, 2026External Links:LinkCited by:§1.3.
[55]W. Wu, Y. Zhao, Y. Wang, S. Li, J. Shao, Y. Long, and C. Zhang(2026)NovBench: evaluating large language models on academic paper novelty assessment.Note:arXiv:2604.11543; checked May 10, 2026External Links:2604.11543,LinkCited by:§1.2,§2.4.
[56]G. Xiong, E. Xie, A. H. Shariatmadari, S. Guo, S. Bekiranov, and A. Zhang(2024)Improving scientific hypothesis generation with knowledge grounded large language models.Note:arXiv:2411.02382; checked May 10, 2026External Links:2411.02382,LinkCited by:§1.2,§2.3.
[57]Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha(2025)The AI scientist-v2: workshop-level automated scientific discovery via agentic tree search.Note:arXiv:2504.08066; checked May 10, 2026External Links:2504.08066,LinkCited by:§2.1.
[58]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao(2023)ReAct: synergizing reasoning and acting in language models.InInternational Conference on Learning Representations,Note:ICLR 2023; arXiv:2210.03629; checked May 10, 2026External Links:LinkCited by:§1.3.
[59]M. Zhang, K. Tan, Y. Huang, Y. Shen, C. Ma, L. Ju, X. Zhang, Y. Wang, W. Jing, J. Deng, H. Sha, B. Hu, J. Tong, C. Jiang, Y. Geng, Y. Ying, Y. Zhang, Z. Yin, Z. Xi, S. Dou, T. Gui, Q. Zhang, and X. Huang(2026)OpenNovelty: an LLM-powered agentic system for verifiable scholarly novelty assessment.Note:arXiv:2601.01576; checked May 10, 2026External Links:2601.01576,LinkCited by:§1.2,§2.4.
[60]K. Zhao, W. Lin, Q. Zheng, F. Xu, and Y. Li(2025)Deep ideation: designing LLM agents to generate novel research ideas on scientific concept network.Note:arXiv:2511.02238; checked May 10, 2026External Links:2511.02238,LinkCited by:§1.2,§2.2.
[61]T. Zheng, Z. Deng, H. T. Tsang, W. Wang, J. Bai, Z. Wang, and Y. Song(2025)From automation to autonomy: a survey on large language models in scientific discovery.Note:arXiv:2505.13259; checked May 10, 2026External Links:2505.13259,LinkCited by:§1.2,§2.5.
[62]T. Zheng, K. K. Tam, N. H. K. Nguyen, B. Xu, Z. Wang, J. Cheng, H. T. Tsang, W. Wang, J. Bai, T. Fang, Y. Song, G. Y. Wong, and S. See(2025)NewtonBench: benchmarking generalizable scientific law discovery in LLM agents.Note:arXiv:2510.07172; checked May 10, 2026External Links:2510.07172,LinkCited by:§2.4.

相似文章

ResearchStudio-Idea：基于证据的科研构思技能套件——源自机器学习会议成果

Hugging Face Daily Papers

ResearchStudio-Idea 是一个技能套件，结合了文献搜索、新颖性检查和模式引导生成，能够产生可追溯的研究提案，该套件基于对1,947篇机器学习会议论文的分析构建而成。

@Xudong07452910: 现在很多人聊 Research Agent，默认期待是：读论文、找 gap、想 idea、做实验、写 paper。但耶鲁大学的这篇论文问了一个更深入的问题： LLM 生成的研究想法，和人类研究者真正做出来的 paper idea，差距…

X AI KOLs Timeline

耶鲁大学的论文通过构建大规模评估框架，比较了LLM与人类研究者在生成研究想法上的分布差距，发现LLM想法高度集中于bridge和synthesis类型，而人类想法分布更广，揭示了'研究品味'的差异，对Research Agent的多样性提出挑战。

@gaoren7716: 写论文这件事，可能要被流程化 AI 系统重写了不是帮你润色，不是帮你改一句话，而是从选题开始就有 13 个 Agent 在协作功能清单： Deep Research（13-agent 调研团队） Systematic Review（P…

X AI KOLs Timeline

介绍了一套名为Academic Research Skills的开源AI工具，通过13个Agent协作实现从选题到写作、审稿的全流程学术研究自动化，可作为Claude Code插件使用，将学术研究变为标准化生产线。

@XAMTO_AI: 20分钟捅出一篇能投顶刊的论文，这事儿现在真不是吹牛批。 https://github.com/brycewang-stanford/Auto-Empirical-Research-Skills… 以前做实证有多熬人你们心里都有数：选题、…

X AI KOLs Timeline

Stanford REAP and CoPaper.AI have released Auto-Empirical Research Skills (AERS), an open-source toolkit with over 23,000 agent skills that automates the entire empirical research pipeline for social sciences, from topic selection to journal submission.

@yrzhe_top: 试了一下 Claude Science，感觉很适合做科研早期调研。它会把一个研究问题拆成多个检索面，连接 OpenAlex、arXiv 这样的文献库，抓取真实文献数据，然后自动去重、过滤、聚类、画图，最后生成一份带主题地图、增长趋势和代表…

X AI KOLs Timeline

Claude Science是一款科研早期调研工具，可自动检索、去重、聚类文献并生成研究地图和趋势报告，对探索新领域很有帮助。