@Xudong07452910: Nowadays, many people talk about Research Agents, with the default expectation being: read papers, find gaps, come up with ideas, run experiments, write papers. But this paper from Yale University asks a deeper question: How far apart are LLM-generated research ideas from the paper ideas that human researchers actually produce…

Many people now talk about Research Agents, with the default expectation being: read papers, find gaps, generate ideas, run experiments, and write papers. But this paper from Yale asks a deeper question: what is the actual gap between LLM-generated research ideas and the paper ideas that human researchers truly produce? The core concept is called research taste — it can be understood as the kind of problems you typically notice and how you turn them into a contribution.

The experimental design is clever. The authors extract human ideas from real papers, then reverse-engineer 4 to 8 prior works that most likely inspired each idea. They feed the titles and abstracts of these prior works to LLMs, asking the models to generate a new research idea from the same literature context. This shifts the comparison from “whether a single idea looks novel” to “whether the distribution of a batch of ideas resembles that of human researchers.”

The paper uses 11,683 human ideas in total, covering ML conference papers and natural science papers from Nature Communications. The models include 9 settings such as Claude, Gemini, GPT, Qwen, DeepSeek, etc.

The results are striking: the distribution of human ideas is clearly broader. Among human papers, only 12.1% of ideas fall under “bridge opportunity” — i.e., connecting different literature, methods, or evidence streams. But among LLM-generated ideas, this proportion jumps to 47.1%–64.2%. The pattern is similar for methodology: in human papers, synthesis/unification accounts for only 5.1%; for LLMs, it’s 22.5%–38.7%. In short, models tend to frame research problems as: “here are two things, let’s integrate them.”

The paper also finds that giving the model more context does not significantly fix this. The full-paper context version does not bring the distribution closer to humans; enabling thinking mode may even reinforce the bridge-and-synthesis tendency.

The most interesting aspect of this paper is that it does not simply judge LLM ideas as good or bad, but points out a more subtle problem: LLMs can generate many seemingly reasonable research ideas, but the types of those ideas may be highly concentrated.

This is crucial for AI for Science and Research Agents. For a truly strong Research Agent, the challenge is whether it can develop a broader problem-finding ability: discovering failures, identifying mechanisms, proposing measurement tools, constructing systems, modifying local assumptions — rather than always falling back on the safe path of “integrating two existing things.” If automated research lacks this diversity, it may ultimately become a very fluent but homogeneous machine.

arxiv: https://arxiv.org/abs/2607.01233

— # Measuring the Gap Between Human and LLM Research Ideas

Source: https://arxiv.org/html/2607.01233

Ziyu ChenC\hskip 1.00006pt{}^{{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\boldsymbol{C}}}Yilun ZhaoY\hskip 1.00006pt{}^{{\color[rgb]{0,0.20703125,0.41796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.20703125,0.41796875}\boldsymbol{Y}}}Arman CohanY\hskip 1.00006pt{}^{{\color[rgb]{0,0.20703125,0.41796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.20703125,0.41796875}\boldsymbol{Y}}} Y\hskip 1.00006pt{}^{{\color[rgb]{0,0.20703125,0.41796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.20703125,0.41796875}\boldsymbol{Y}}}Yale UniversityC\hskip 1.00006pt{}^{{\color[rgb]{0.5,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0,0}\boldsymbol{C}}}University of Chicago ziyuuc/TasteGapIdeaLand/IdeaSeed ###### Abstract LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste. Refer to captionFigure 1:Overview of our research-taste gap analysis. From a shared literature context, humans contribute the paper idea while LLMs generate new ideas from the same prior works. Each idea is decomposed into amotivationand amethod, then annotated with a research-taste taxonomy. Comparing the resulting distributions reveals that LLM ideas are substantially narrower than human ideas, with strong biases toward bridge-like motivations and explicit synthesis methods.## 1Introduction Research ideation is one of the most ambitious proposed uses of LLMs. Some controlled human studies have shown that LLM-generated ideas can match or approach those from human experts in terms of judged novelty and feasibility, demonstrating the potential of LLMs as ideation tools(Si et al.,2025a (https://arxiv.org/html/2607.01233#bib.bib1); Baek et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib2); Si et al.,2025b (https://arxiv.org/html/2607.01233#bib.bib3)). More recently, AI-scientist systems have already begun adapting LLMs to generate research ideas, execute experiments, and write paper drafts, bringing this capability into scientific process(Boiko et al.,2023 (https://arxiv.org/html/2607.01233#bib.bib4); Lu et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib5); Zhang et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib6); Zhao et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib7); Vasu et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib8); Zhao et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib9); Wu et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib10)). Despite this progress, a basic empirical question remains unanswered:What kinds of research ideas do LLMs tend to produce, and how far are current LLM-generated ideas from human researchers? Most existing evaluations of LLM ideation judge ideas individually, using criteria such as novelty, feasibility, impact, or preference(Si et al.,2025a (https://arxiv.org/html/2607.01233#bib.bib1); Baek et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib2); Garikaparthi et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib11); Tong et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib12)). In this work, we take a complementary distributional view. We useresearch tasteto refer to the kinds of problems, gaps, and contributions that a source tends to produce across many comparable literature-grounded ideation contexts. Under this view, research taste concerns not only whether an individual idea is reasonable, but also what kinds of gap framings and contribution strategies repeatedly appear when the same source is asked to generate ideas under comparable constraints. This distributional view matters because a single idea may appear novel, feasible, and coherent, while the broader set of ideas from the same source may still reflect a narrow range of research taste. Research communities generate many kinds of contributions: some papers discover a failure mode, some relax an assumption, some build a measurement instrument, some introduce a formal explanation, and others construct a system or artifact. An LLM that generates reasonable ideas one at a time may therefore still be behaviorally narrow if its outputs repeatedly identify the same kinds of gaps, use the same methodological paradigms, or rely on the same contribution templates(Meincke et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib13); Smith et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib14); Sourati et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib15)). Such concentration would affect how LLMs are used for brainstorming, literature exploration, and automated research agents, even when many individual outputs appear coherent. We study this question through a constrained literature-grounded ideation task. Each example consists of a small set of closely related prior papers, represented by their titles and abstracts. The target output is a new research idea, separated into amotivationand amethod. This differs from open-ended ideation such aswrite an idea about topic X. Grounding each generation in a small related-work context makes the task comparable across human and LLM outputs. The human idea is the idea realized in the real paper, while the LLM idea is generated from the reconstructed local context that seemingly reasonably preceded that paper. This setting also reduces the chance that differences are driven only by broad topic choice or by an LLM’s preferred generic paper format. To build the evaluation framework, we collect real papers in machine learning and natural science domains. For each paper, we use a strong LLM-assisted extraction pipeline to identify the paper’s core human idea and to reverse-engineer several closely related prior works from which that idea can be understood. We retrieve these prior works and prompt evaluated LLMs to produce a new motivation and method from the same prior-work context. This yields paired human and LLM idea corpora over the same inputs. We compare these corpora at the distributional level through a two-dimensional view of research taste. One dimension characterizes how a proposal frames the underlying research opportunity, ranging from identifying missing explanations or overlooked failures to exposing structural disconnects or limitations in existing understanding. The other captures the style of intellectual contribution through which the opportunity is developed into a research idea, including analytical, constructive, integrative, and exploratory forms of proposed methods. We introduce a taxonomy along these two axes, constructed by human experts through a review of research guidance from NSF, NIH, AHRQ and DARPA, and then iteratively refined using a held-out set of papers to ensure applicability across both machine learning and natural science domains. We apply the taxonomy at scale using an LLM annotator validated against independent human judgments. We find that LLM-generated ideas occupy a substantially narrower region of the research-taste taxonomy than human ideas. This narrowing is most visible in an ideation pattern centered on connection, where model ideas more often frame the motivation as a need to link previously literatures, methods, or evidence streams, and more often develop the method by integrating, reconciling, or unifying existing approaches. In our evaluation, only 12.1% of human ideas motivated by the pattern of connection, and only 5.1% use synthesis or unification as the central method paradigm. By contrast, across the nine main evaluated LLMs, the corresponding rates range from 47.1% to 64.2% and from 22.5% to 38.7%, respectively. Human ideas also exhibit consistently higher normalized entropy on both taxonomy axes. This pattern remains stable across model families and scientific domains, indicating that current LLM ideation is disproportionately concentrated around integrative and synthesis-oriented types, while human research ideas span a substantially broader range of opportunity patterns and methodological paradigms. ## 2Related Work #### LLMs for Research Ideation. Recent work has explored LLMs for scientific ideation, including generating, refining, and evaluating research hypotheses and directions. Early studies show that directly prompted LLMs can produce ideas perceived as highly novel, though often less feasible or well-grounded than human proposals(Si et al.,2025a (https://arxiv.org/html/2607.01233#bib.bib1)). Building on this, subsequent work has explored iterative refinement, retrieval-augmented generation, and search-based ideation pipelines(Wang et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib16); Baek et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib2); Sanyal et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib17)). Other approaches ground generation in external scientific signals such as retrieved literature, knowledge graphs, or emerging research trends(Ghafarollahi and Buehler,2024 (https://arxiv.org/html/2607.01233#bib.bib18); Hu et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib19); Pu et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib20)). Multi-agent collaboration has also become a common paradigm for improving idea diversity and critique through simulated scientific discussion(Gu et al.,2024 (https://arxiv.org/html/2607.01233#bib.bib21); Su et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib22)). Benchmarks are proposed to evaluate dimensions including novelty, feasibility, and impact of generated ideas(Ruan et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib23); Guo et al.,2025a (https://arxiv.org/html/2607.01233#bib.bib24); Liu et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib25)). Our work differs from this line of research by studying whether the overalldistributionof LLM-generated ideas resembles the distribution of ideas realized in human-written scientific papers. #### Gaps between Human and LLM-Generated Content. Even when LLM outputs appear fluent and useful, research shows that they can differ systematically from human outputs. Early detection work found that neural generations contain statistical artifacts in token ranks, sampling behavior, and likelihood geometry that distinguish them from human text(Gehrmann et al.,2019 (https://arxiv.org/html/2607.01233#bib.bib26); Ippolito et al.,2020 (https://arxiv.org/html/2607.01233#bib.bib27); Mitchell et al.,2023 (https://arxiv.org/html/2607.01233#bib.bib28)); broader comparison corpora and distributional metrics similarly show measurable gaps between ChatGPT or neural text and human expert writing(Pillutla et al.,2021 (https://arxiv.org/html/2607.01233#bib.bib29); Guo et al.,2023 (https://arxiv.org/html/2607.01233#bib.bib30)). These gaps become more consequential when LLM outputs are used as substitutes for human populations or human judgments. In social simulation, LLMs can reproduce some aggregate patterns while still exhibiting distortions or demographic misalignment relative to real human responses(Aher et al.,2023 (https://arxiv.org/html/2607.01233#bib.bib31); Santurkar et al.,2023 (https://arxiv.org/html/2607.01233#bib.bib32)). In evaluation and review settings, LLM judges vary substantially across tasks and require validation against human annotations, while LLM-generated paper reviews over-focus on technical validity and under-attend to novelty compared with expert reviewers(Bavaresco et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib33); Shin et al.,2025 (https://arxiv.org/html/2607.01233#bib.bib34)). Our work follows this perspective of comparison between human and LLMs, shifting the target to the distribution of research ideas. ## 3Evaluation Framework for Ideation We propose an idea-level evaluation framework to systematically analyze what kinds of research opportunities and contribution strategies LLMs emphasize or overlook during scientific ideation. We first define a literature-grounded ideation task and construct paired human and LLM idea corpora accordingly. We represent each idea through its motivation and method, annotate these ideas with a research-taste taxonomy, and analyze. ### 3.1Literature-Grounded Ideation Task Each instance contains a set of related prior worksXi={(ti1,ai1),…,(tik,aik)}X_{i}=\{(t_{i1},a_{i1}),\ldots,(t_{ik},a_{ik})\}, withttdenoting the title andaathe abstract. The target output is a research ideayi=(mi,si)y_{i}=(m_{i},s_{i}), wheremim_{i}refers to the motivation andsis_{i}stands for the proposed method. Guided by the provided literature context, the prompt directs models to identify research gaps across papers and generate a coherent research idea. This task is constrained. In an open-ended ideation setting, differences between human and model outputs can be confounded by topic selection, prior knowledge, and generic paper-writing templates, making the comparison difficult to interpret(Si et al.,2025a (https://arxiv.org/html/2607.01233#bib.bib1); Ruan et al.,2026 (https://arxiv.org/html/2607.01233#bib.bib23)). By anchoring both human and model ideas to the same set of related prior works, we focus the analysis on how