提炼、辩论、发现：以图谱蒸馏洞见为基础的多智能体研究构思

# Distill, Debate, Discover: Grounding Multi-Agent Research Ideation in Graph-Distilled Insights
Source: [https://arxiv.org/html/2610.33366](https://arxiv.org/html/2610.33366)
Muxin Liu\*Grace Kim\*Xiao LiangChris Callison\-BurchAffiliation:University of PennsylvaniaAffiliation:\{qianranm, muxinliu, yeeunk\}@engineering\.upenn\.edu

###### Abstract

Large language models \(LLMs\) are increasingly used to support research idea generation and scientific discovery\. Multi\-agent debate offers a collaborative approach to this task by bringing together diverse perspectives to develop research ideas\. However, a key challenge is how to equip agents with distinct and meaningful expertise to guide their contributions\. Inspired by how researchers draw on accumulated research experience during collaboration, we introduceInsightAgents, a framework that constructs literature\-grounded agent expertise by distilling insights from graphs of related papers\. Guided by insights that trace evolving research approaches, reveal underlying assumptions, and identify unresolved challenges, within a line of research, agents jointly identify a research problem and propose, challenge, and refine potential approaches through debate to develop a research proposal\. We evaluateInsightAgentsagainst three research idea generation frameworks on 100 NLP research topics through pairwise comparisons by two LLM judges\. In both online and offline knowledge settings,InsightAgentsachieves the highest average score across the five evaluation dimensions under both judges\.

\*\*footnotetext:Equal contribution\.††footnotetext:Code available at[https://github\.com/MichelleMa8/InsightAgents](https://github.com/MichelleMa8/InsightAgents)\.## 1Introduction

Recent research frameworks use large language models \(LLMs\) to automate multiple stages of scientific work, from literature review and idea development to experimentation and paper writing\([Yamada et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib11);[Schmidgall et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib18);[Tang et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib19)\)\. The quality of an initial research idea can shape subsequent research outcomes\([Si et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib6)\)\. We therefore focus on generating research ideas grounded in relevant literature through multi\-agent collaboration\.

Prior work has explored different ways of using literature and agent collaboration for research idea generation\. ResearchAgent retrieves related papers and entity knowledge to support idea generation, with reviewing agents providing feedback\([Baek et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib8)\)\. Chain of Ideas organizes papers into development chains and uses their trends to guide idea generation\([Li et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib9)\)\. Multi\-agent approaches also vary in how they initialize agents: VirSci uses real researcher profiles, ResearchTown constructs researcher profiles from papers, and Ueda et al\. explore assigning domain\-specific personas to agents\([Su et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib10);[Yu et al\., 2025a](https://arxiv.org/html/2610.33366#bib.bib17);[Ueda et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib4)\)\. These works illustrate literature\-guided idea generation and profile\- or persona\-based agent initialization\. However, few approaches initialize debate agents with insights distilled from different lines of research\.

Researchers develop expertise by tracing how lines of research evolve across related papers, learning from prior approaches, their assumptions, and unresolved challenges\([Boote and Beile, 2005](https://arxiv.org/html/2610.33366#bib.bib1);[Li et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib9)\)\. Inspired by this process, we equip agents with research perspectives grounded in the literature\. We define a*research insight*as a synthesis of how a line of research has evolved, capturing its key approaches, underlying assumptions, supporting findings, and unresolved tensions\.

Figure 1:Overview ofInsightAgents\. Insights distilled from graph\-structured literature define agent expertise, with each insight linked to its supporting evidence\. The agents use these perspectives to develop a research proposal through multi\-agent debate\.To support research idea generation through literature\-grounded multi\-agent debate, we introduceInsightAgents, a framework that uses distilled research insights to initialize expert agents for multi\-agent debate\. Given a research topic,InsightAgentsbuilds a graph of related papers and distills multiple research insights, each linked to supporting findings and source papers\. Each expert agent is initialized with one insight and its supporting findings, which guide the agents as they identify a research problem and propose, challenge, and refine approaches through debate\. A moderator coordinates the discussion, while a critic provides feedback on draft proposals to guide further refinement\. Together, the agents develop a research proposal specifying the problem and motivation, proposed method, and evaluation plan\. The overall process is illustrated in Figure[1](https://arxiv.org/html/2610.33366#S1.F1)\.

We evaluateInsightAgentson 100 NLP research topics, comparing it with ResearchAgent, AI Scientist\-v2, and Chain of Ideas\. Two LLM judges assess the proposals through pairwise comparisons on novelty, significance, feasibility, clarity, and expected effectiveness\.InsightAgentsachieves the highest average score among the compared methods under both judges\.

Our contributions are as follows:

- •We define*research insights*as syntheses of how lines of research evolve and distill them from a topic\-specific paper graph, together with supporting findings, to initialize expert agents\.
- •We introduceInsightAgents, a framework that brings these experts together to identify research problems and propose and refine approaches through multi\-agent debate, ultimately producing a research proposal\.
- •We evaluateInsightAgentson 100 NLP research topics, where it achieves the highest average score among the compared methods in two knowledge settings under two LLM judges\.

## 2Related Work

Table 1:Comparison with research idea generation frameworks\. Shaded framework names indicate the three baselines used in our experiments\.✓\\checkmark: documented support; —: not documented or no official code located\. Open source denotes public official code\. For topic control,✓\\checkmarkdenotes direct topic\-to\-idea input;∼\\simdenotes indirect or constrained control\. Literature graphs include chains and citation or knowledge graphs\. Insight extraction denotes intermediate trends, patterns, gaps, or transferable mechanisms\. Debate requires agents to exchange and respond to arguments, beyond independent reviews\. Agent expertise source describes what defines the expertise of debating agents; — marks frameworks without debate\.##### Literature\-grounded research idea generation

Prior work has explored different ways of grounding research idea generation in scientific literature\. SciMON, ResearchAgent, and AI Scientist\-v2 retrieve relevant papers to inform idea generation and refinement\([Wang et al\., 2024](https://arxiv.org/html/2610.33366#bib.bib7);[Baek et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib8);[Yamada et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib11)\)\. Other methods organize literature into chains or graphs to capture research developments and relationships among scientific knowledge\([Li et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib9);[Gao et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib14);[Li et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib13);[Xu et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib15)\)\. While retrieval\-based methods typically do not explicitly trace how research evolves, structure\-aware methods primarily use these structures to guide idea generation, rather than explore how different lines of research might challenge or complement one another\. In contrast,InsightAgentsgrounds multi\-agent debate in insights distilled from distinct lines of research, allowing these perspectives to interact during idea development\.

##### Multi\-agent research ideation

Multi\-agent debate allows LLM agents to exchange and critique ideas, encouraging diverse perspectives and iterative refinement\([Du et al\., 2024](https://arxiv.org/html/2610.33366#bib.bib23);[Liang et al\., 2024](https://arxiv.org/html/2610.33366#bib.bib24)\)\. Existing scientific collaboration frameworks introduce agent diversity through functional roles or domain\-specific personas\([Ghafarollahi and Buehler, 2025](https://arxiv.org/html/2610.33366#bib.bib16);[Ueda et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib4)\), researcher profiles\([Su et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib10);[Yu et al\., 2025a](https://arxiv.org/html/2610.33366#bib.bib17)\), or different knowledge backgrounds\([Yu et al\., 2025b](https://arxiv.org/html/2610.33366#bib.bib12)\)\. However, these forms of diversity do not equip agents with distinct and topic\-specific expertise grounded in prior research findings and assumptions\. In contrast,InsightAgentsinitializes each expert with insights distilled from a different line of research, providing a concrete and literature\-grounded basis for proposing and challenging ideas during debate\. Table[1](https://arxiv.org/html/2610.33366#S2.T1)summarizes these differences across research idea generation frameworks\.

## 3Method

InsightAgentstransforms insights distilled from graph\-structured literature into agent expertise for research idea generation\. As shown in Figure[1](https://arxiv.org/html/2610.33366#S1.F1), the framework comprises two components: graph\-based expertise construction and multi\-agent debate\.

### 3\.1Constructing Agent Expertise

##### Paper graph construction

We construct a paper graphG=\(V,E\)G=\(V,E\)to organize related literature for insight distillation\. The seed nodev0v\_\{0\}is either a user\-specified paper or one paper retrieved for the input topic\. Starting fromv0v\_\{0\}, we expand each nodevvthrough Semantic Scholar’s recommendation API, which returns a one\-hop candidate set𝒞⁡\(v\)\\mathcal\{C\}\(v\)of papers similar tovv\. Among new candidates𝒞⁡\(v\)∖V\\mathcal\{C\}\(v\)\\setminus V, an LLM selects papers relevant to both the topic andvv, favoring distinct methods, settings, or failure modes\. Candidates already inVVare linked directly tovv, while selected new papers are added toVVand connected tovvthrough new edges inEE\. We orient recommendation links from older to newer papers by publication date\. We recursively expand newly added nodes for three rounds, yielding up to three expansion levels beyondv0v\_\{0\}\. The resulting graph collects related papers to the topic for methodological comparison and insight distillation\.

##### Insight distillation

Given the paper graphG=\(V,E\)G=\(V,E\), we prompt an LLM to jointly distill three research insightsℐ=\{I1,I2,I3\}\\mathcal\{I\}=\\\{I\_\{1\},I\_\{2\},I\_\{3\}\\\}using the paper titles and abstracts stored in its nodes and the relationships between papers represented by its edges\. The LLM first groups papers by the type of research they pursue, such as measurement, training, or dataset construction, and selects three distinct lines of work, each supported by at least two papers\. For each line, it synthesizes recurring challenges, the evolution of research approaches, underlying assumptions, and unresolved tensions into an insightIkI\_\{k\}\. Each insight is grounded in a supporting subgraphGk=\(Vk,Ek\)G\_\{k\}=\(V\_\{k\},E\_\{k\}\)and a set of findingsℱk\\mathcal\{F\}\_\{k\}, retaining specific methods, benchmarks, and reported results linked to their source papers\.

##### Expertise formation

For each research insightIkI\_\{k\}, we initialize an expert agentAkA\_\{k\}with a system prompt containing the insight, its supporting findingsℱk\\mathcal\{F\}\_\{k\}, and the titles and abstracts of papers inVkV\_\{k\}\. The insight provides the agent with a perspective on a line of research, while the findings and source abstracts ground this perspective in specific methods, benchmarks, and reported results\. During debate, the agent is instructed to reason from this perspective while drawing on its broader knowledge to propose and evaluate research approaches\.

### 3\.2Research Idea Generation through Multi\-Agent Debate

The debate involves three roles: expert agents, a moderator, and a critic\. Each expert agentAkA\_\{k\}draws on its insightIkI\_\{k\}and supporting findings to identify research problems, propose approaches, and evaluate the shared proposal\. The moderator coordinates the discussion and integrates expert contributions into a common draft\. To encourage critical examination rather than premature consensus\([Pitre et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib3)\), the critic acts as an independent reviewer, identifying weaknesses in the current draft\. It receives the draft and the latest discussion summary, without access to the full debate history, and provides feedback to guide subsequent expert revisions\. Together, these roles develop one research proposal through the following three stages\.

##### Stage 1: Problem selection

Each expert proposes a research problem based on its insight\. The moderator consolidates these proposals into three anonymous candidates, which are shuffled before the experts rank them independently\. Rankings are aggregated using Borda scoring\([Young, 1974](https://arxiv.org/html/2610.33366#bib.bib2)\), with 3, 2, and 1 points for first, second, and third place, respectively\. The moderator summarizes the highest\-scoring problem for subsequent debate, choosing among the tied candidates only when necessary\.

##### Stage 2: Proposal development and debate

For the selected problem, experts develop a shared proposal through three types of rounds: proposal, refinement, and convergence\. In the proposal round, each expert proposes an approach informed by its expertise and, where useful, broader knowledge\. The moderator identifies the strongest direction supported across the proposals, combining complementary approaches, and produces an initial draft\. The critic then reviews the draft for scientific contribution, soundness, consistency, feasibility, and clarity\.

In subsequent refinement rounds, experts respond to the critic’s feedback, proposing improvements to the method, claims, or evaluation\. The moderator updates the draft and summarizes resolved, unresolved, and newly raised issues\. The critic reviews each revised draft and identifies issues to be addressed in the next round\.

In the final convergence round, experts address remaining concerns and state their final positions without introducing new research directions\.

##### Stage 3: Final synthesis

After the debate, the moderator integrates the latest draft and the experts’ final positions into a final research proposal\. It retains the selected research problem, incorporates the methods the experts agreed on during debate, and reports unresolved concerns as limitations\. The final proposal specifies the problem and motivation, method, experiment, anticipated novelty, and potential impact\.

## 4Experimental Setup

### 4\.1Pipeline Setup

Table 2:Comparison of the two knowledge settings used byInsightAgents\. In the offline setting, contribution nodes are mapped to their source papers\.##### Knowledge settings

We evaluateInsightAgentsunder two knowledge settings that reflect different ways of accessing prior research: retrieving papers on demand from an evolving body of literature and exploring a fixed and structured knowledge resource\.

In theonlinesetting, we dynamically construct a paper graph using the Semantic Scholar API, as described in Section[3](https://arxiv.org/html/2610.33366#S3)\. In theofflinesetting, we use a preconstructed scientific contribution graph\([Jansen, 2026](https://arxiv.org/html/2610.33366#bib.bib5)\)and map its contribution nodes to their source papers, preserving the prerequisite relationships between contributions as paper\-level connections\. Table[2](https://arxiv.org/html/2610.33366#S4.T2)summarizes the key differences between the two settings\. Both use the same insight distillation, expertise formation, and debate procedures, while graph retrieval and expansion differ\. Evaluating both settings allows us to examine whetherInsightAgentsremains effective across different knowledge sources and graph structures\. Appendix[B](https://arxiv.org/html/2610.33366#A2)details the offline graph retrieval process and the evidence provided to agents\.

Since the settings differ in their underlying corpora, graph construction procedures, edge semantics, and seed selection, we treat them as two configurations ofInsightAgentsrather than a controlled comparison of knowledge sources\.

##### Topic selection

We evaluate our framework on the same set of 100 NLP research topics in both knowledge settings\. To construct the candidate topic pool, we randomly sample papers from the 2025 main proceedings and available Findings volumes of ACL, EMNLP, NAACL, and COLING\. We use GPT\-5\.6 Luna to derive research topics from their titles and abstracts, exclude unsuitable source papers \(e\.g\., surveys\), and remove semantic duplicates\. From the resulting pool, we randomly select 100 evaluation topics spanning 20 research areas\. Appendix[A](https://arxiv.org/html/2610.33366#A1)describes the sampling, abstraction, filtering, and deduplication rules and reports the area distribution\.

##### Generation configuration

We use GPT\-5\.6 Luna for insight extraction and all debate roles inInsightAgents\. Each topic is processed by three insight\-based experts, a moderator, and a critic\. Problem selection uses two rounds, followed by three debate rounds and a final rewrite that produces one research proposal\. Graph construction parameters and generation budgets are provided in Appendix[C](https://arxiv.org/html/2610.33366#A3)\. Prompt templates are provided in Appendix[H](https://arxiv.org/html/2610.33366#A8)\.

### 4\.2Baselines

We select baselines from the frameworks in Table[1](https://arxiv.org/html/2610.33366#S2.T1)that have public code and can generate ideas for a given topic, either from the topic itself or from a single seed paper\. Three frameworks meet both criteria: ResearchAgent\([Baek et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib8)\), AI Scientist\-v2\([Yamada et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib11)\), and Chain of Ideas\([Li et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib9)\)\. AI Scientist\-v2 and Chain of Ideas take the topic directly; ResearchAgent takes the seed paper selected byInsightAgentsin the online setting, with the topic text added to its problem prompts\. The multi\-agent frameworks in Table[1](https://arxiv.org/html/2610.33366#S2.T1)do not meet these criteria\. VirSci forms a research team that chooses its own topic, SciAgents generates hypotheses from keywords in a fixed concept graph, and IDVSCI has no public code\. Adapting them to a given topic would change their task, so we do not include them as baselines\.

We evaluate all three baselines using the same 100 research topics and generation model \(GPT\-5\.6 Luna\) asInsightAgents\. We use the official implementations with default pipeline settings, except for the adaptations detailed in Appendix[D](https://arxiv.org/html/2610.33366#A4)\. Since we compare complete pipelines, we do not control for generation costs across methods\.

### 4\.3Evaluation Protocol

##### LLM evaluation

We adopt pairwise evaluation, as prior studies suggest that LLM judges align more closely with human judgments when comparing outputs than when assigning absolute scores\([Liusie et al\., 2024](https://arxiv.org/html/2610.33366#bib.bib20);[Liu et al\., 2024](https://arxiv.org/html/2610.33366#bib.bib21)\)\. Following Chain of Ideas\([Li et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib9)\), we evaluate research ideas along five dimensions: novelty, significance, feasibility, clarity, and expected effectiveness\. Table[39](https://arxiv.org/html/2610.33366#A8.T39)provides detailed definitions of these dimensions and the evaluation prompt\.

We use GPT\-5\.6 Sol and Claude Opus 4\.8 as judges, with the same evaluation prompt across methods, and report their results separately\. For each topic, the judge compares every pair of methods and selects a winner or a tie on each dimension, with a brief rationale\. To reduce position and stylistic biases, we evaluate each pair in both presentation orders and instruct the judge to focus on content rather than surface features such as length and formatting\([Feuer et al\., 2025](https://arxiv.org/html/2610.33366#bib.bib22)\)\.

##### Human–LLM agreement

To assess agreement between human and LLM judgments, two NLP researchers independently evaluate proposals on 10 research topics selected based on their familiarity with the topics\. For each topic, annotators rank the four proposals fromInsightAgents\(online setting\) and the three baselines along the same five evaluation dimensions, allowing ties\. Annotators receive the same criteria and instructions as the LLM judges, and method identities are hidden during annotation\. We derive pairwise preferences from these rankings and measure both inter\-annotator and human–LLM agreement\.

## 5Results

Figure 2:Elo profiles on 100 topics per setting: \(a–b\) online and \(c–d\) offline, each evaluated by GPT\-5\.6 Sol and Claude Opus 4\.8\. Ours denotesInsightAgents; Avg\., Signif\., Feasib\., and Eff\. denote average Elo, significance, feasibility, and expected effectiveness\. All panels use the same radial scale \(250–1,650; higher is better\)\.##### Main results

We aggregate pairwise judgments into Elo ratings for each evaluation dimension, knowledge setting, and judge, with equal weights across the five dimensions for the average Elo\. Figure[2](https://arxiv.org/html/2610.33366#S5.F2)presents the comparative evaluation results\.

InsightAgentsachieves the highest average Elo in all four combinations of knowledge setting and judge, followed by AI Scientist\-v2\. Both judges rankInsightAgentsfirst in novelty, significance, and expected effectiveness under both settings\.

Feasibility represents the main trade\-off, with AI Scientist\-v2 ranking first andInsightAgentssecond across both settings and judges\. Appendix[E\.2](https://arxiv.org/html/2610.33366#A5.SS2)further examines the judges’ rationales for this gap in the online setting\. Clarity rankings vary by judge: GPT\-5\.6 Sol ranksInsightAgentsfirst, while Claude Opus 4\.8 favors AI Scientist\-v2, placingInsightAgentssecond in both settings\. Detailed Elo ratings and direct pairwise comparison scores are provided in Appendix[E](https://arxiv.org/html/2610.33366#A5)\.

The consistent results across both settings suggest thatInsightAgentscan effectively leverage different forms of literature\-grounded knowledge without relying on a specific graph construction procedure\.

##### Human–LLM agreement

Figure[3](https://arxiv.org/html/2610.33366#S5.F3)shows thatInsightAgentsalso achieves the highest average Elo on the human\-evaluated topics, with the largest margins over AI Scientist\-v2 in novelty, significance, and expected effectiveness\. The two methods receive similar ratings for clarity and feasibility, withInsightAgentsslightly ahead on clarity and AI Scientist\-v2 on feasibility\. These results support the main pattern of the LLM evaluation on this subset\.

Figure 3:Human Elo ratings on 10 online topics, derived from rankings across all four methods\. Ours denotesInsightAgents\. Axes follow Figure[2](https://arxiv.org/html/2610.33366#S5.F2); the radial scale is 800–1,250\.Figure 4:Agreement onInsightAgents–baseline pairs over 150 topic–dimension–pair instances from 10 online topics\. Each bar splits one comparison into exact agreement \(same winner or both ties\), tie vs\. preference \(one judge returns a tie and the other does not\), and opposite preference\. Non\-contradiction is the sum of the two blue segments\. Sol and Opus denote GPT\-5\.6 Sol and Claude Opus 4\.8\.We compare human and LLM judgments onInsightAgents–baseline pairs using two measures\.*Exact agreement*requires both judges to select the same winner or both declare a tie\.*Non\-contradiction*counts all cases except those where the two judges prefer opposite methods, including cases where both select the same winner, both declare a tie, or one declares a tie while the other selects a winner\. Figure[4](https://arxiv.org/html/2610.33366#S5.F4)shows that opposing preferences account for only 7\.3–10\.0% of human–LLM comparisons, yielding non\-contradiction rates of 90\.0–92\.7%, close to the 93\.3% between the two LLM judges\. Exact agreement is lower \(63\.7–65\.0%\), with most disagreements involving ties rather than opposing preferences\. Human annotators show a similar pattern, with tie–preference disagreements \(46\.0%\) exceeding exact agreement \(44\.0%\)\. Appendix[E\.4\.2](https://arxiv.org/html/2610.33366#A5.SS4.SSS2)provides confidence intervals and dimension\-level results\.

## 6Analysis

### 6\.1Ablation Study

We ablate three steps ofInsightAgents: insight distillation, retrieval grounding, and multi\-agent debate\. Each variant changes one step and keeps the rest of the pipeline fixed\.

*w/o insight*replaces each expert’s insightIkI\_\{k\}and findingsℱk\\mathcal\{F\}\_\{k\}with the titles and abstracts of its supporting papersVkV\_\{k\}\. The paper graphGGis unchanged\.*w/o retrieval*removes retrieval that the generation model writes three insights and their findings from its own knowledge, in the same format and length asInsightAgents\.*w/o debate*provides all three insights to a single agent, which selects a research problem, drafts a proposal, and self\-reviews and revises it using the same five criteria as the critic in the full system\. This is completed in a single response without multi\-round discussion or a separate critic, followed by the same final rewrite step to maintain consistent output format and length\.

We run each variant on 30 topics from the online setting with the same generation model\. Claude Opus 4\.8 compares each variant’s proposal againstInsightAgents’s on every dimension\. We report each variant’s win rate againstInsightAgents, counting ties as half a win\. Scores below 50% indicate that the removed component benefitsInsightAgents\.

Table 3:Ablation results againstInsightAgents, which scores 50 by construction\. Scores below 50 indicate that the removed step benefits the full framework\.Table[3](https://arxiv.org/html/2610.33366#S6.T3)shows that insight distillation and retrieval both improve novelty and significance without reducing effectiveness, because they give the experts more context about prior work\. w/o insight and w/o retrieval score slightly higher on feasibility and clarity because they propose simpler ideas\. Counting datasets, baselines, experimental designs, and human annotations, they propose 8\.6 and 7\.5 experimental components, against 9\.1 forInsightAgents\. w/o retrieval also cites fewer and older works: 1\.9 works from 2021 on average, against 3\.6 from 2026 forInsightAgents\. The judge knows this older work well and finds these proposals easier to follow\. As a result, w/o retrieval reaches an average score close toInsightAgents’s, but its ideas are poorly grounded and partly favored for their familiarity to the judge\.InsightAgentsinstead prioritizes pushing novelty and significance over ease of execution \(Appendix[E\.2](https://arxiv.org/html/2610.33366#A5.SS2)\)\. Removing debate leads to the largest overall performance drop \(33\.0\), particularly in novelty \(19\.2\) and expected effectiveness \(11\.7\), while feasibility remains nearly unchanged \(51\.7\)\. This suggests that multi\-agent discussion and iterative refinement help develop more novel and potentially effective research proposals\.

### 6\.2Generation Process Analysis

##### Evidence diversity and use

We first examine how literature\-grounded expertise is distributed and used across 100 online topics\. Among 300 within\-topic expert pairs, 289 share no supporting papers, indicating largely non\-overlapping evidence sets\. In the first proposal round, 299 of 300 expert proposals reference at least one paper from their own supporting sets, with an average of 4\.3 papers per proposal\. In 67 of 100 topics, the final proposal’sbuilds onfield references evidence from at least two experts\. These statistics characterize how distinct sources of evidence enter the proposal development process\.

##### Case study: Diversity of research insights

We examine the topicPersonalized dialogue generation through evolving user\-specific memoryto illustrate the distinct research perspectives distilled from the same paper graph\. Expert A \(Memory Utilization Evaluation\) focuses on the gap between retrieving user memories and appropriately integrating them into responses\([Sumida et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib30)\)\. Expert B \(Evolving Memory Architecture\) emphasizes how memory systems represent evolving evidence, including temporal changes, contradictions, and contextual validity\([Choi et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib37);[Liu et al\., 2026a](https://arxiv.org/html/2610.33366#bib.bib35);[Yacoubi et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib33)\)\. Expert C \(Continual Personalization Alignment\) focuses on the challenges of adapting to evolving user preferences while avoiding irrelevant personalization, preference narrowing, and sycophancy\([Wang et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib25)\)\. These insights provide complementary perspectives on evaluating memory use, representing evolving memory states, and regulating personalization\.

##### Case study: Expertise\-guided debate and refinement

To illustrate how experts interact during debate, we examine one of the issues they discussed: how to control the influence of user memory on generated responses\. Expert C initially proposed an influence budget to regulate the effects of uncertain preferences\. Expert B challenged the reliance on confidence alone, arguing that memory influence must also account for lifecycle states, such as whether a memory is current, superseded, or context\-specific\. The initial draft combined these perspectives but left the influence budget loosely defined\. In response, the experts jointly refined this mechanism: Expert A specified four discrete influence levels, Expert B reinforced their lifecycle constraints, and Expert C proposed applying these interventions to a shared candidate pool before generation\. The revised draft incorporated these contributions, turning the qualitative influence budget into explicit and testable operations\.

### 6\.3Comparison with Other Frameworks

Figure 5:LLM judges’ preferences by proposal property, based on their stated reasoning\. Each point shows the percentage of explanations favoringInsightAgentsover a baseline among those discussing that property\. Results pool both LLM judges, GPT\-5\.6 Sol and Claude Opus 4\.8 \(3,000 explanations each\), in the online setting\. Values above 50% \(dashed line\) favorInsightAgents\. See Appendix[E\.3](https://arxiv.org/html/2610.33366#A5.SS3)for details\.To understand why LLM judges preferInsightAgents, we analyze the reasoning they provide with their pairwise judgments\. We use Claude Opus 5\.5 to code 6,000 such explanations from the online setting, 3,000 from each of the two LLM judges, across seven proposal properties: originality \(new framing or mechanisms\), breadth \(generalizability\), focus \(one clear research question\), payoff \(perceived value of the result\), precision \(specified methods and success criteria\), evidence \(controls that rule out alternative explanations\), and executability \(feasibility given available resources\)\. The detailed descriptions of each property are included in Appendix[E\.3](https://arxiv.org/html/2610.33366#A5.SS3)\.

Figure[5](https://arxiv.org/html/2610.33366#S6.F5)shows that judges’ reasoning discussing originality or evidence favorsInsightAgentsacross all three baselines \(81\.9–98\.9%\)\. Judges highlight how its proposals reframe research problems and use experimental controls to test their claims\. Against Chain of Ideas and ResearchAgent, they also favor its focus and precision\. The main exception is executability against AI Scientist\-v2, where 80\.1% of relevant rationales favor the baseline\. Judges cite the scale ofInsightAgents’s proposed studies, human annotation, and custom experimental setups as obstacles\.

## 7Conclusion

We introducedInsightAgents, a framework for research idea generation through multi\-agent debate\. Its key feature is constructing agent expertise from insights distilled across related papers, giving each agent a distinct research perspective grounded in supporting findings\. Evaluations on 100 NLP research topics show thatInsightAgentsachieves the highest average Elo among the compared methods under both LLM judges in both online and offline knowledge settings\.

Our findings suggest that how agents acquire and use scientific knowledge is an important consideration in automated research\. Literature\-grounded expert debate could serve as an ideation component in larger automated research workflows, connecting prior research to collaborative idea development\.

## Limitations

Our evaluation covers only NLP research topics, leaving the applicability ofInsightAgentsto other scientific disciplines untested\. Moreover, sinceInsightAgentsstarts from a predefined topic and builds expertise from existing literature, its ability to independently discover emerging or entirely new research directions remains limited\.

InsightAgentsgenerates research proposals without executing the proposed experiments or incorporating experimental feedback\. Our evaluation therefore assesses the quality of research ideas before execution, rather than whether they lead to meaningful scientific findings in practice\. Evaluating ideas through experimental execution would provide stronger evidence of their research value\.

Finally, our evaluation relies primarily on LLM judges\. Although we include human evaluation to assess agreement with LLM judgments, it covers only a small subset of topics\. LLM\-based assessments may not fully reflect expert judgments of research quality\.

## Ethical considerations

Our framework relies on LLMs to synthesize scientific literature and generate research proposals\. Although the proposals are grounded in prior research, they may contain unsupported claims, inaccurate interpretations of existing work, or ideas that are not genuinely novel\. These proposals have not been experimentally validated and require expert review before use\. Our evaluation also relies primarily on LLM judges, which may favor particular writing styles or types of proposals\. To assess this potential bias, we report results from two LLM judges separately and compare their judgments with human rankings on a subset of topics\.

Automated research idea generation could also facilitate the large\-scale production of unvetted proposals or submissions\. We intendInsightAgentsto support researchers in exploring and critically assessing ideas, rather than replacing human judgment or scientific validation\.

##### Use of AI assistants

We used GPT\-5\.6 Luna for research topic generation and proposal development, GPT\-5\.6 Sol and Claude Opus 4\.8 for evaluation, and Claude Opus 5\.5 to categorize judges’ explanations\. We used ChatGPT and Claude to assist with manuscript writing and revision\. The authors reviewed the AI\-assisted content and code and take full responsibility for the paper\.

## Acknowledgments

This research was developed with funding from the Defense Advanced Research Projects Agency’s \(DARPA\) SciFy program \(Agreement No\. HR00112520300\) and is based upon work supported in part by the Office of the Director of National Intelligence \(ODNI\), Intelligence Advanced Research Projects Activity \(IARPA\), via 56000026C0019 \(the BENGAL program\)\. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNI, IARPA, the Department of Defense, or the U\.S\. Government\. The U\.S\. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein\.

## References

- Baeket al\.\(2025\)J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. HwangResearchAgent: iterative research idea generation over scientific literature with large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 6709–6738\.External Links:[Link](https://aclanthology.org/2025.naacl-long.342/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.342),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p2.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.4.1.1),[§4\.2](https://arxiv.org/html/2610.33366#S4.SS2.p1.1)\.
- Boote and Beile \(2005\)D\. N\. Boote and P\. BeileScholars before researchers: on the centrality of the dissertation literature review in research preparation\.Educational Researcher34\(6\),pp\. 3–15\.External Links:[Document](https://dx.doi.org/10.3102/0013189X034006003)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p3.1)\.
- Chang and Chen \(2026\)W\. Chang and Y\. ChenWhen users don’t ask: benchmarking context\-driven memory retrieval in conversational agents\.arXiv preprint arXiv:2609\.03467\.External Links:[Link](https://arxiv.org/abs/2609.03467)Cited by:[Table 15](https://arxiv.org/html/2610.33366#A7.T15.2.10.1.1.1)\.
- Choiet al\.\(2026\)W\. Choi, Y\. Kim, Y\. Lee, and S\. YoonPGMem: tightly coupled persona\-memory graph for lifelong personalized agents\.arXiv preprint arXiv:2608\.01708\.External Links:[Link](https://arxiv.org/abs/2608.01708)Cited by:[Table 16](https://arxiv.org/html/2610.33366#A7.T16.2.11.1.1.1),[§6\.2](https://arxiv.org/html/2610.33366#S6.SS2.SSS0.Px2.p1.1)\.
- Duet al\.\(2024\)Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. MordatchImproving factuality and reasoning in language models through multiagent debate\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 11733–11763\.External Links:[Link](https://proceedings.mlr.press/v235/du24e.html)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1)\.
- Feueret al\.\(2025\)B\. Feuer, M\. Goldblum, T\. Datta, S\. Nambiar, R\. Besaleli, S\. Dooley, M\. Cembalest, and J\. P\. DickersonStyle outweighs substance: failure modes of LLM judges in alignment benchmarking\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=MzHNftnAM1)Cited by:[§4\.3](https://arxiv.org/html/2610.33366#S4.SS3.SSS0.Px1.p2.1)\.
- Gaoet al\.\(2026\)S\. Gao, Y\. Xia, S\. Liu, and H\. XiongGraphs of research: citation evolution graphs as supervision for research idea generation\.arXiv preprint arXiv:2605\.14790\.External Links:[Link](https://arxiv.org/abs/2605.14790)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.10.1)\.
- Ghafarollahi and Buehler \(2025\)A\. Ghafarollahi and M\. J\. BuehlerSciAgents: automating scientific discovery through bioinspired multi\-agent intelligent graph reasoning\.Advanced Materials37\(22\),pp\. 2413523\.External Links:[Document](https://dx.doi.org/10.1002/adma.202413523),[Link](https://doi.org/10.1002/adma.202413523)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.7.1)\.
- Hwanget al\.\(2026\)E\. Hwang, K\. Mitra, D\. Zhang, H\. Kim, and E\. HruschkaHypotheses\-guided self distillation for continual personalization\.arXiv preprint arXiv:2609\.00251\.External Links:[Link](https://arxiv.org/abs/2609.00251)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1)\.
- Jansen \(2026\)P\. A\. JansenThe Scientific Contribution Graph: automated literature\-based technological roadmapping at scale\.arXiv preprint arXiv:2605\.15011\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.15011),[Link](https://arxiv.org/abs/2605.15011)Cited by:[Appendix B](https://arxiv.org/html/2610.33366#A2.p1.1),[§4\.1](https://arxiv.org/html/2610.33366#S4.SS1.SSS0.Px1.p2.1)\.
- Liet al\.\(2025\)L\. Li, W\. Xu, J\. Guo, R\. Zhao, X\. Li, Y\. Yuan, B\. Zhang, Y\. Jiang, Y\. Xin, R\. Dang, Y\. Rong, D\. Zhao, T\. Feng, and L\. BingChain of ideas: revolutionizing research via novel idea development with LLM agents\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8971–9004\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.477/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.477),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p2.1),[§1](https://arxiv.org/html/2610.33366#S1.p3.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.6.1.1),[§4\.2](https://arxiv.org/html/2610.33366#S4.SS2.p1.1),[§4\.3](https://arxiv.org/html/2610.33366#S4.SS3.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)X\. Li, H\. Tu, and X\. HanGraph2Idea: retrieval\-augmented scientific idea generation with graph\-structured contexts\.arXiv preprint arXiv:2606\.09105\.External Links:[Link](https://arxiv.org/abs/2606.09105)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.11.1)\.
- Lianget al\.\(2024\)T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. TuEncouraging divergent thinking in large language models through multi\-agent debate\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17889–17904\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.992/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.992)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1)\.
- Liaoet al\.\(2026\)Y\. Liao, L\. Wu, M\. Hou, H\. Liu, H\. Wu, and Z\. WangLeanMem: simple and efficient long\-term memory for LLM agents\.arXiv preprint arXiv:2608\.03463\.External Links:[Link](https://arxiv.org/abs/2608.03463)Cited by:[Table 16](https://arxiv.org/html/2610.33366#A7.T16.2.11.1.1.1)\.
- Liuet al\.\(2026a\)J\. Liu, H\. Sun, Y\. Wang, B\. Zhang, D\. Sui, D\. Chu, and Z\. TuEMIR2\{\}^\{2\}: evolution\-aware memory with intent\-guided multi\-round retrieval\.arXiv preprint arXiv:2609\.32584\.External Links:[Link](https://arxiv.org/abs/2609.32584)Cited by:[Table 16](https://arxiv.org/html/2610.33366#A7.T16.2.11.1.1.1),[§6\.2](https://arxiv.org/html/2610.33366#S6.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026b\)M\. Liu, Q\. Qiu, Y\. Wang, H\. Shen, and X\. ChengLess is personal: learning minimal sufficient user profiles for personalized language models\.arXiv preprint arXiv:2609\.08180\.External Links:[Link](https://arxiv.org/abs/2609.08180)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1)\.
- Liuet al\.\(2024\)Y\. Liu, H\. Zhou, Z\. Guo, E\. Shareghi, I\. Vulić, A\. Korhonen, and N\. CollierAligning with human judgement: the role of pairwise preference in large language model evaluators\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=9gdZI7c6yr)Cited by:[§4\.3](https://arxiv.org/html/2610.33366#S4.SS3.SSS0.Px1.p1.1)\.
- Liusieet al\.\(2024\)A\. Liusie, P\. Manakul, and M\. GalesLLM comparative assessment: zero\-shot NLG evaluation through pairwise comparisons using large language models\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),St\. Julian’s, Malta,pp\. 139–151\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.8),[Link](https://aclanthology.org/2024.eacl-long.8/)Cited by:[§4\.3](https://arxiv.org/html/2610.33366#S4.SS3.SSS0.Px1.p1.1)\.
- Pitreet al\.\(2025\)P\. Pitre, N\. Ramakrishnan, and X\. WangCONSENSAGENT: towards efficient and effective consensus in multi\-agent LLM interactions through sycophancy mitigation\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 22112–22133\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1141/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1141)Cited by:[§3\.2](https://arxiv.org/html/2610.33366#S3.SS2.p1.1)\.
- Qinget al\.\(2026\)P\. Qing, F\. Shi, and S\. VosoughiUTILMEM: benchmarking evidence utilization in long\-term conversational memory\.arXiv preprint arXiv:2608\.30508\.External Links:[Link](https://arxiv.org/abs/2608.30508)Cited by:[Table 15](https://arxiv.org/html/2610.33366#A7.T15.2.10.1.1.1)\.
- Schmidgallet al\.\(2025\)S\. Schmidgall, Y\. Su, Z\. Wang, X\. Sun, J\. Wu, X\. Yu, J\. Liu, M\. Moor, Z\. Liu, and E\. BarsoumAgent laboratory: using LLM agents as research assistants\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5977–6043\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.320/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.320),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p1.1)\.
- Shenet al\.\(2026\)J\. Shen, K\. Mao, M\. Shao, C\. Jin, Y\. Wang, A\. Lin, K\. Funakoshi, M\. Okumura, T\. Shu, and M\. ShafiqueHyperTrace: hypothesis\-based preference tracing for online LLM personalization\.arXiv preprint arXiv:2609\.09835\.External Links:[Link](https://arxiv.org/abs/2609.09835)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1)\.
- Siet al\.\(2025\)C\. Si, T\. Hashimoto, and D\. YangThe ideation–execution gap: execution outcomes of LLM\-generated versus human research ideas\.arXiv preprint arXiv:2506\.20803\.External Links:[Link](https://arxiv.org/abs/2506.20803)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p1.1)\.
- Songet al\.\(2026\)J\. Song, B\. Li, H\. Yi, Z\. Ji, M\. Zhang, H\. Fei, M\. Zhang, M\. Lee, and W\. HsuFrom profiling to synthesis: benchmarking implicit behavioral alignment in personalized LLM agents\.arXiv preprint arXiv:2608\.02171\.External Links:[Link](https://arxiv.org/abs/2608.02171)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1)\.
- Suet al\.\(2025\)H\. Su, R\. Chen, S\. Tang, Z\. Yin, X\. Zheng, J\. Li, B\. Qi, Q\. Wu, H\. Li, W\. Ouyang, P\. Torr, B\. Zhou, and N\. DongMany heads are better than one: improved scientific idea generation by a LLM\-based multi\-agent system\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 28201–28240\.External Links:[Link](https://aclanthology.org/2025.acl-long.1368/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1368),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p2.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.8.1)\.
- Sumidaet al\.\(2026\)R\. Sumida, K\. Inoue, and T\. KawaharaMemUse: moving memory evaluation from direct QA to natural integration in long\-term human\-AI conversation\.arXiv preprint arXiv:2608\.24189\.External Links:[Link](https://arxiv.org/abs/2608.24189)Cited by:[Table 15](https://arxiv.org/html/2610.33366#A7.T15.2.10.1.1.1),[§6\.2](https://arxiv.org/html/2610.33366#S6.SS2.SSS0.Px2.p1.1)\.
- Tanget al\.\(2025\)J\. Tang, L\. Xia, Z\. Li, and C\. HuangAI\-Researcher: autonomous scientific innovation\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 10477–10516\.External Links:[Document](https://dx.doi.org/10.52202/085713-0320),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/0d904d300a105809a2114d727851e759-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p1.1)\.
- Uedaet al\.\(2025\)K\. Ueda, W\. Hirota, T\. Asakura, T\. Omi, K\. Takahashi, K\. Arima, and T\. IshigakiExploring the design of multi\-agent LLM dialogues for research ideation\.InProceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue,Avignon, France,pp\. 322–337\.External Links:[Link](https://aclanthology.org/2025.sigdial-1.26/),[Document](https://dx.doi.org/10.18653/v1/2025.sigdial-1.26)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p2.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024\)Q\. Wang, D\. Downey, H\. Ji, and T\. HopeSciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 279–299\.External Links:[Link](https://aclanthology.org/2024.acl-long.18/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.18)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.3.1)\.
- Wanget al\.\(2026\)Y\. Wang, Y\. Wu, C\. Qian, Z\. Fan, H\. Ha, S\. Wu, J\. Liu, H\. Ji, and G\. WangEvaluating the hidden costs of personalization in large language models\.arXiv preprint arXiv:2608\.28833\.External Links:[Link](https://arxiv.org/abs/2608.28833)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1),[§6\.2](https://arxiv.org/html/2610.33366#S6.SS2.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026\)T\. Xu, Z\. Qian, G\. Liu, L\. Ling, Z\. Zhang, B\. Wu, S\. Zhang, K\. Lu, W\. Shi, Z\. Wang, Z\. Feng, Y\. Luo, S\. Xu, Y\. Chen, Z\. Feng, Z\. Chen, B\. Yuan, H\. Wang, and K\. ChenIdea2Story: an automated pipeline for transforming research concepts into complete scientific narratives\.arXiv preprint arXiv:2601\.20833\.External Links:[Link](https://arxiv.org/abs/2601.20833)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1)\.
- Yacoubiet al\.\(2026\)M\. Yacoubi, P\. Schmidt, N\. Petrovic, A\. Frikha, M\. Kirchhoff, and A\. KnollMemoryLACE: memory lifecycle\-aware consolidation and evidence retrieval\.arXiv preprint arXiv:2609\.03201\.External Links:[Link](https://arxiv.org/abs/2609.03201)Cited by:[Table 16](https://arxiv.org/html/2610.33366#A7.T16.2.11.1.1.1),[§6\.2](https://arxiv.org/html/2610.33366#S6.SS2.SSS0.Px2.p1.1)\.
- Yamadaet al\.\(2025\)Y\. Yamada, R\. T\. Lange, C\. Lu, S\. Hu, C\. Lu, J\. Foerster, J\. Clune, and D\. HaThe AI scientist\-v2: workshop\-level automated scientific discovery via agentic tree search\.arXiv preprint arXiv:2504\.08066\.External Links:[Link](https://arxiv.org/abs/2504.08066)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p1.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.5.1.1),[§4\.2](https://arxiv.org/html/2610.33366#S4.SS2.p1.1)\.
- Young \(1974\)H\. P\. YoungAn axiomatization of borda’s rule\.Journal of Economic Theory9\(1\),pp\. 43–52\.External Links:[Document](https://dx.doi.org/10.1016/0022-0531%2874%2990073-8)Cited by:[§3\.2](https://arxiv.org/html/2610.33366#S3.SS2.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025a\)H\. Yu, Z\. Hong, Z\. Cheng, K\. Zhu, K\. Xuan, J\. Yao, T\. Feng, and J\. YouResearchTown: simulator of human research community\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 73051–73096\.External Links:[Link](https://proceedings.mlr.press/v267/yu25i.html)Cited by:[§1](https://arxiv.org/html/2610.33366#S1.p2.1),[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2026\)H\. Yu, H\. Koh, M\. Kim, Y\. Jang, and K\. JungPRAGMA: evaluating personalized guidance with memory alignment in lifelong conversations\.arXiv preprint arXiv:2609\.09664\.External Links:[Link](https://arxiv.org/abs/2609.09664)Cited by:[Table 15](https://arxiv.org/html/2610.33366#A7.T15.2.10.1.1.1)\.
- Yuet al\.\(2025b\)W\. Yu, S\. Tang, Y\. Huang, N\. Dong, L\. Fan, H\. Qi, W\. Liu, X\. Diao, X\. Chen, and W\. OuyangDynamic knowledge exchange and dual\-diversity review: concisely unleashing the potential of a multi\-agent research team\.arXiv preprint arXiv:2506\.18348\.External Links:[Link](https://arxiv.org/abs/2506.18348)Cited by:[§2](https://arxiv.org/html/2610.33366#S2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.33366#S2.T1.2.9.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, S\. Li, F\. Liu, and J\. LiToward robust personalized alignment for LLMs: mitigating persona drift in multi\-turn dialogue\.arXiv preprint arXiv:2609\.12373\.External Links:[Link](https://arxiv.org/abs/2609.12373)Cited by:[Table 17](https://arxiv.org/html/2610.33366#A7.T17.2.12.1.1.1)\.
- Zhouet al\.\(2026\)W\. Zhou, J\. Lu, Y\. Wang, Z\. Xing, Z\. Chen, A\. Han, and H\. ShiRemember by asking: retrieval\-induced memory evolution for LLM agents\.arXiv preprint arXiv:2609\.34438\.External Links:[Link](https://arxiv.org/abs/2609.34438)Cited by:[Table 16](https://arxiv.org/html/2610.33366#A7.T16.2.11.1.1.1)\.

## Appendix

## Appendix ATopic Selection

We construct the topic pool from the ACL Anthology’s 2025 main proceedings for ACL, EMNLP, NAACL, and COLING, together with the ACL, EMNLP, and NAACL Findings volumes\. We exclude demonstrations, tutorials, student research workshops, and industry tracks, and require an abstract of at least 200 characters\. This yields 8,207 eligible papers, from which we uniformly sample 600 without replacement\. Research\-area labels are used to inspect coverage after sampling, without imposing area quotas\.

GPT\-5\.6 Luna derives one topic from each sampled paper’s title and abstract\. The model is instructed to express the underlying research problem in at most 12 words, without naming specific models, datasets, or methods\. Papers without a suitable research problem may be rejected, including surveys and resource releases that pose no research question\. The finalized run rejects 10 surveys and 4 resource papers, leaving 586 candidate topics\.

We embed the topics with Qwen3\-Embedding\-0\.6B and process them in the original sampling order\. A topic is removed if its cosine similarity to any previously retained topic exceeds 0\.85\. This removes 30 duplicates, leaving 556 unique topics\.

We randomly sample 100 topics from the deduplicated pool for evaluation and use the same topics for all methods and both knowledge settings\. In the same call that derives a topic, GPT\-5\.6 Luna also assigns the source paper one of the 22 ACL Rolling Review research areas\. Table[4](https://arxiv.org/html/2610.33366#A1.T4)reports the area distribution\. The 100 topics span 20 of these areas, and no area contributes more than 12 topics\.

Table 4:Research areas of the 100 evaluation topics\. Each source paper is assigned one ACL Rolling Review area by GPT\-5\.6 Luna when its topic is derived\.
## Appendix BOffline Knowledge Setting

The offline setting replaces the Semantic Scholar API with release 1\.0 of the Scientific Contribution Graph\([Jansen, 2026](https://arxiv.org/html/2610.33366#bib.bib5)\), a local corpus of 230,454 papers whose coverage ends in 2025\. The graph retrieval stage uses different seed selection and neighbor expansion procedures, while insight distillation, expertise formation, and debate remain unchanged\.

##### From contributions to a paper graph

The Scientific Contribution Graph represents scientific contributions as nodes and their prerequisite relationships as directed edges\. These relationships are identified from paper texts and citations through LLM\-based extraction and alignment\. We map contribution nodes to their source papers to construct a paper\-level graph\. Two papers are connected when a contribution in one paper depends on a contribution in the other, with edges directed from the prerequisite paper to the dependent paper\.

##### Seed retrieval and expansion

Without a search API, we find seed candidates by embedding similarity between the topic and the corpus contributions, using Qwen3\-Embedding\-0\.6B\. The top five candidates, with recent papers ranked first, go to the same seed\-selection prompt as in the online setting\. During expansion, we retrieve both incoming and outgoing neighbors from the mapped prerequisite graph, replacing the recommendation set𝒞⁡\(v\)\\mathcal\{C\}\(v\)used in the online setting\. We select the top 20 candidates based on contribution relevance to the topic and apply the same LLM filtering procedure as in the online setting, with additional instructions to favor recent papers and filter out unsuitable dataset or toolkit references\. If the resulting graph is too small or lacks papers from 2024 or later, we retry with the next seed candidate\.

##### Comparing the two settings

Offline graphs contain 38\.0 papers on average, compared with 43\.4 online, and the two settings select the same seed paper for 9 of the 100 topics\. Because the settings also differ in corpus, edge relation, and paper evidence, we do not attribute differences between their results to the knowledge source alone\. We instead treat them as two configurations ofInsightAgents, each compared with the same baseline proposals, which were generated once per topic\. ResearchAgent retains the online seed specified in Section[4\.2](https://arxiv.org/html/2610.33366#S4.SS2), so it usually starts from a different paper thanInsightAgentsin the offline comparison\.

## Appendix CImplementation Details

##### Graph construction

Given an input topic, we use GPT\-5\.6 Luna to select a seed paper from recent Semantic Scholar search results, prioritizing topical relevance and representativeness of the research area\. Starting from the seed, we retrieve up to 20 related candidates per expanded paper and retain up to 5, 3, and 2 new papers per parent across three expansion levels\. This gives an upper bound of 51 papers, including the seed\. The graph is passed to a single call that extracts three insights, each supported by at least two papers\.

##### Generation budgets

The configured output allowances are 8000 tokens for insight extraction, 2500 per expert speech, 4000 per critic review, and 6000 for the final proposal rewrite\. For reasoning models, the client adds a 2000\-token reserve to each output allowance to accommodate reasoning tokens\.

Table 5:Generation cost ofInsightAgentsper topic by stage, averaged over the 100 online topics\. Cost is in US cents\. Time sums the recorded stage durations and includes Semantic Scholar API latency\. Share is the stage’s percentage of total cost; Proposal dev\. denotes proposal development\.
##### Cost and efficiency

Table[5](https://arxiv.org/html/2610.33366#A3.T5)breaks down the generation cost ofInsightAgentsin the online setting\. We estimate cost from logged token counts at GPT\-5\.6 Luna prices of $0\.20 per million input tokens, $0\.02 per million cached input tokens, and $1\.20 per million output tokens\. Generating one proposal takes 42\.5 model calls and $0\.104 on average\. Summed over stages, one topic takes 7\.1 minutes on average\.

Table 6:Generation cost per topic in the online setting, averaged over 100 topics and taken from each system’s own usage log\. All methods use GPT\-5\.6 Luna at the same token prices\. Tokens are in thousands; input includes cached input where logged\. AI Scientist\-v2 runs only its idea generation stage; Chain of Ideas includes embedding calls\.Table[6](https://arxiv.org/html/2610.33366#A3.T6)compares generation cost across methods using each system’s own usage log\.InsightAgentscosts about as much as ResearchAgent, 1\.6 times as much as Chain of Ideas, and more than 30 times as much as the idea generation stage of AI Scientist\-v2\. Figure[2](https://arxiv.org/html/2610.33366#S5.F2)shows that cost alone does not explain the ranking: AI Scientist\-v2 has the second\-highest average Elo despite its lower cost, while ResearchAgent ranks third despite costing slightly more thanInsightAgents\.

## Appendix DBaseline Configurations

Table 7:Baseline configurations using official implementations and GPT\-5\.6 Luna\. Adaptations are detailed in Appendix[D](https://arxiv.org/html/2610.33366#A4)\. Executed counts are averaged over 100 online evaluation topics, with ranges in parentheses where available\.We run each baseline from its official code using GPT\-5\.6 Luna, retaining its stages, iteration counts, retrieval, and selection rules\. Table[7](https://arxiv.org/html/2610.33366#A4.T7)summarizes the configuration of each baseline and how its runs executed\. We adapt API clients, output parsing, and usage logging, with the generation\-setting and input changes described below\. Three settings differ from the original releases\. First, GPT\-5\.6 Luna does not accept a custom sampling temperature, so all baselines use the API default\. Second, AI Scientist\-v2 and Chain of Ideas keep their default output limits of 4,096 and 4,000 tokens per call \(16,000 for Chain of Ideas’ auxiliary model\), whereas we raise ResearchAgent’s default of 1,000 tokens to 4,096\. On a reasoning model these limits cover both reasoning and answer tokens\. Third, ResearchAgent receives the topic text in its problem prompts in addition to the seed paper, so that its problems address the same topic as the other methods\.

Although the final\-rewrite allowance ofInsightAgentslisted in Appendix[C](https://arxiv.org/html/2610.33366#A3)is higher, observed usage is at most 2,820 output tokens \(about 2,170 on average\) in both knowledge settings, below the baselines’ 4,000–4,096\-token limits for primary generation calls\.

## Appendix EAdditional Evaluation Results

This section supplements the main results in four parts\. Section[E\.1](https://arxiv.org/html/2610.33366#A5.SS1)reports detailed scores and comparesInsightAgentsdirectly with each baseline\. Section[E\.2](https://arxiv.org/html/2610.33366#A5.SS2)examines whyInsightAgentsfalls behind AI Scientist\-v2 on feasibility\. Section[E\.3](https://arxiv.org/html/2610.33366#A5.SS3)analyzes the LLM judges’ reasoning across all dimensions and baselines\. Section[E\.4](https://arxiv.org/html/2610.33366#A5.SS4)tests whether the LLM evaluation behind these results is reliable\.

### E\.1Detailed results and direct comparison

Table[8](https://arxiv.org/html/2610.33366#A5.T8)reports the Elo ratings shown in Figure[2](https://arxiv.org/html/2610.33366#S5.F2)for both knowledge settings, five dimensions, and two judges\. Elo ratings summarize each method’s overall performance across all pairwise comparisons\. To show howInsightAgentsperforms against each baseline individually, Table[9](https://arxiv.org/html/2610.33366#A5.T9)reports direct comparison scores, counting wins as 1 and ties as 0\.5\.

Table 8:Pairwise Elo ratings \(↑\\uparrow\)\. Setup lists the knowledge setting, topic countnn, and judge\. Eff\. denotes expected effectiveness; Avg\. is the mean of the five dimensions\. Shading highlightsInsightAgents; bold numbers mark the highest rating within each setup\.Table 9:Direct comparison scores \(%,↑\\uparrow\) ofInsightAgentsagainst each baseline, with half credit for ties\. Each dimension uses 200 online or 200 offline judgments\. A score above 50 meansInsightAgentsis preferred more often; Eff\. denotes expected effectiveness and Avg\. equally weights all five dimensions\. Brackets give 95% percentile bootstrap intervals for Avg\., resampling topics 10,000 times while retaining both orders and all dimensions together\.OpponentNoveltySignificanceFeasibilityClarityEff\.Avg\. \[95% CI\]Online, GPT\-5\.6 SolAI Scientist\-v289\.074\.026\.077\.088\.070\.8 \[67\.6, 73\.8\]Chain of Ideas97\.589\.080\.097\.5100\.092\.8 \[90\.7, 94\.8\]ResearchAgent98\.575\.590\.094\.089\.589\.5 \[87\.0, 91\.9\]Online, Claude Opus 4\.8AI Scientist\-v293\.080\.816\.034\.083\.861\.5 \[58\.0, 64\.9\]Chain of Ideas99\.595\.068\.586\.2100\.089\.8 \[87\.8, 91\.8\]ResearchAgent100\.084\.273\.284\.592\.286\.8 \[84\.0, 89\.6\]Offline, GPT\-5\.6 SolAI Scientist\-v286\.582\.036\.574\.589\.573\.8 \[70\.5, 77\.1\]Chain of Ideas96\.086\.581\.098\.599\.592\.3 \[90\.3, 94\.2\]ResearchAgent96\.072\.590\.598\.592\.590\.0 \[87\.8, 92\.2\]Offline, Claude Opus 4\.8AI Scientist\-v287\.878\.823\.038\.588\.563\.3 \[60\.0, 66\.5\]Chain of Ideas98\.593\.272\.090\.899\.290\.8 \[88\.7, 92\.8\]ResearchAgent97\.583\.279\.886\.594\.888\.3 \[85\.9, 90\.7\]The direct comparisons show similar patterns across both knowledge settings\. Against Chain of Ideas and ResearchAgent,InsightAgentsscores above 68% on every dimension under both judges, with average scores ranging from 86\.8% to 92\.8%\. Against AI Scientist\-v2,InsightAgentsachieves average scores of 70\.8% online and 73\.8% offline under GPT\-5\.6 Sol, and 61\.5% and 63\.3% under Claude Opus 4\.8, with all four confidence intervals above 50%\. Scores for novelty, significance, and expected effectiveness remain at or above 74%, while feasibility is substantially lower \(16\.0–36\.5%\), as further examined in Section[E\.2](https://arxiv.org/html/2610.33366#A5.SS2)\.

Clarity is the only dimension where the two judges favor different methods in both settings\. GPT\-5\.6 Sol favorsInsightAgentsover AI Scientist\-v2 \(77\.0% online; 74\.5% offline\), whereas Claude Opus 4\.8 favors AI Scientist\-v2 \(34\.0% online; 38\.5% offline forInsightAgents\)\. Section[E\.4\.3](https://arxiv.org/html/2610.33366#A5.SS4.SSS3)further examines this disagreement\.

### E\.2Feasibility trade\-off

Table[8](https://arxiv.org/html/2610.33366#A5.T8)shows that AI Scientist\-v2 outperformsInsightAgentson feasibility across both knowledge settings and judges\. To better understand this gap, we analyze their 200 online pairwise comparisons per judge, whereInsightAgentsloses 148 under GPT\-5\.6 Sol and 161 under Claude Opus 4\.8\.

Table 10:What feasibility rationales mention when AI Scientist\-v2 wins \(% of rationales\)\.nncounts rationales that refer to both proposals by their anonymous labels\. Categories are keyword matches on the text describing each proposal, so a mention is not a criticism; a rationale can match several categories\.##### Judge rationale analysis

We examine the rationales favoring AI Scientist\-v2, retaining 127 from GPT\-5\.6 Sol and 143 from Claude Opus 4\.8 that explicitly reference both proposals by their anonymous labels\. Table[10](https://arxiv.org/html/2610.33366#A5.T10)summarizes feasibility\-related factors identified through keyword matching\. ForInsightAgents, the rationales frequently mention human annotation and multiple components, while Claude Opus 4\.8 also highlights validation requirements such as preregistration and audits\. Compute\-related concerns appear in fewer than 10% of the rationales\. For AI Scientist\-v2, more than 80% mention simple or standard resources\. Only one of the 270 rationales explicitly describesInsightAgentsas infeasible, impossible, or impractical\.

##### Discussion

These patterns suggest that the feasibility gap may be associated more with the perceived scope and validation requirements of the proposed studies than with computational demands\. However, keyword matches indicate which factors appear in the rationales, rather than which factors determine the judges’ preferences\.

### E\.3LLM Judge Reasoning Analysis

For every pairwise judgment, each judge writes a one\- or two\-sentence explanation per evaluation dimension\. We analyze all 6,000 online explanations that compareInsightAgentswith a baseline: 300 topic–baseline pairs, two presentation orders, five dimensions, and two judges\. We derived the properties in Table[11](https://arxiv.org/html/2610.33366#A5.T11)through LLM\-assisted open coding of a stratified sample of 170 explanations, oversampling comparisons whereInsightAgentsloses, the judges disagree, or the verdict changes with presentation order\. The authors reviewed and revised the resulting categories, and the definitions were refined until two independent LLM coders applied them consistently \(Cohen’sκ≥0\.90\\kappa\\geq 0\.90per property\)\. The labeler also codes a readability property, which we do not report because it overlaps with the clarity dimension\.

Table 11:Proposal properties used to label judge explanations\. For each explanation and property, the labeler records which proposal the judge favors, or that the property is not discussed\. A property can be cited under any evaluation dimension\.Claude Opus 5\.5 labels each explanation through the Batch API with a JSON schema that fixes the output to one label per property\. The labeler sees the evaluated dimension, the winner, and the explanation exactly as the judge wrote it, with the proposals named Idea0 and Idea1 by presentation order\. When an explanation cites an execution barrier, the labeler also records its type, using the four types in Table[11](https://arxiv.org/html/2610.33366#A5.T11)\.

To test whether the labeler favors our system when it knows which one it is, we repeated the labeling with a prompt that identified our proposal\. The two runs agree on 97\.7% of property labels, and no share in Figure[7](https://arxiv.org/html/2610.33366#A5.F7)changes by more than 3\.0 points\.

Figure 6:LLM judges’ preferences by proposal property, based on their stated reasoning\. Bars show the percentage of explanations favoringInsightAgents\(right\) or the baseline \(left\), among those discussing each property\. Numbers beside the bars give the total count of explanations discussing that property\. Results pool both judges in the online setting\.Figure 7:Execution barriers identified in the LLM judges’ stated reasoning\. Bars show the number of explanations citing each barrier againstInsightAgentsor the baseline\. Each explanation may cite multiple barrier types\. Results pool both judges in the online setting\.Figure[7](https://arxiv.org/html/2610.33366#A5.F7)breaks down the execution barriers\. Against AI Scientist\-v2, barriers are cited againstInsightAgentsfar more often than against the baseline, with the relative largest gaps for human annotation \(187 vs\. 29\) and custom apparatus \(131 vs\. 13\)\. Among the 309 feasibility judgments thatInsightAgentsloses to AI Scientist\-v2, every explanation cites at least one barrier againstInsightAgents: scale, data access, or integration in 76\.1%, human annotation in 52\.8%, custom apparatus in 39\.8%, and an unsolved prerequisite in 34\.6%\. These shares are consistent with the keyword analysis in Table[10](https://arxiv.org/html/2610.33366#A5.T10)\. Against Chain of Ideas and ResearchAgent the direction reverses, and every barrier type is cited more often against the baseline\.

### E\.4Reliability of the LLM evaluation

We conduct four analyses to assess the consistency and potential biases of our LLM\-based evaluation\. Section[E\.4\.1](https://arxiv.org/html/2610.33366#A5.SS4.SSS1)examines whether judgments remain consistent when presentation order is reversed and how often the two judges agree\. Section[E\.4\.2](https://arxiv.org/html/2610.33366#A5.SS4.SSS2)compares LLM judgments with human annotations\. Section[E\.4\.3](https://arxiv.org/html/2610.33366#A5.SS4.SSS3)examines how agreement varies across evaluation dimensions\. Section[E\.4\.4](https://arxiv.org/html/2610.33366#A5.SS4.SSS4)investigates whether proposal length is associated with judge preferences\.

#### E\.4\.1Consistency within and across LLM judges

Table 12:Evaluation agreement \(%\) on 100 topics per setting\. Order agreement measures whether swapping presentation order preserves the winner or tie \(n=600n=600pairs per dimension and setting\)\. Judge agreement compares the two judges on identical pairs and orders \(n=1,200n=1\{,\}200\)\.Table[12](https://arxiv.org/html/2610.33366#A5.T12)reports consistency within each LLM judge under reversed presentation orders and agreement between the two judges\. For within\-judge consistency, decisions remain unchanged in at least 89\.0% of comparisons for GPT\-5\.6 Sol and 80\.3% for Claude Opus 4\.8 on novelty, feasibility, and expected effectiveness across both settings\. Claude Opus 4\.8 shows lower consistency on significance and clarity \(72\.3–76\.0%\)\. For cross\-judge agreement, the two judges agree on at least 83\.1% of comparisons on novelty, feasibility, and expected effectiveness, but only 68\.9–72\.5% on significance and clarity\. Overall, judgments are more consistent on novelty, feasibility, and expected effectiveness, with greater variation on significance and clarity\.

#### E\.4\.2Agreement with human annotators

We compare human and LLM preferences on the 150InsightAgents–baseline instances described in Appendix[F](https://arxiv.org/html/2610.33366#A6)\. For each LLM judge, we combine judgments from both presentation orders, treating equal win counts as ties\. We compute exact agreement and non\-contradiction as defined in Section[5](https://arxiv.org/html/2610.33366#S5)\. We report 95% confidence intervals \(in brackets\) from 2,000 topic\-level bootstrap resamples, preserving all comparisons within each sampled topic\. Figure[8](https://arxiv.org/html/2610.33366#A5.F8)presents these estimates for each dimension and overall\.

Figure[4](https://arxiv.org/html/2610.33366#S5.F4)shows that human and LLM judges rarely express opposing preferences\. Human–LLM non\-contradiction reaches 90\.0% \[87\.0, 92\.3\] for GPT\-5\.6 Sol and 92\.7% \[89\.7, 95\.3\] for Claude Opus 4\.8, compared with 90\.0% \[86\.0, 94\.0\] between human annotators and 93\.3% \[89\.3, 96\.7\] between LLM judges\. All four confidence intervals overlap, suggesting no clear difference in non\-contradiction rates\.

Exact agreement is lower: 63\.7% \[55\.7, 70\.0\] and 65\.0% \[55\.3, 74\.3\] for human–LLM comparisons, compared with 44\.0% \[30\.0, 58\.0\] between humans and 82\.0% \[78\.7, 84\.7\] between LLM judges\. The non\-overlapping human–human and LLM–LLM intervals indicate substantially higher exact agreement between the two LLM judges in this sample\. Together, these results suggest that many human–LLM disagreements involve ties rather than directly opposing preferences\.

#### E\.4\.3Agreement by dimension

Figure 8:Agreement by dimension on the sameInsightAgents–baseline comparisons as Figure[4](https://arxiv.org/html/2610.33366#S5.F4)\. Each dimension contains 30 topic–pair instances, and Overall pools all 150\. Error bars give 95% topic\-bootstrap intervals\. Sol and Opus denote GPT\-5\.6 Sol and Claude Opus 4\.8\.Figure 9:Outcomes ofInsightAgents–baseline comparisons for each judge and dimension\. Each bar gives the share of 30 topic–pair instances in which the judge prefersInsightAgents, returns a tie, or prefers the baseline\. Labels show shares of at least 15%\. Ours denotesInsightAgents\.Figure[8](https://arxiv.org/html/2610.33366#A5.F8)shows that human–LLM exact agreement is highest for novelty \(73\.3–78\.3%\) and lowest for clarity \(48\.3–50\.0%\)\. Non\-contradiction remains at least 83\.3% across all judge pairs and dimensions\. However, each dimension contains only 30 topic–pair instances, resulting in wide confidence intervals\.

Figure[9](https://arxiv.org/html/2610.33366#A5.F9)helps explain these differences\. Annotator 1 returns ties in 30–83% of comparisons across dimensions, while the other judges do so in at most 17%\. This contrast is particularly pronounced for clarity, where human–human exact agreement is only 20\.0%, despite 100\.0% non\-contradiction\. Thus, low exact agreement on clarity largely reflects differences in tie frequency, although the LLM judges also show different clarity preferences in the full evaluation \(Section[E\.1](https://arxiv.org/html/2610.33366#A5.SS1)\)\.

Feasibility shows a different pattern\. All four judges favor the baseline overInsightAgentsin 13–43% of feasibility comparisons, compared with at most 7% for novelty\. Feasibility also has the lowest non\-contradiction between human annotators \(83\.3%\) and between humans and GPT\-5\.6 Sol \(85\.0%\), indicating more directly opposing preferences\. This is consistent with the feasibility trade\-off observed in the main evaluation\.

#### E\.4\.4Proposal length

Table 13:Proposal length and judgments\. Top: share \(%\) of decided judgments won by the longer proposal, pooled over all six method pairs and both orders\. Bottom: Pearsonrrover 100 topics between the word\-count difference \(InsightAgentsminus AI Scientist\-v2\) andInsightAgents’s average score across the five dimensions\.The evaluation prompt instructs judges to assess proposal quality regardless of length \(Section[4\.3](https://arxiv.org/html/2610.33366#S4.SS3)\)\. We examine whether proposal length is associated with judge preferences\. Proposal lengths vary substantially across methods: AI Scientist\-v2 averages 931 words,InsightAgents1,297 online and 1,281 offline, Chain of Ideas 2,175, and ResearchAgent 3,274\. Table[13](https://arxiv.org/html/2610.33366#A5.T13)shows that longer proposals win only 18\.1–46\.5% of non\-tied comparisons across dimensions, settings, and judges, considering all six method pairs\.

We also examineInsightAgentsagainst AI Scientist\-v2, its closest competitor, which outperformsInsightAgentson some evaluation dimensions\. Across topics, the correlation between proposal length differences andInsightAgents’s average score is weak \(r=0\.06r=0\.06–0\.210\.21\), with dimension\-level correlations ranging from−0\.08\-0\.08to0\.270\.27\. Overall, these results provide no clear evidence of a systematic preference for longer proposals\.

## Appendix FHuman Annotation Protocol

Table 14:Instructions shown to human annotators\. The bracketed block stands for the criteria text, which is identical to the LLM judge prompt in Table[39](https://arxiv.org/html/2610.33366#A8.T39)and is not repeated here\. Instructions about saving are omitted\.1\. What you are asked to doEach of the 10 topics in this study shows 4 research ideas side by side, labelled A, B, C, D\. Which system wrote which idea is hidden from you, and the order is randomised per annotator\.Read them, then rank them from best to worst separately for each of the five criteria: one text box per criterion,\>= strictly better,== equally good, every label used exactly once\. For example:B \> A \> CB best, then A, then CB \> A = CB best, A and C equally goodA = B = Call indistinguishable on this criterionA topic counts as done only once all five criteria are ranked\.2\. What to judge on\[The five evaluation criteria and the general rules for all criteria, copied word for word from the LLM judge prompt \(Table[39](https://arxiv.org/html/2610.33366#A8.T39)\)\.\]The LLM judge is given the criteria above word\-for\-word, plus this instruction, which applies to you as well:\[The note on position and length bias, copied word for word from the same prompt\.\]We conduct a human evaluation to assess agreement with LLM judges\. Two NLP researchers independently evaluate proposals on 10 online topics selected based on their familiarity with the topics\. Each topic includes four proposals generated byInsightAgents, AI Scientist\-v2, Chain of Ideas, and ResearchAgent\. We use the same proposal texts as in the LLM evaluation, without modification\.

##### Annotation procedure

Annotators follow the same evaluation criteria and judging instructions as the LLM judges, including guidance on avoiding position and length bias \(Table[14](https://arxiv.org/html/2610.33366#A6.T14)\)\. A web\-based interface presents the four proposals side by side, with method identities hidden and display positions independently randomized for each topic and annotator\. For each of the five dimensions, annotators rank all four proposals from best to worst, allowing ties \(e\.g\.,B \> A = C \> D\)\. Unlike LLM judges, who compare proposals pairwise, human annotators rank all four proposals together\.

##### Agreement computation

We convert each ranking into six pairwise outcomes, treating equally ranked proposals as ties\. Human Elo ratings are computed using the same procedure as the LLM evaluation \(Figure[3](https://arxiv.org/html/2610.33366#S5.F3)\)\. For agreement analysis, we consider the threeInsightAgents–baseline pairs on topics completed by both annotators, yielding 150 topic–dimension–pair instances \(10 topics×\\times5 dimensions×\\times3 pairs\)\. These outcomes are compared with LLM judgments in Section[E\.4\.2](https://arxiv.org/html/2610.33366#A5.SS4.SSS2)\.

## Appendix GDetailed Generation Case Study

This appendix presents the online case analyzed in Section[6\.2](https://arxiv.org/html/2610.33366#S6.SS2), for the topicPersonalized dialogue generation through evolving user\-specific memory\. Tables headedGenerated outputreproduce saved model outputs verbatim, with shaded cells marking text written by the system; unshaded rows, step labels, and the prose between tables are written by the authors\. Paper labels such as \[P18\] are local to this case; each insight table ends with a key that identifies the cited papers\.

### G\.1Insights and supporting findings

Tables[15](https://arxiv.org/html/2610.33366#A7.T15)–[17](https://arxiv.org/html/2610.33366#A7.T17)show the three insights distilled from the paper graph, one per expert, with the supporting findings each expert received\.

Generated output:Expert A insight — Memory Utilization EvaluationCore insight\.This line keeps finding that retrieving a remembered fact is easier than recognizing when it matters and integrating it naturally into an answer\. Evaluation has shifted from direct question answering toward implicit, composed, guidance, and evidence\-synthesis tasks\. It assumes useful memory must be judged in context, not as isolated retrieval\. The field remains stuck at the boundary between recovered evidence and coherent, appropriately personalized reasoning\.Historical tension\.Broader, more realistic evaluation exposes genuine failures but sacrifices the clean attribution and comparability of pointwise recall benchmarks\.Grounding summary\.This reading rests on P18, P37, P44, and P5, representing the graph’s evaluation and benchmarking line\. The graph shows a clear progression from elicited recall toward natural integration, distributed evidence use, and decision\-support evaluation\.Supporting findings\.\[P18\]In a four\-month deployment with 40 users, 1,872 sessions, and seven memory conditions, Direct QA ranged from 19\.7% to 70\.1% without changing satisfaction; only 7\.9% of recalled facts were naturally referenced\.\[P37\]On LOCOMO\-CONV, conversational framing exposed retrieval gaps for implicit and composed queries; multi\-facet rewriting helped raw\-turn memory but not abstractive memory\.\[P44\]UtilMem found that strong factual\-memory scores did not reliably predict memory utilization: retrieval alone often failed to integrate distributed evidence or reject plausible distractors\.\[P5\]PRAGMA’s longitudinal guidance scenarios, with evidence annotations and evolving contexts, showed current retrieval, memory, and long\-context systems struggling to use evidence for personalized guidance\.Paper key\.\[P18\]MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long\-Term Human\-AI Conversation\([Sumida et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib30)\);\[P37\]When Users Don’t Ask: Benchmarking Context\-Driven Memory Retrieval in Conversational Agents\([Chang and Chen, 2026](https://arxiv.org/html/2610.33366#bib.bib39)\);\[P44\]UTILMEM: Benchmarking Evidence Utilization in Long\-Term Conversational Memory\([Qing et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib29)\);\[P5\]PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations\([Yu et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib31)\)Table 15:Insight distilled for Expert A \(Memory Utilization Evaluation\), reproduced verbatim\. Shaded rows are model output; the paper key below them is added by the authors\.Generated output:Expert B insight — Evolving Memory ArchitectureCore insight\.This line keeps running into the fact that static summaries and independent top\-k retrieval cannot represent changing truths, provenance, conflicts, and query\-specific evidence needs together\. Systems have progressed from profile\-memory graphs to temporal state graphs, multi\-round retrieval, selective memory types, and lifecycle\-aware relations\. They assume explicit structure can make memory auditable without retaining the full history\. The unresolved cost is deciding which structure to maintain and when updates or retrieval should reshape it\.Historical tension\.More explicit temporal and evidential structure improves correctness and traceability, but increases maintenance, retrieval complexity, and dependence on intermediate decisions\.Grounding summary\.This reading rests on P10, P9, P24, P11, and P30, the graph’s densest line of structured, evolving, and lifecycle\-aware memory systems\. It shows a progression from organizing memory to explicitly governing evidence state and retrieval\-driven maintenance\.Supporting findings\.\[P10\]PGMem linked persona nodes to event nodes through typed provenance and evidence edges, consistently outperforming summary\-based, persona\-aware, graph, and agentic baselines across three benchmarks\.\[P9\]EMIR2used a State\-Evolving Memory Graph and intent\-guided multi\-round retrieval, producing relative improvements exceeding 12% in some LoCoMo and MemConflict categories\.\[P24\]LeanMem routed information into profile, event, or source\-grounded record memory; on LoCoMo and LongMemEval\-S it improved over the strongest memory baseline by up to 15\.1 points\.\[P11\]RIME used self\-questions to retrieve evidence during memory formation and achieved the best results across all three reported quality metrics on LoCoMo with Qwen3\-235B\-A22B and GPT\-5\.6 Sol\.\[P30\]MemoryLACE modeled merge, supersession, and contradiction relations while preserving atomic memories; it reduced BEAM end\-to\-end runtime by 66\.6% versus Hindsight\.Paper key\.\[P10\]PGMem: Tightly Coupled Persona\-Memory Graph for Lifelong Personalized Agents\([Choi et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib37)\);\[P9\]EMIR2: Evolution\-Aware Memory with Intent\-Guided Multi\-Round Retrieval\([Liu et al\., 2026a](https://arxiv.org/html/2610.33366#bib.bib35)\);\[P24\]LeanMem: Simple and Efficient Long\-Term Memory for LLM Agents\([Liao et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib27)\);\[P11\]Remember by Asking: Retrieval\-Induced Memory Evolution for LLM Agents\([Zhou et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib26)\);\[P30\]MemoryLACE: Memory Lifecycle\-Aware Consolidation and Evidence Retrieval\([Yacoubi et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib33)\)Table 16:Insight distilled for Expert B \(Evolving Memory Architecture\), reproduced verbatim\. Shaded rows are model output; the paper key below them is added by the authors\.Generated output:Expert C insight — Continual Personalization AlignmentCore insight\.This line keeps finding that personalization signals are latent, noisy, time\-varying, and easy to over\-apply\. It has moved from static profiles toward uncertainty\-aware belief revision, preference hypotheses, adaptive stopping, and online tracing of short\- and long\-term intent\. It assumes alignment should reflect inferred user constraints rather than merely repeat stored preferences\. It remains stuck between adapting quickly enough to matter and resisting transient evidence, preference narrowing, sycophancy, and irrelevant personalization\.Historical tension\.Personalization must be responsive to emerging preferences without turning every weak signal into a persistent commitment or an excuse to flatter the user\.Grounding summary\.This reading rests on P19, P16, P17, P42, P47, and P46, spanning personalization\-risk evaluation, behavioral alignment, belief revision, hypothesis tracking, and context\-budget control\. The graph shows adaptation becoming more explicit and selective, but not yet reliably safe\.Supporting findings\.\[P19\]Across 13 LLMs, PRISK found personalization increased irrelevant personalization, preference narrowing, and sycophantic bias, with reported average drops of 45\.9%, 41\.7%, and 61\.7%, respectively\.\[P16\]IBA\-Bench showed a knowledge\-to\-action gap: agents must execute tasks under implicit, noisy, temporally inconsistent constraints, and IBA\-Agent improved behavioral alignment across nine application domains\.\[P17\]CORE separated turn\-local evidence from persistent persona revision through uncertainty\-aware belief updates, improving alignment and robustness on ALOE, PersonaChat, and PERSIST\.\[P42\]HypReflect represented uncertain preference hypotheses, refined them as evidence accumulated, and used hypothesis\-guided self\-distillation across online, multi\-session, and implicit\-behavior settings\.\[P47\]HyperTrace maintained interpretable short\-term\-intent and long\-term\-preference hypotheses with an SMC\-style reweighting process, improving alignment, preference prediction, and profile consistency on PRISM and PersonaMem\-v2\.\[P46\]ENOUGH learned to append records or stop, optimizing downstream utility, user specificity, and token cost to construct minimal sufficient profiles across six personalized tasks\.Paper key\.\[P19\]Evaluating the Hidden Costs of Personalization in Large Language Models\([Wang et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib25)\);\[P16\]From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents\([Song et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib38)\);\[P17\]Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi\-Turn Dialogue\([Zhang et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib34)\);\[P42\]Hypotheses\-Guided Self Distillation for Continual Personalization\([Hwang et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib32)\);\[P47\]HyperTrace: Hypothesis\-Based Preference Tracing for Online LLM Personalization\([Shen et al\., 2026](https://arxiv.org/html/2610.33366#bib.bib28)\);\[P46\]Less Is Personal: Learning Minimal Sufficient User Profiles for Personalized Language Models\([Liu et al\., 2026b](https://arxiv.org/html/2610.33366#bib.bib36)\)Table 17:Insight distilled for Expert C \(Continual Personalization Alignment\), reproduced verbatim\. Shaded rows are model output; the paper key below them is added by the authors\.
### G\.2Final research proposal

Table[18](https://arxiv.org/html/2610.33366#A7.T18)shows the complete reader\-facing proposal\. Its claims and experimental requirements are retained as generated\.

Generated output:Final research proposalTitle\.Lifecycle\-Aware Counterfactual Control of Memory InfluenceProblem and motivation\.Can a personalized dialogue system determine which evolving user\-memory evidence is currently justified, whether it applies to the present request, and how strongly it should change the response? A memory may be current, historical, context\-specific, superseded, unresolved, or irrelevant\. The appropriate behavior may therefore be silent personalization, explicit grounding, confirmation, abstention, or no behavioral change\. Existing evaluations often reduce personalization to retrieval accuracy, answer correctness, or explicit memory mention\.This misses useful silent grounding and cannot distinguish missing evidence from a system that retrieved the right evidence but gave it unjustified authority\. The proposal studies whether lifecycle\-aware evidence states and auditable influence control can produce more useful and less harmful personalization in recommendation, planning, and decision\-support tasks\.Method\.The system constructs a lifecycle\-aware evidence unit from retrieved memories\. It represents whether claims are current, historical, context\-specific, superseded, unresolved, or irrelevant, together with provenance and supporting or conflicting relations\. A lightweight controller receives the current request, this evidence unit, and a no\-memory shadow response\. It selects an action—silent use, explicit grounding, confirmation, or abstention—and one of four discrete influence levels\. Level 0 prevents memory from changing ranking, membership, or plan constraints\. Level 0\.25 permits reranking only among options that already satisfy the explicit task constraints\.Level 0\.5 permits a soft preference or constraint while retaining at least one viable alternative\. Level 1 permits filtering options or materially changing a plan only when the evidence is a confirmed current constraint, such as an explicit recent statement or concordant evidence meeting a preregistered rule\. If level 1 is predicted without confirmation, the system downgrades to level 0\.5 or asks for confirmation\. The controller applies the intervention to a common candidate pool or structured plan representation; the generator then receives only that representation, not the retrieved evidence or controller action, and verbalizes it with deterministic decoding\.Confirmation produces only a clarification question, explicit grounding must accurately mention the relevant evidence, and abstention uses the no\-memory response\. Influence is measured primarily through changes in option ranking, membership, retained alternatives, and plan constraints, and secondarily through textual grounding, usefulness, and naturalness\. This makes memory influence an auditable intervention rather than a latent score or a proxy based on memory mention\.Experiment\.The primary study will evaluate structured recommendation and planning cases drawn from eligible examples in MemUse, When Users Don’t Ask: Benchmarking Context\-Driven Memory Retrieval in Conversational Agents, UTILMEM: Benchmarking Evidence Utilization in Long\-Term Conversational Memory, and PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations\. A preliminary audit will determine whether these resources contain sufficient suitable instances; if not, the natural\-data component will be reported as exploratory rather than presented as representative\. The benchmark will have two separated components\.The controlled component will contain approximately 50 cases for each of five conditions: supersession, contextual coexistence, contradiction, stale distractors, and irrelevance\. These labels are known by construction, but cases with the same lifecycle state will vary in request risk and applicability so that state classification alone cannot solve the task\. The natural component will use unmodified or minimally transformed histories where possible: 300 cases for development and calibration and 100 held\-out cases for final evaluation\. Three annotators will label each case’s lifecycle state, acceptable action set, maximum permitted influence, and whether the correct behavior is no change\.They will not label system outputs\. Three blinded raters will independently judge held\-out outputs against the same case and its no\-memory reference for useful improvement, harmful degradation, grounding accuracy, and naturalness\. Agreement will be reported before adjudication, and mixed\-effects or paired\-bootstrap intervals will account for case and rater clustering\. All systems will use the same request, history, retrieved evidence, candidate pool or initial plan, prompt template, model, decoding settings, and random seed\. The no\-memory and memory branches will differ only in the prescribed intervention, and verbalization will receive only the resulting structured representation\.Baselines will include no memory, aggressive top\-k retrieval, lifecycle retrieval without influence control, influence control without lifecycle relations, a confidence\-only controller, and random action or budget policies calibrated on development data to match abstention and mean influence\. The primary estimand is the held\-out paired difference in useful influence and harmful influence between the complete method and the strongest matched control\. The preregistered success criterion is at least a 0\.10\-point improvement in useful influence without more than a 0\.05\-point increase in harmful influence on the rating scale\.Additional measures are lifecycle\-state accuracy, action\-set accuracy, influence\-level accuracy, no\-change accuracy, missed and unnecessary confirmation, and explicit\-grounding correctness\. A result supporting the claim must show higher useful and lower harmful behavioral change at matched abstention and average influence, with lifecycle ablations failing especially on supersession, contextual coexistence, and contradiction cases\. A null result will be interpreted narrowly as non\-improvement for this controller, benchmark, and task domain\.If the primary result holds, a follow\-up study can test longitudinal authority decay and recommendation\-induced self\-confirmation; these are not claims of the present paper\.Novelty\.The proposal makes memory’s permitted behavioral influence—not retrieval, explicit mention, or binary utilization—the central object of study\. It combines lifecycle\-aware evidence reconstruction with a discrete, auditable policy over how much that evidence may change a recommendation or plan\. The no\-memory shadow response exposes silent adaptation, while fixed intervention levels make behavioral changes attributable and testable\. The design also treats justified no\-change behavior as a first\-class outcome and controls for improvements caused merely by retrieving more evidence, abstaining more often, or changing prompt format\.Builds on\.MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long\-Term Human\-AI Conversation \(2026\) reports that a system can perform well on direct fact questions while naturally referencing very few corresponding facts, and that natural integration better tracks satisfaction; this motivates evaluating behavioral change and silent grounding rather than explicit memory mention\. MemoryLACE: Memory Lifecycle\-Aware Consolidation and Evidence Retrieval \(2026\) motivates organizing evidence through lifecycle relations such as merge, supersession, and contradiction before using it; this proposal uses that lifecycle state as the input to its influence controller\.PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations \(2026\) motivates testing memory use in evolving personalized guidance, where incorrect assumptions can alter recommendations and plans rather than merely affect factual recall\.Distinct from\.Unlike MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long\-Term Human\-AI Conversation \(2026\), which establishes the importance of natural integration but does not provide matched interventions for attributing whether memory changed behavior appropriately, this proposal compares paired no\-memory, justified\-memory, stale\-memory, and conflict conditions and evaluates the resulting structured and textual changes\.Unlike MemoryLACE: Memory Lifecycle\-Aware Consolidation and Evidence Retrieval \(2026\), which organizes and retrieves memories using lifecycle relations, this proposal studies the subsequent governance decision: whether that evidence should silently change behavior, be stated, trigger confirmation, or be withheld, and how much influence it may exert\.Potential impact\.If successful, the work will give personalized\-dialogue research a measurable distinction between possessing a justified memory and granting it justified behavioral authority\. It will help separate missing evidence, incorrect lifecycle interpretation, excessive influence, ignored evidence, and generation failures, while supporting useful personalization without treating every retrieved fact as an instruction\. Its fixed interventions and matched controls provide a practical evaluation and control layer for public\-model systems under university\-lab compute\.If it does not improve human\-rated usefulness and harm at matched abstention and influence levels, the result will show that this lifecycle\-aware controller does not improve personalization under the tested conditions, or that generation and user modeling remain the dominant bottlenecks; it will not by itself establish that memory authority is generally invalid\.Table 18:Final proposal produced by Stage 3, reproduced verbatim \(all reader\-facing fields; the internal expert\-attribution field is omitted\)\. Long fields are split into rows at sentence boundaries for page layout only\.
### G\.3Debate excerpts and proposal revisions

The tables below quote exact spans from the debate record; the text before each table summarizes the process\. Turn numbers refer to proposal development \(Stage 2\)\. Two intermediate drafts were saved, after turns 1 and 2; the convergence speeches in turn 3 were incorporated during final synthesis \(Stage 3\)\.

##### Expertise informing the initial design

Table[19](https://arxiv.org/html/2610.33366#A7.T19)illustrates how the experts contributed to the initial proposal\. Expert A emphasized comparing responses with and without memory to measure its influence, while Expert C proposed limiting the influence of uncertain preferences\. Expert B challenged these proposals, arguing that memory influence must also account for its lifecycle state\. Although Experts A and C both discussed counterfactual comparisons, their proposals emphasized different aspects of the problem\.

Generated output:Expert proposals in the first debate turnExpert A, turn 1“A system receives credit when the memory\-induced change is useful and warranted—even if the response never mentions the memory\.”Expert C, turn 1“Rather than passing retrieved memories directly to a generator, IGP produces aninfluence budgetfor each hypothesis\.”Expert B, turn 1“From the evolving\-memory perspective, however, influence cannot be calibrated reliably if the input to the controller is merely a bag of retrieved memories with confidence scores\. Confidence is not a substitute for lifecycle state\.”Table 19:Excerpts from the three expert proposals in Stage 2, turn 1 \(speaking order C, A, B\), reproduced verbatim\.
##### Making the influence budget operational

Table[20](https://arxiv.org/html/2610.33366#A7.T20)shows how the influence budget evolved through debate\. The initial draft combined evidence\-state reconstruction, influence control, and response comparison, but defined the budget only through qualitative examples\. The critic called for explicit, testable interventions\. In response, Expert A specified four discrete influence levels, Expert B reinforced their constraints, and Expert C proposed applying the interventions to a shared structured representation before response generation\. The second draft incorporated these changes\. Following another critic review, the experts further specified matched candidate pools and deterministic decoding, restricting the generator to the post\-intervention representation\. These refinements were retained in the final proposal, making the influence control mechanism more explicit and testable\.

Generated output:Revision chain 1 — making the influence budget operationalDraft, turn 1“The budget determines how strongly a memory may alter the response: a confirmed current constraint may substantially change a plan, while a weak inferred preference may only softly rerank options\.”Critic, after turn 1“Otherwise the claimed budget may merely be a latent label and the observed behavior could be caused by generation noise or prompt wording\.”Expert C, turn 2“The system should first generate a common candidate pool or structured plan representation\. The influence gate then modifies that representation according to the budget, and the same frozen generator verbalizes the result\.”Draft, turn 2“Level 0 excludes memory from candidate selection and planning; level 0\.25 permits reranking compatible options without removing alternatives; level 0\.5 permits a soft constraint while retaining alternatives; and level 1 permits filtering or materially changing a plan only when the evidence is a confirmed current constraint\.”Final proposal“The controller applies the intervention to a common candidate pool or structured plan representation; the generator then receives only that representation, not the retrieved evidence or controller action, and verbalizes it with deterministic decoding\.”Table 20:Excerpts tracing the influence budget from the first draft to the final proposal, reproduced verbatim\.
##### Aligning the claim with the evaluation scope

Table[21](https://arxiv.org/html/2610.33366#A7.T21)shows how the experts refined the proposal’s scope through debate\. The initial draft included a mechanism to reduce the influence of uncertain memories through repeated use, aiming to prevent self\-reinforcing personalization\. The critic questioned whether the proposed case\-based benchmark could adequately evaluate this longitudinal mechanism\. Experts A and B suggested deferring it, and Expert C, who originally proposed it, agreed\. The revised draft removed the mechanism from the primary study, and the final proposal retained it only as future work\.

Generated output:Revision chain 2 — aligning the claim with the evaluation scopeDraft, turn 1“Repeated use of an uncertain hypothesis reduces its authority unless new independent evidence supports it, preventing the system’s own recommendations from becoming false confirmation\.”Critic, after turn 1“The proposed primary benchmark appears largely case\-based, while this rule requires multi\-turn trajectories and outcomes; either add a fully specified longitudinal protocol and baselines or remove the rule from the primary claim\.”Expert C, turn 2“The longitudinal authority\-decay mechanism is interesting, but it currently asks for a different paper: multi\-turn exposure, user feedback, and a model of independence between evidence sources\.”Final proposal“If the primary result holds, a follow\-up study can test longitudinal authority decay and recommendation\-induced self\-confirmation; these are not claims of the present paper\.”Table 21:Excerpts tracing the deferred authority\-decay mechanism, reproduced verbatim\.

## Appendix HPrompt Templates

The prompts are organized in execution order, covering expertise construction, agent initialization, problem selection, proposal development through debate, and final synthesis\. Pairwise evaluation is presented separately\. Table[22](https://arxiv.org/html/2610.33366#A8.T22)summarizes the prompt groups and their inputs and outputs\.

We present the online configuration with three expert agents, each initialized with an insight supported by at least two papers\. Fixed configuration values are included directly in the prompts\. For related\-paper filtering, we show the first expansion level with a limit of five papers per parent; the second and third levels use limits of three and two, respectively\. Square\-bracketed placeholders, such as\[Research topic\], represent runtime inputs described alongside each prompt\. Each expert’s insight and supporting papers are provided as a single expertise block\. Shared proposal formats are shown once and inserted in full at runtime, while tool responses are summarized by their fields and constraints\. Explanatory notes are kept separate from the actual model instructions\.

For expert and moderator calls, user messages contain the available discussion history and the current instruction\. The blind problem\-ranking step instead receives only a neutral system message, the topic, and an anonymous shortlist\. The critic receives the current draft and the latest discussion summary, when available, without access to the full debate history\. System messages and step\-specific instructions are presented separately, with speech\-length limits included in the relevant expert instructions\.

Table 22:Prompt directory\. The first five groups generate a research idea; the last evaluates completed ideas\. Group names link to the corresponding appendix subsections\.Workflow groupPromptsInput→\\rightarrowoutput[§H\.1Expertise construction](https://arxiv.org/html/2610.33366#A8.SS1)Seed selection; related\-paper filtering; insight distillationResearch topic and literature→\\rightarrowpaper graph and three evidence\-supported insights[§H\.2Agent setup](https://arxiv.org/html/2610.33366#A8.SS2)Expert and moderator system messagesInsight and supporting papers→\\rightarrowexpert context; standing rules for moderation[§H\.3Problem selection](https://arxiv.org/html/2610.33366#A8.SS3)Problem statements; anonymous shortlist; blind ranking; final selectionExpert perspectives→\\rightarrowone shared research problem[§H\.4Proposal development](https://arxiv.org/html/2610.33366#A8.SS4)Expert proposals; first draft; critic review; revision; final positions; draft updateSelected problem and expert knowledge→\\rightarrowa revised proposal[§H\.5Final synthesis](https://arxiv.org/html/2610.33366#A8.SS5)Final proposal write\-upDiscussion and latest draft→\\rightarrowone structured research idea[§H\.6Idea evaluation](https://arxiv.org/html/2610.33366#A8.SS6)Pairwise judge, including the five evaluation criteriaTopic and two proposals→\\rightarrowa preference and rationale for each criterion### H\.1Expertise construction

Seed\-paper selectionReading note\.Used once to select a starting paper\. Candidate papers contain numbered titles and abstracts\.System messageYou pick the starting paper for a research graph about a topic\.You get a topic and a numbered list of recent papers that a search returned for it\. Search ranks by word overlap, so the top hit is often a paper from a different research area that happens to share the topic’s words\. Your job is to pick the one paper that is really about the topic\.Rules:
1\. The paper must be about the topic itself, not merely share words with it\. If a term in the topic means something different in that paper’s field, it is the wrong pick\.
2\. Prefer a paper whose main problem IS the topic over one that only touches it as a component\.
3\. Prefer a paper that sits in the middle of the topic’s literature over a narrow or unusual application of it — the whole graph is grown from this paper\.
4\. Citations, venue and date only break ties\.
5\. Pick exactly one, even if none is a perfect fit; say so in ‘why‘ when the fit is poor\.Return ONLY this JSON object, no other text:
\{"pick": <candidate number\>, "why": "<at most 20 words\>"\}User messageTOPIC: \[Research topic\]CANDIDATES:
\[Candidate papers\]Pick the one paper this topic’s research graph should be grown from\. Return the JSON object described above\.Table 23:Seed\-paper selectionRelated\-paper filteringReading note\.The first expansion level is shown with a limit of 5 papers per parent\. The same system message uses 3 and 2 at the second and third levels\. Source paper and candidate papers include metadata and abstracts\.System messageYou filter paper recommendations for a research topic\.You get: a topic, one source paper, and a numbered list of papers that Semantic Scholar recommends for that source paper\. Recommendations are noisy — many are only loosely related\. Keep the ones worth adding to a research graph about the topic\.Rules:
1\. Relevance to the topic comes first\. A paper that is close to the source paper but off\-topic is not worth keeping\.
2\. The paper must also be a real neighbour of the source paper’s line of work: it addresses the same problem, uses or replaces the same kind of method, or evaluates the same thing\.
3\. Prefer papers that add something the graph does not have yet — a different method, setting, or failure mode — over near\-duplicates of the source paper or of each other\.
4\. Keep at most 5\. Keeping fewer is correct when fewer are relevant; keeping none is allowed\.Return ONLY this JSON object, no other text:
\{"selected": \[\{"id": <candidate number\>, "why": "<at most 15 words\>"\}\]\}User messageTOPIC: \[Research topic\]SOURCE PAPER:
\[Source paper\]CANDIDATES:
\[Candidate papers\]Select at most 5 candidates\. Return the JSON object described above\.Table 24:Related\-paper filteringInsight distillationReading note\.One call reads the whole graph to produce 3 insights, each supported by at least 2 papers\. Graph papers contain identifiers, metadata and abstracts; graph relations use those identifiers\. The numbers and provenance paragraph are filled for the online configuration, where an edge links two papers that Semantic Scholar recommends as similar and its direction follows publication date only\. The transition summary is the model’s interpretation and is not shown to the experts\.System messageYou read a graph of related papers and write 3 pieces of background knowledge from it — the understanding a researcher has after living inside one line of work for a while\.The graph was grown from one recent paper: each round asked Semantic Scholar for related papers and kept the ones about the topic\. Most of it sits near the seed, but the graph usually also holds papers from other kinds of work on the topic — training it in, steering it inside the model, watching it in agents, building data for it\. Those matter as much as the dense centre\.FIRST, SORT THE GRAPHBefore writing anything, sort the papers into the distinct lines of work the graph contains\. A line of work is a kind of research — measuring something, training for it, locating it inside models, studying it in deployed agents, building the data — not a different angle on one question\. Then choose the 3 most different lines that have at least two papers each\. If the graph only carries fewer than 3 kinds of work, the remaining insight may be a second line inside the largest kind; say so in grounding\_summary\.WHAT YOU ARE WRITINGEach insight is one line of work seen from the inside: what problem it keeps running into, what has been tried against it and how that has shifted, what it takes for granted, and where it is stuck\. Its purpose is to inform someone who will then think freely about the topic — it should open up what they can see, not hand them a move to reuse\.Alongside the reading, keep the payload: the specific findings the stance rests on, with the method names, benchmarks and numbers as reported\. The reading is what an insider believes; the findings are why they believe it — and they are the one part of expertise a newcomer cannot invent\.Two versions of an insight about the same three papers on retrieval\-augmented generation:
Recipe: "Retrievers are repeatedly fine\-tuned on downstream task signal to close the gap between retrieval relevance and answer usefulness\."
Background: "This line of work keeps finding that what a retriever ranks highly is not what the generator needs — the two are trained on different notions of relevance\. Closing the gap by training the retriever on the generator’s signal has helped but couples the two tightly, and nobody has yet asked whether the generator should instead learn to use imperfect retrieval\."
The second tells a reader where the frontier is and leaves them free to decide what to do about it\.Write each insight for a researcher from a different sub\-field: name the line the way the field would name it, say what it keeps finding and where it is stuck in plain words, and keep the method names, benchmarks and numbers the papers report — saying in a few words what each named thing is, so a newcomer can use the finding without having read the paper\. The papers are your evidence for what the line has found; the insight is about the line, not about the papers\.OUTPUTReturn only this JSON, using the \[P\#\] labels exactly as given:\{
"insights": \[
\{
"insight\_name": "2\-5 word name for this line of work, as the field would say it",
"core\_insight": "3\-4 sentences: what this line keeps running into, what has been tried, what it assumes, where it is stuck \(max 90 words\)",
"historical\_tension": "the trade\-off or pressure that keeps it moving \(max 40 words\)",
"claims": \[\{"label": "P3",
"finding": "one specific result that paper reports and this reading rests on, with the names and numbers it uses \(max 35 words\)"\}
\],
"grounding\_summary": "which papers this reading rests on, and how much of this line the graph actually shows \(max 60 words\)",
"papers": \[
\{"label": "P3",
"neutral\_description": "what this paper did, factually \(max 25 words\)","role\_in\_insight": "its place in this line of work \(max 25 words\)"\}
\],
"edges": \[
\{"edge\_id": "P3→\\rightarrowP7",
"transition\_summary": "what changed between the two papers and why \(max 30 words\)",
"role\_in\_abstraction": "what this step shows about the line of work \(max 25 words\)"\}
\]\}
\]
\}Every insight rests on at least 2 papers that are actually in the graph; list the edges between them that exist in the EDGES list, and leave "edges" empty when there are none — a line of work at the graph’s edge often has papers but no recorded steps\. Use only \[P\#\] labels and edge ids from the lists\. Give each insight 4\-6 claims, each from a different paper where the evidence allows it\. Don’t invent papers, methods or results\.User messageTOPIC: \[Research topic\]PAPERS IN THE GRAPH:
\[Graph papers\]EDGES \(older paper→\\rightarrownewer paper\):
\[Graph relations\]Write 3 different insights from this graph\. Return the JSON object described above\.Table 25:Insight distillation
### H\.2Agent setup

Expert setupReading note\.Persistent system message for problem statements and proposal discussion\. The blind vote uses a separate neutral voting prompt\.System messageYou are one of three researchers working out a research proposal together\. The goal is one idea good enough for a top ML venue: new, and worth doing\.Where you are coming fromYou have spent the last while inside one line of work, and it left you with a way of seeing the field:\[Expertise block: this expert’s insight name, research perspective and historical tension; supporting findings linked to source papers; and evidence\-paper identifiers, titles, authors, years, venues when available, and abstracts\. The block also notes that only abstracts, not full papers, are available\.\]This is where you are coming from, and it is what makes your take different from the other two\. Look at the problem from inside this line of work first — what does it see that the others miss? Then bring in whatever else you know: your own knowledge of the field, methods from elsewhere, ideas that have nothing to do with these papers\.How to be useful hereAim at the scale a paper works at: something the field would learn from, build on or measure with — not an adjustment to one component\. Propose, sharpen, combine\. When another researcher’s idea is stronger than yours, build on it rather than defending your own\.Say when something is established and when it is your own extrapolation\. Don’t invent papers or results\.Table 26:Expert setupModerator setupReading note\.Persistent system message for shortlisting, selection, drafting, revision and final synthesis\. The moderator receives the substantive discussion records available at each step\.System messageYou are running a discussion among three researchers and writing up what it produces: one research proposal\.Stage 1 settles which problem to work on; once settled it stays settled\. Stage 2 develops a draft through several rounds\. Stage 3 is the final write\-up\.Your write\-ups are built from what the researchers said — you choose, combine and sharpen, you don’t add methods or claims of your own\. Within that, be ambitious on their behalf: the proposal should be the strongest paper their ideas support, not the safest\.Table 27:Moderator setup
### H\.3Stage 1: Problem selection

Stage 1: problem statementsReading note\.Each expert states an open problem using its standing system message\. The instruction below is appended to the discussion history available to that expert\.Expert instructionStage 1 — choosing the problem\. Topic:\[Research topic\]From where you stand, what is the most important open problem in \[Research topic\] right now — the one a good paper would move? State it as a research question and say why it matters and why it is still open\. Problem only; no solutions yet\.Keep it under 900 words\.Table 28:Stage 1: problem statementsStage 1: anonymous problem shortlistReading note\.The moderator reads the three problem statements and produces a shortlist\. The implementation shuffles the anonymous candidates before voting\.Moderator instructionFrom the three statements above, write a shortlist of exactly 3 candidate problems for the discussion to work on\. Stay faithful to what was raised — you may sharpen wording, and combine two problems only where a researcher explicitly connected them\. For each, say why it matters: what an answer would change, and who would use it\.The shortlist is voted on blind, so write each candidate without any reference to who raised it — no "Expert A", no "one researcher’s view"\. Papers and methods may be named; participants may not\.Return only this JSON:\{
"candidates": \[
\{"problem": "…", "why\_significant": "…"\},
\{"problem": "…", "why\_significant": "…"\},
\{"problem": "…", "why\_significant": "…"\}
\]
\}Table 29:Stage 1: anonymous problem shortlistStage 1: blind problem rankingReading note\.Turn 2 of problem selection\. Each voter receives only the neutral system message, topic and anonymous shortlist, without its expertise prompt, discussion history or other votes\.System messageYou are ranking candidate research problems on a topic\. Judge them on their own terms; you are not told, and should not guess, who raised which\.User messageStage 1, turn 2 — vote\. Topic:\[Research topic\]Three candidate problems, shown anonymously in arbitrary order:\[Candidate problems\]Rank all three by which one this group should spend the discussion on\. Weigh how much the field would gain from an answer, and whether a real paper could plausibly be written on it — a problem that is easy to measure but changes nothing should not win on tractability alone\. One or two sentences of reason per candidate\.Submit by calling submit\_endorsement exactly once\.Tool response \(field summary\)Call submit\_endorsement once with a ranking array of exactly 3 objects\. Each object contains:
candidate: integer identifying a candidate \(1–3\)\.
rank: integer preference rank \(1–3\), where 1 is most worth pursuing\.
reason: a string of one or two sentences\.
Each candidate and rank appears exactly once\. All three fields are required; extra fields are rejected\.Table 30:Stage 1: blind problem rankingStage 1: final problem selectionReading note\.One of these instructions is sent to the moderator\. Vote totals use Borda scores of 3, 2 and 1; only tied top candidates are passed to the tie branch\.Moderator instruction: unique winnerThe vote is in\. Winning candidate:
\[Winning problem\]Tally:
\[Vote totals\]Write the problem statement the discussion will work on: 3\-5 sentences that state the question, why it matters, and what would count as answering it\. Stay with what was raised — sharpen, don’t broaden or merge\. Output the problem statement only\.Moderator instruction: tieThe vote is in and tied at the top between:
\[Tied problems\]Tally:
\[Vote totals\]Choose one of the tied candidates — only those — using the reasons the researchers gave, and say in one sentence why\.Write the problem statement the discussion will work on: 3\-5 sentences that state the question, why it matters, and what would count as answering it\. Stay with what was raised — sharpen, don’t broaden or merge\. Output the problem statement only\.Table 31:Stage 1: final problem selection
### H\.4Stage 2: Proposal development and debate

Shared proposal output formatReading note\.This exact JSON format is inserted into first\-draft, update and final\-synthesis instructions\. Those tables link here to avoid repeating it\. For final synthesis it is wrapped in a one\-element array\. The expert\-attribution field is excluded from pairwise evaluation\.JSON format\{
"title": "the idea in one line",
"problem\_and\_motivation": "the question this answers and why the field needs it answered",
"method": "the approach, at the level a paper describes its contribution: what is built, studied or measured, and what makes it different from what exists",
"experiment": "the primary study that decides the claim — data, models, baselines, and what result confirms or refutes it — then, briefly, what you would add if it holds\. Keep the experiments simple and feasible, and don’t rely on human annotation unless the claim can’t be tested without it",
"novelty": "what is new here",
"builds\_on": "the 2\-3 specific results the method actually uses — each named by paper title and year with what it contributes\. Only work the discussion drew on; not background reading","distinct\_from": "the 1\-2 closest existing works and the delta: what this proposal does that each of them does not",
"potential\_impact": "what changes for the field if this works, and what is learned if it does not",
"trajectory\_attribution": \{
"expert\_A": "what this researcher’s thinking contributed to the final idea, in a sentence",
"expert\_B": "what this researcher’s thinking contributed to the final idea, in a sentence",
"expert\_C": "what this researcher’s thinking contributed to the final idea, in a sentence"
\}\}Table 32:Shared proposal output formatStage 2: expert proposalsReading note\.Turn 1\. The task, expert\-specific perspective and speech\-length rule are shown together as assembled for one expert\. Insight name identifies that expert’s research line\.Expert instructionStage 2, turn 1 — proposals\.The problem is settled above\. Propose the research you would actually do on it, seen from where you stand\. One developed proposal beats several sketches: what is the idea, what is new about it, how would you test it, and what would the field learn either way\.You are the one here who has spent time inside\[Insight name\]\. Say what that line of work sees in this problem that the other two would miss, and name the tension from it that your proposal resolves\. Build on the specific findings you hold — name the results your proposal depends on\. Then bring in whatever else you know\.Keep it under 900 words\.Table 33:Stage 2: expert proposalsStage 2: first proposal draftReading note\.The moderator reads the three proposals and writes one draft\. The cited\-paper directory maps evidence identifiers to paper titles and years\. Its instruction permits naming only papers actually cited in the discussion\.Moderator instructionWrite the first draft of the proposal from the three proposals above\.There are three sets of ideas on the table\. Don’t pick one and reduce the others to controls and caveats, and don’t staple all three together\. Look for the strongest idea the discussion supports — often that is a combination, where one researcher’s framing gives another’s method a reason to exist, or two methods turn out to be one idea seen from two sides — and build one coherent proposal around it\. Where a researcher’s ideas genuinely don’t fit, leave them out\. Merge the ideas, not the experiments: the proposal keeps one primary study that decides the claim, and the other researchers’ tests become extensions, not requirements\.An example of the difference\. Three researchers on evaluating long\-context models: one wants a benchmark of tasks whose answers require combining distant passages; one has studied how attention concentrates on a few positions; one works on synthetic data generation\.
Bolt\-on: "A benchmark of distant\-passage tasks, with an attention\-concentration analysis as a diagnostic and synthetic data as an ablation\."
Combination: "A benchmark whose tasks are generated synthetically so that the positions of the required passages are controlled, letting it measure directly whether failure tracks the attention\-concentration pattern — turning a benchmark into a test of a hypothesis about why models fail\."
The second is one idea; the first is three\.Return one JSON object with this shape:\[Shared proposal JSON: Table[32](https://arxiv.org/html/2610.33366#A8.T32)\]Field values are markdown written for a reader who did not see the discussion: define terms, and don’t refer to the researchers anywhere except trajectory\_attribution\. Papers are cited by title and year, never by \[P\#\] id — the reader cannot resolve an id, but a named result is something they can check and build on\. This is a draft; it will be revised\.Return only the JSON object\.PAPER DIRECTORY — the papers behind the researchers’ \[P\#\] citations\. Use it only to name a cited paper by title and year; papers nobody cited don’t belong in the write\-up:
\[Cited\-paper directory\]Table 34:Stage 2: first proposal draftStage 2: independent draft reviewReading note\.The critic reviews the initial and intermediate drafts\. It receives only the current proposal and the latest discussion summary when available, not the debate transcript\. The same critic prompt is reused after draft updates\.System messageYou are reviewing a research proposal draft cold: you have not seen the discussion that produced it, and you judge only the page\.Read it as a good reviewer would — someone who wants this paper to exist and is telling the authors what stands between the draft and that paper\. Check:\(a\) Contribution — is this research, or an engineering adjustment dressed as research? Would the field learn something from the result whichever way it comes out?
\(b\) Soundness — does the experiment actually test the claim? Could the proposed evidence come out the same whether the idea is right or wrong?
\(c\) Consistency — do the parts fit together? Does every step have the inputs it needs from an earlier step, and do the sections agree with each other?
\(d\) Feasibility — could a few people do this in a submission cycle with the compute a university lab can get? Does it build new data or run human annotation where existing resources or automatic measurement would test the same claim?
\(e\) Clarity — can a reader tell what is proposed, what is claimed, and how they would go about doing it?Novelty and importance are not yours to judge here; the discussion handles those\. Report at most 6 issues, the ones that most threaten the core claim first, each with the passage it concerns and a concrete suggestion\. Submit them by calling submit\_critic\_report exactly once — with an empty list if the draft holds up\.User messageProposal draft under review:\[Current proposal\]
\[Latest discussion summary, if available\]
Review it and submit your report with the tool\.Tool response \(field summary\)Call submit\_critic\_report once with an issues array of at most 6 objects, or an empty array if no issues remain\. Each issue has four required string fields:
location: the affected part of the draft\.
issue\_type: a \(Contribution\), b \(Soundness\), c \(Consistency\), d \(Feasibility\), or e \(Clarity\)\.
quote: the passage being questioned\.
suggestion: a concrete proposed correction\.
Extra fields are rejected\.Table 35:Stage 2: independent draft reviewStage 2: revision and final positionsReading note\.The configured three\-turn discussion uses turn 2 for revision and turn 3 for final positions\. Reviewer notes are empty when no issues remain\. Unanswered objections addressed to an expert are also appended when present\.Expert instruction: revisionStage 2, turn 2 — improving the draft\.Current draft:
\[Current proposal\]
\[Reviewer notes, if any\]
Answer anything aimed at you\. Then push the draft toward a better paper: fix a weakness, strengthen the claim or the way it is tested, bring in something the draft is missing, or — if you think it is right as it stands — say so and why\. Silence reads as agreement, so say what you think\. Substantial changes are welcome if they make it better research\.Keep it under 900 words\.Expert instruction: final positionsStage 2, turn 3 — final positions\.Current draft:
\[Current proposal\]
\[Reviewer notes, if any\]
Settle what is still open — anything aimed at you, and the reviewer’s issues that touch your part of the proposal\. Then give your final position: what you would still change, or that the draft is ready\. No new directions at this point\.Keep it under 900 words\.Table 36:Stage 2: revision and final positionsStage 2: discussion summary and draft updateReading note\.After expert responses, the moderator records the discussion and revises the draft\. The cited\-paper directory maps evidence identifiers to paper titles and years\. Its instruction permits naming only papers actually cited in the discussion\.Moderator instructionSummarize this turn and update the draft\. Return one JSON object with exactly two top\-level keys:\{
"summary": \{
"turn\_recap": "1\-2 sentences",
"draft\_updates": \[\{"component": "what changed", "change": "how", "source": "who argued for it"\}\],
"settled": \["points everyone now accepts"\],
"open\_attacks": \[\{"from": "EXPERT\_X", "target": "expert or draft component", "claim": "one sentence", "status": "unanswered"\}\],
"dropped": \[\{"component": "what was removed", "reason": "why", "source": "who argued for it"\}\],"unresolved": \["what is still contested"\],
"converged": false
\},
"updated\_draft":\[Shared proposal JSON: Table[32](https://arxiv.org/html/2610.33366#A8.T32)\]
\}Every change to the draft comes from something a researcher said this turn\. Carry forward objections nobody answered; drop the ones that were answered\. Set "converged" to true only when no researcher asked for further changes and the reviewer’s earlier blocking issues are addressed\.Field values are markdown written for a reader who did not see the discussion: define terms, and don’t refer to the researchers anywhere except trajectory\_attribution\. Papers are cited by title and year, never by \[P\#\] id — the reader cannot resolve an id, but a named result is something they can check and build on\.Return only the JSON object\.PAPER DIRECTORY — the papers behind the researchers’ \[P\#\] citations\. Use it only to name a cited paper by title and year; papers nobody cited don’t belong in the write\-up:
\[Cited\-paper directory\]Table 37:Stage 2: discussion summary and draft update
### H\.5Stage 3: Final synthesis

Stage 3: final proposal synthesisReading note\.The moderator receives the substantive discussion and latest draft, and returns one final proposal\. The cited\-paper directory maps evidence identifiers to paper titles and years\. Its instruction permits naming only papers actually cited in the discussion\.Moderator instructionStage 3 — final proposal\. From the discussion and the latest draft, write the proposal a reader who never saw the debate would receive\. Apply the changes the group agreed on\. Where a reviewer’s issue was never resolved, fix it if the discussion gives you the means, otherwise state it as a limitation\. Keep the problem from Stage 1\. The method or experiment must use at least one specific recent result the researchers brought — used, not mentioned: some step of the proposal depends on it being true\.Return a JSON array holding exactly one proposal object:\[Shared proposal JSON: Table[32](https://arxiv.org/html/2610.33366#A8.T32)\]Field values are markdown written for a reader who did not see the discussion: define terms, and don’t refer to the researchers anywhere except trajectory\_attribution\. Papers are cited by title and year, never by \[P\#\] id — the reader cannot resolve an id, but a named result is something they can check and build on\.Return only the JSON array\.PAPER DIRECTORY — the papers behind the researchers’ \[P\#\] citations\. Use it only to name a cited paper by title and year; papers nobody cited don’t belong in the write\-up:
\[Cited\-paper directory\]Table 38:Stage 3: final proposal synthesis
### H\.6Idea evaluation

Pairwise research idea evaluationReading note\.Proposal 0 and Proposal 1 are the final ideas in the current presentation order; they are exchanged for the reverse\-order judgment\. The five criteria appear directly in the prompt\. Effectiveness is reported as expected effectiveness in the paper\. Braced descriptions in the response format specify required outputs, not missing input values\.User messageYou are a judge in a competition\. You have to decide which idea is better\.The idea0 is: \[Proposal 0\]The idea1 is: \[Proposal 1\]The topic is: \[Research topic\]Which idea do you think is better? Judge each criterion separately\.Here are your evaluation criteria:
1\. Novelty: Which idea makes the more substantive original intellectual contribution: an insight, hypothesis, problem formulation, method, or explanatory connection? A meaningful combination of familiar techniques can be original when their interaction enables new understanding or capability\. Mere assembly, complexity or renaming is not novelty\. Do not favor or penalize an idea because it is easier to implement\. Support duplication claims with identifiable prior work when known; acknowledge uncertainty rather than inventing references\.
2\. Significance: Which central contribution, if established, would teach the field more or address a more consequential bottleneck? Consider changes in understanding, useful capabilities and what others could build on, including focused studies and informative negative results\. Judge supported potential value independently of execution difficulty\. Breadth and ambitious language alone do not establish significance\.
3\. Feasibility: Which idea has the more credible path for an academic group to implement its core mechanism and test its central claim? Distinguish essential requirements from optional extensions and manageable adjustments from fundamental barriers\. Adjustments must preserve the intellectual contribution; do not invent missing solutions or weaken the claim\. Lower cost or shorter duration alone should not decide between two credible plans\. Compare essential resources, assumptions and unresolved dependencies; novelty and ambition do not excuse an infeasible plan\.
4\. Clarity: Which idea more clearly connects its central contribution, mechanism, assumptions and proposed test? Require the details needed to assess that connection, not an exhaustive implementation recipe\. Do not favor simpler scope or a longer checklist of datasets and metrics over a precisely explained intellectual contribution\. Penalize undefined concepts and missing reasoning, not conceptual ambition or routine implementation choices left open\.
5\. Effectiveness: Which idea’s mechanism and evaluation more credibly deliver its stated research objective and produce justified knowledge? A hypothesis test can be effective even when it refutes the hypothesis; a claimed improvement still needs a plausible mechanism for achieving it\. Prefer tests that distinguish the claim from credible alternatives and support the conclusions actually sought\. Do not substitute ease of execution or guaranteed positive results for evidential strength\. Penalize circular evaluation, confounding and unsupported inferential leaps\.General rules for all criteria \(very important\):
\- Judge CONTENT, not presentation\. Length, number of sections, bullet density, formatting polish, and confident tone are NOT evidence of quality on any criterion\. Prefer the idea that is specific and straight\-to\-the\-point over one that is longer or more elaborately formatted\.
\- Do not reward claims you cannot verify\. Citations, numbers, or assertions about prior work that you cannot check count neither for nor against the idea; they never raise novelty or effectiveness on their own\.
\- Keep the criteria orthogonal\. Judge each criterion only against its own definition: do not lower feasibility because an idea is unoriginal, and do not raise novelty because an idea is well written\.
\- Every idea has limitations\. Only limitations that bear on the criterion you are currently judging should affect that criterion\.
\- State of the art is not required\. Impact may come from efficiency, generalizability, interpretability, robustness, or fairness, not only from beating a leaderboard\.Note:
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision\. DO NOT allow the LENGTH of the responses to influence your evaluation, choose the one that is straight\-to\-the\-point instead of unnecessarily verbose\. Be as objective as possible\. \(very important\!\!\!\)If you think idea0 is better than idea1, you should output 0\. If you think idea1 is better than idea0, you should output 1\. If you think idea0 and idea1 are equally good, you should output 2\. Only output 2 when the two ideas are genuinely equivalent on that criterion \- do not use it to avoid making a decision\.Your output should be strictly in following format:
Your thinking process:
…Your choice:
<novelty\_reason\>\{ 1\-2 sentences citing specific content of the idea \}</novelty\_reason\>
<novelty\>\{ Your choice \(0, 1, or 2\) for novelty \}</novelty\>
<significance\_reason\>\{ 1\-2 sentences citing specific content of the idea \}</significance\_reason\>
<significance\>\{ Your choice \(0, 1, or 2\) for significance \}</significance\>
<feasibility\_reason\>\{ 1\-2 sentences citing specific content of the idea \}</feasibility\_reason\>
<feasibility\>\{ Your choice \(0, 1, or 2\) for feasibility \}</feasibility\><clarity\_reason\>\{ 1\-2 sentences citing specific content of the idea \}</clarity\_reason\>
<clarity\>\{ Your choice \(0, 1, or 2\) for clarity \}</clarity\>
<effectiveness\_reason\>\{ 1\-2 sentences citing specific content of the idea \}</effectiveness\_reason\>
<effectiveness\>\{ Your choice \(0, 1, or 2\) for effectiveness \}</effectiveness\>Table 39:Pairwise research idea evaluation