DeepInstructor：一种用于经验驱动创意评估的智能代理AI导师

# An Agentic AI Instructor for Experience-Driven Idea Evaluation
Source: [https://arxiv.org/html/2609.22104](https://arxiv.org/html/2609.22104)
Fang GuoQinglin QiQi ZhuYun LuoJianhao YanMinjun ZhuQiujie XieDehong ZhengAffiliation:Tongji University, Westlake University, Zhejiang University, Shanghai AI Lab, Fudan UniversityCorrespondence:[prc@tongji\.edu\.cn](mailto:[email protected])Yue Zhang

###### Abstract

As automated scientific discovery advances, Large Language Models \(LLMs\) can now generate research ideas at an unprecedented scale, shifting the bottleneck from idea generation to idea evaluation\. Existing evaluators mainly rely on parametric LLM knowledge or unstructured retrieval, producing judgments that lack the experience\-grounded reasoning used by human instructors\. To address this, we proposeDeepInstructor, an agentic framework that formulates idea evaluation as reasoning over structured scholarly experience\. DeepInstructor constructs an Experience Graph from 58,607 peer reviews and employs a ReAct\-based agent to retrieve dimension\-specific evidence for traceable evaluation\. We further introduceDeepInstruct, a dataset with controlled pairwise comparisons across novelty, significance, and feasibility\. Experiments show that DeepInstructor substantially outperforms existing baselines, improving Hit@1 and Hit@2 alignment with human judgments by 24\.4% and 29\.7%, respectively\. Our findings suggest that scientific idea evaluation can be grounded in explicit reasoning over structured scholarly experience\.222Code, prompts, datasets and interview study results can be found on[Anonymous GitHub](https://anonymous.4open.science/r/ExpInstructor-5543/README.md)\.

††footnotetext:\* Equal contribution\.†††Corresponding Author\.## 1Introduction

The rapid advancement of AI enables both human researchers and emerging AI Scientist systems to generate research ideas at an unprecedented scale\([19](https://arxiv.org/html/2609.22104#bib.bib9);[24](https://arxiv.org/html/2609.22104#bib.bib2);[23](https://arxiv.org/html/2609.22104#bib.bib3);[8](https://arxiv.org/html/2609.22104#bib.bib1);[7](https://arxiv.org/html/2609.22104#bib.bib24)\)\. As idea generation becomes increasingly cheap and automated\([17](https://arxiv.org/html/2609.22104#bib.bib23);[6](https://arxiv.org/html/2609.22104#bib.bib21);[13](https://arxiv.org/html/2609.22104#bib.bib22)\), evaluating idea quality remains costly and difficult, creating a growing imbalance in the scientific discovery process\. Consequently, the bottleneck of scientific discovery is shifting from generating ideas to judging them\([14](https://arxiv.org/html/2609.22104#bib.bib6);[10](https://arxiv.org/html/2609.22104#bib.bib10)\), making robust idea evaluation a fundamental requirement for both AI Scientist systems and scalable human\-AI collaboration\([23](https://arxiv.org/html/2609.22104#bib.bib3);[12](https://arxiv.org/html/2609.22104#bib.bib4)\)\.

Recent work on idea evaluation can be characterized along two orthogonal dimensions: what criteria to evaluate and how to evaluate\. Existing approaches vary widely in criteria—from single acceptability scores\([3](https://arxiv.org/html/2609.22104#bib.bib14)\)to pairwise ranking and multi\-dimensional rubrics—but it remains unclear whether these reflect how human instructors assess ideas in practice\([3](https://arxiv.org/html/2609.22104#bib.bib14);[18](https://arxiv.org/html/2609.22104#bib.bib5);[10](https://arxiv.org/html/2609.22104#bib.bib10);[1](https://arxiv.org/html/2609.22104#bib.bib13)\)\. Methodologically, prompt\-based approaches rely on parametric LLM knowledge and often misalign with human judgment\([14](https://arxiv.org/html/2609.22104#bib.bib6)\), while retrieval\-based methods introduce noise due to irrelevant evidence from the content of papers\([10](https://arxiv.org/html/2609.22104#bib.bib10)\)\. Despite these differences, a shared limitation persists:current evaluators fail to align with human expert judgment in either their criteria or their methodology\.

![Refer to caption](https://arxiv.org/html/2609.22104v1/pipeline.png)Figure 1:Framework of our proposed DeepInstructor\.To better characterize human\-aligned evaluation, we first conduct a pilot study with 15 Computer Science faculty members \(see Appendix[A](https://arxiv.org/html/2609.22104#A1)for details\)\. The study reveals two key insights: \(1\)Novelty,Significance, andFeasibilityare the three criteria that instructors consistently prioritize, and \(2\)the prior experience instructors rely on often takes the form of stance\-bearing judgments toward existing techniques, algorithms, and tasks\.These findings suggest that idea evaluation is fundamentally an experience\-grounded reasoning process, where judgments are formed through accumulated stances toward entities and their relations\. We therefore conceptualize academic experience as a collection of such stance\-bearing statements over conceptual entities and relations, providing a structured and human\-aligned basis for evaluation\.

For example, as shown in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), consider an idea proposing to improve low\-resource machine translation by intentionally generating hallucinated incorrect translations and using them for self\-correction\. Rather than viewing the idea in isolation, experienced reviewers may relate it to existing paradigms, such as learning from incorrect answers and self\-correction prompting\. Such experience\-grounded reasoning may lead reviewers to conclude that the idea recombines existing self\-correction mechanisms rather than representing a fundamentally new paradigm\.

Based on these insights, we proposeDeepInstructor, an experience\-driven framework for research idea evaluation that explicitly models scholarly experience and enables grounded, human\-aligned reasoning\. Our approach consists of three key components\. First, we construct anExperience Graphfrom large\-scale peer\-review corpora with 58,607 cases, where entities and their evaluative relations encode accumulated scholarly experience as structured, stance\-bearing knowledge\. Second, we design anExperience Retriever\. This graph\-based retrieval module enables structured access to relevant experience by first identifying key entities in the input idea and then retrieving their associated evaluative relations conditioned on the target dimension\. Third, we develop anidea evaluation frameworkbased on the ReAct paradigm\([21](https://arxiv.org/html/2609.22104#bib.bib18)\), which integrates reasoning and retrieval to gather evidence and produce dimension\-specific judgments dynamically\.

To enable controlled and systematic evaluation of research idea assessment, we further constructDeepInstruct, a benchmark derived from large\-scale peer\-review corpora containing nearly 20,000 ICLR submissions and reviews\. By organizing papers along their task–method structure, we create controlled pairwise comparisons: ideas with similar tasks but different methods enable evaluation of novelty and feasibility, while those with similar methods across tasks support assessment of significance\.

DeepInstructor is, to our knowledge, the first framework to model scholarly experience as structured evaluative relations and to leverage them for idea evaluation\. We make three main contributions: \(1\) We propose DeepInstructor, an experience\-driven evaluation paradigm that represents scholarly experience as a structured graph and enables dimension\-specific retrieval for grounded idea assessment\. \(2\) We introduce DeepInstruct, a principled benchmark construction pipeline that transforms peer\-review corpora into controlled pairwise evaluation tasks\. \(3\) We demonstrate that grounding evaluation in structured scholarly experience substantially improves alignment with human judgments, highlighting the potential of experience\-grounded reasoning for scientific idea evaluation\.

## 2Related Work

Automatic Research Idea Evaluation\.Prior systems differ widely in evaluation criteria: DeepReview\([25](https://arxiv.org/html/2609.22104#bib.bib12)\)offers reviewer\-style judgments but lacks instructor\-oriented dimensions; GraphEval\([3](https://arxiv.org/html/2609.22104#bib.bib14)\)predicts only a single overall score for the entire paper without dimension\-specific evaluation; ResearchAgent\([1](https://arxiv.org/html/2609.22104#bib.bib13)\)uses 15 redundant dimensions; SciJudge\([16](https://arxiv.org/html/2609.22104#bib.bib19)\)regards citation count as a metric; and[15](https://arxiv.org/html/2609.22104#bib.bib15)evaluates along four axes\. Methodologically, literature\-based Retrieval\-Augmented\-Generation \(RAG\) tools\([10](https://arxiv.org/html/2609.22104#bib.bib10)\)retrieve evidence but miss reviewers’ critical expertise, while review\-based fine\-tuning\([25](https://arxiv.org/html/2609.22104#bib.bib12)\)and manual annotation\([15](https://arxiv.org/html/2609.22104#bib.bib15)\)remain costly\. We instead extract reviewer experience directly and build an experience\-grounded RAG agent for more actionable evaluations\.

Knowledge Graph Construction and RAG\.LLM\-based OpenIE increasingly produces schema\-free triples\([11](https://arxiv.org/html/2609.22104#bib.bib17)\); Knowledge Graph Construction pipelines treat these as high\-recall inputs that require normalization and linking\([22](https://arxiv.org/html/2609.22104#bib.bib7);[9](https://arxiv.org/html/2609.22104#bib.bib16)\), which we apply when constructing our Experience Graph\. Integrating KGs with RAG has further improved complex reasoning\([4](https://arxiv.org/html/2609.22104#bib.bib8);[5](https://arxiv.org/html/2609.22104#bib.bib11)\), motivating our tailored retriever to efficiently locate reviewer experience\.

## 3Methodology

##### Problem Formulation\.

Given a research ideaIIthat consists of a problem formulation, a methodological sketch, and an experimental design, the idea evaluatorffis expected to generate structured evaluations along certain dimensions\. Based on pilot interviews with domain experts, we focus on three key dimensions:Novelty, which measures the originality of the idea relative to existing approaches;Significance, which captures the importance and potential impact of the underlying problem; andFeasibility, which reflects both the practicality of implementation and the expected effectiveness of the proposed method\.

For each dimension, the evaluator outputs a scalar score reflecting its judgment\. Formally, this process is defined as:f⁡\(I\)→\{sd\}d∈𝒟,f\(I\)\\rightarrow\\\{s\_\{d\}\\\}\_\{d\\in\\mathcal\{D\}\},where𝒟=\{Novelty,Significance,Feasibility\}\\mathcal\{D\}=\\\{\\text\{Novelty\},\\text\{Significance\},\\text\{Feasibility\}\\\}, andsds\_\{d\}is the corresponding scalar score for each dimension\.

##### Method Overview\.

DeepInstructoris an agentic framework for research idea evaluation that leverages structured experience distilled from peer reviews\. To encode and reuse historical judgements and experience, DeepInstructor consists of three components: \(1\) anExperience Graphthat stores entities and their evaluative relations as externalized scholarly experience; \(2\) anExperience Retrieverthat performs entity\- and relation\-aware retrieval over the graph; and \(3\) anidea evaluation frameworkbased on ReAct that integrates reasoning and retrieval to produce dimension\-specific, experience\-grounded judgments\. The overall pipeline is illustrated in Figure[1](https://arxiv.org/html/2609.22104#S1.F1)\.

### 3\.1Experience Graph Construction

Peer reviews encode rich evaluative knowledge grounded in prior research experience\. To capture and reuse such historical knowledge, we construct anExperience Graphfrom ICLR 2024 and ICLR 2025 reviews\.

##### Extracting Triples\.

We treat each review as a fundamental unit for experience extraction\. Using a prompted LLM, we extract scientific entities \(nodes\) and identify their precise evaluative or methodological relations \(edges\), accompanied by the original textual evidence\. For example, as shown in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), given the review statement, “wrong\-over\-wrong alignment allows LLMs to learn correctness by distinguishing between varying shades of wrong,” the model extracts two entities \(wrong\-over\-wrong alignment \(α\\alpha\),learning from incorrect answers \(β\\beta\)\), connected by the relationenables\. In this way, stance\-bearing scholarly judgments are transformed into structured experience triples\. Finally, we extract 416,330 triplets in total\.

![Refer to caption](https://arxiv.org/html/2609.22104v1/figure2_triple_extraction.png)Figure 2:Examples of Triplets Extraction\.
##### Aggregating and Embedding\.

We aggregate extracted triples into a unified Experience Graph by merging identical nodes, transforming scattered reviews into subgraphs that capture both positive and negative historical evidence\. Finally, we compute embeddings for all nodes and edges to enable semantic retrieval, establishing this graph as DeepInstructor’s external memory\. Formally, we define the Experience Graph as a directed labeled graph:𝒢=\(𝒱,ℰ\),\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\),where𝒱\\mathcal\{V\}denotes the set of entities andℰ\\mathcal\{E\}denotes the set of relational edges\. Each edge is represented as:e=\(vi,r,vj,t\)∈ℰ,e=\(v\_\{i\},r,v\_\{j\},t\)\\in\\mathcal\{E\},wherevi,vj∈𝒱v\_\{i\},v\_\{j\}\\in\\mathcal\{V\}are scientific entities after aggregating,rris the extracted relation , andttis the associated textual evidence\.

Figure[2](https://arxiv.org/html/2609.22104#S3.F2)shows more specific examples of how a graph is built from reviews\. For detailed statistics of the constructed Experience Graph and the corresponding graph coverage evaluation, please refer to Appendix[C\.2](https://arxiv.org/html/2609.22104#A3.SS2)\.

### 3\.2Experience Retriever

We design a graph\-based retrieval tool called Experience Retriever\. As illustrated in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), the retriever is designed to mimic how human experts recall prior experience: they first identify the concepts most relevant to the current idea, and then focus on the particular evaluative relations that matter for the target judgment\. Following this intuition, our retriever operates in two stages:*knowledge entity retrieval*and*relationship retrieval*\.

Formally, we define a retrieval operator:ℛ:\(K,Kc,R,Rc\)↦ℰ′,\\mathcal\{R\}:\(K,K\_\{c\},R,R\_\{c\}\)\\mapsto\\mathcal\{E\}^\{\\prime\},whereKKdenotes the knowledge entity query,KcK\_\{c\}is the number of retrieved entities,RRis the relationship query, andRcR\_\{c\}is the number of returned experience edges\. The outputℰ′⊆ℰ\\mathcal\{E\}^\{\\prime\}\\subseteq\\mathcal\{E\}is a set of relevant edges in the Experience Graph\. Given this input, the retriever operates in the following two steps:

##### Step 1: Knowledge Entity Retrieval\.

The first step aims to identify graph nodes that are semantically related to the query conceptKK\. Embedding\-based retrieval is adopted here to guarantee semantic alignment across different surface forms, enabling robust matching of conceptually similar entities\.

Letϕv​\(⋅\)\\phi\_\{v\}\(\\cdot\)denote the embedding function for entity nodes\. Given the entity queryKK, we retrieve the top\-KcK\_\{c\}most similar entities:𝒱K=Top⁡\-​Kc​\(\{sim⁡\(ϕv​\(K\),ϕv​\(v\)\)∣v∈𝒱\}\),\\mathcal\{V\}\_\{K\}=\\operatorname\{Top\}\\text\{\-\}K\_\{c\}\\Big\(\\\{\\mathrm\{sim\}\(\\phi\_\{v\}\(K\),\\phi\_\{v\}\(v\)\)\\mid v\\in\\mathcal\{V\}\\\}\\Big\),wheresim⁡\(⋅,⋅\)\\mathrm\{sim\}\(\\cdot,\\cdot\)is cosine similarity\. These retrieved entities define the candidate retrieval space\. Specifically, we collect all edges incident to any retrieved entity:ℰcand=\{e∈ℰ∣∃v∈𝒱K,v∈e\}\.\\mathcal\{E\}\_\{\\mathrm\{cand\}\}=\\\{\\,e\\in\\mathcal\{E\}\\mid\\exists v\\in\\mathcal\{V\}\_\{K\},\\;v\\in e\\,\\\}\.

##### Step 2: Relationship Retrieval\.

After identifying the relevant entities, the second step focuses on selecting the relations that best match the current evaluative need\. By matching the edges against a relationship query, this step effectively filters out irrelevant connections, ensuring that only the historical evidence tailored to the current evaluation dimension is retrieved\.

Letϕr​\(⋅\)\\phi\_\{r\}\(\\cdot\)denote the embedding function for relation texts\. For each candidate edgee=\(vi,re,vj,x\)∈ℰcand,e=\(v\_\{i\},r\_\{e\},v\_\{j\},x\)\\in\\mathcal\{E\}\_\{\\mathrm\{cand\}\},we compute its relevance to the relationship queryRRby matchingRRwith the edge relationrer\_\{e\}\. We then retrieve the top\-RcR\_\{c\}most relevant candidate edges:ℰR=Top⁡\-​Rc​\(\{sim⁡\(ϕr​\(R\),ϕr​\(re\)\)∣e∈ℰcand\}\)\.\\mathcal\{E\}\_\{R\}=\\operatorname\{Top\}\\text\{\-\}R\_\{c\}\\Big\(\\\{\\mathrm\{sim\}\(\\phi\_\{r\}\(R\),\\phi\_\{r\}\(r\_\{e\}\)\)\\mid e\\in\\mathcal\{E\}\_\{\\mathrm\{cand\}\}\\\}\\Big\)\.The final retrieved evidence is given byℛ⁡\(K,Kc,R,Rc\)=ℰR,\\mathcal\{R\}\(K,K\_\{c\},R,R\_\{c\}\)=\\mathcal\{E\}\_\{R\},where each edge inℰR\\mathcal\{E\}\_\{R\}is associated with its supporting textual evidencett\. This two\-stage design decouples concept matching from relation matching, enabling dimension\-specific retrieval for different evaluation criteria\.

For example, as shown in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), given the query “self\-correction through wrong answers,” the retriever first identifies the most relevant entities \(e\.g\.,⟨α,Ω,…⟩\\langle\\alpha,\\Omega,\\dots\\rangle\) from the Experience Graph\. It then explores the associated subgraphs to retrieve relations and scholarly judgments that best match the retrieval intent, such as prior evaluations of learning from incorrect answers or self\-correction mechanisms\. The retriever ultimately returns the most relevant experience triples to support downstream idea evaluation\.

### 3\.3Idea Evaluation Process

Equipped with the Experience Graph as external memory and the Experience Retriever as a structured retrieval operator, we develop an agentic evaluation framework that dynamically grounds idea assessment in retrieved scholarly experience\. As illustrated in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), when evaluating the novelty of the translation hallucination idea, the agent retrieves experiences related to learning from incorrect answers and self\-correction prompting, and synthesizes them into a grounded evaluative comment\.

Given a research ideaIIand the dimension\-wise prompt, DeepInstructor performs agentic retrieval and reasoning over the Experience Graph to construct dimension\-specific comments\. Conditioned on the comments, the scorer model produces a scalar scoresds\_\{d\}for each evaluation dimensiond∈𝒟d\\in\\mathcal\{D\}, reflecting its judgment grounded in accumulated reviewer experience\.

#### 3\.3\.1Agentic Retrieval and Reasoning

To effectively leverage the Experience Retriever, we instantiate DeepInstructor as a ReAct\-style agent\([21](https://arxiv.org/html/2609.22104#bib.bib18)\)that interleaves reasoning with tool use\. Instead of following a fixed retrieval pipeline, the agent autonomously decides when to retrieve, which knowledge entities to query, and which evaluative relations are relevant to the current judgment\.

For each evaluation dimension, the agent iteratively queries the Experience Graph for key concepts, accumulating historical evidence in its working memory until sufficient for evaluation\. As illustrated in Figure[1](https://arxiv.org/html/2609.22104#S1.F1), given the idea “Generate incorrect translations and use self\-correction reasoning…”, DeepInstructor first queries the entity “self\-correction through wrong answers”, and then queries related concepts such as “machine translation”\.

The final retrieved evidence is then synthesized into a dimension\-specific comment:cd=L​L​MC​G​\(I,d,Md\),c\_\{d\}=LLM\_\{CG\}\(I,d,M\_\{d\}\),whereMd=\{t∣e∈ℰR\}M\_\{d\}=\\\{t\\mid e\\in\\mathcal\{E\}\_\{R\}\\\}denotes the set of textual evidencettassociated with the retrieved edges for dimensiondd, andL​L​MC​GLLM\_\{CG\}denotes the LLM\-based comment generation function\.

#### 3\.3\.2Score Prediction via LLM Scorer

Directly prompting an LLM to output scores often leads to poorly calibrated predictions and highly compressed score distributions\. We find an effective method to solve the issue–decouple qualitative judgment from quantitative scoring\. Instead of predicting the score directly from the idea, we first generate the textual commentcdc\_\{d\}, and then map it to a scalar score using a separate dimension\-specific, prompted with few\-shot examples scorer:sd=L​L​Ms​c​o​r​e​\(cd\),s\_\{d\}=LLM\_\{score\}\(c\_\{d\}\),whereL​L​Ms​c​o​r​eLLM\_\{score\}is an LLM scorer instantiated with few\-shot exemplars for dimensiondd\. It is proven to have good accuracy on experiment datasets in Appendix[C\.1](https://arxiv.org/html/2609.22104#A3.SS1)\.

## 4Experimental Setup

### 4\.1Datasets

We evaluate DeepInstructor on two datasets\. The first golden evaluation dataset is drawn from a high\-quality human annotations dataset\([15](https://arxiv.org/html/2609.22104#bib.bib15)\), which contains original research ideas \(authored by humans or generated by AI for the NLP domain\), human comments on each idea’s novelty, significance, and feasibility \(including implementability and expected effectiveness\), together with corresponding numeric scores \(0–10\)\. We refer to this dataset asExpertEval\. ExpertEval includes 147 ideas with the expert evaluations and scores of the 3 dimensions\.

To enable large\-scale and fine\-grained evaluation, we further constructDeepInstruct, a dataset derived from ICLR 2026 peer reviews, as they provide reliable evaluation signals\. First, we cluster papers sharing identical tasks or methods\. Based on peer\-review evidence, we then grade each paper into high, medium, or low tiers across three evaluation dimensions\(Novelty, Significance, and Feasibility\)\. Next, we pair clustered papers with contrasting grades\. Finally, we reformat all papers into a standardized research idea representation\.This design supports both single\-paper level\-based evaluation and pairwise comparison between contrasting papers\.This pipeline is driven by LLMs and validated by human experts\. The resulting dataset contains 398 ideas along with evaluation and 289 pairwise comparisons\. Full construction details are provided in Appendix[B](https://arxiv.org/html/2609.22104#A2)\.

### 4\.2Evaluation Metrics

We evaluate model performance under two settings, corresponding to the two datasets used in our experiments\.

For ExpertEval, we focus on measuring alignment between model predictions and human judgments\. Due to natural inter\-rater variance, minor absolute deviations in review scores typically indicate practical equivalence rather than disagreement\. However, standard correlation coefficients assume strict linearity and suffer from range restriction, making them ill\-suited for the ordinal and clustered nature of human grading\. By analyzing the score distribution features of ICLR 2026 reviews, we conclude thatHit@1andHit@2are superior metrics compared to the Spearman correlation\. We detail this proof in Appendix[F](https://arxiv.org/html/2609.22104#A6)\.

For feasibility evaluation, scalar scores alone cannot capture whether a model identifies the same implementation risks as human experts\. We therefore additionally reportConcern\-level coverageandprecision, which measure whether generated feasibility concerns match human\-annotated concerns\. The detailed matching protocol is provided in Appendix[D](https://arxiv.org/html/2609.22104#A4)\.

For DeepInstruct, we evaluate whether models can distinguish ideas of different quality under controlled comparisons\. We reportPairwise Accuracy, which measures whether the model correctly ranks two ideas with different quality levels, andLevel Accuracy, which measures whether the model assigns each idea to the correct quality tier\. Following our dataset labels, scores of 1–4 are classified as low, 5–6 as medium, and 7–9 as high\.

### 4\.3Compared Methods

We compare our method against a diverse set of baselines, including a prior paper evaluation model, an RAG approach, and strong commercial LLMs:

- •DeepReview: A specialized model fine\-tuned on a large\-scale corpus of ICLR peer reviews\([25](https://arxiv.org/html/2609.22104#bib.bib12)\)\. Although it is designed for full\-paper assessment, it can be adapted for idea evaluation by providing only a problem formulation and a methodological sketch as input\.
- •RAG: A baseline that replaces our structured graph\-based retrieval with passage\-level embedding retrieval\. A similar implementation is used in ScholarEval\([10](https://arxiv.org/html/2609.22104#bib.bib10)\)\.
- •Commercial LLMs: Prior work has shown that commercial LLMs can serve as strong baselines for scientific idea evaluation\([10](https://arxiv.org/html/2609.22104#bib.bib10);[16](https://arxiv.org/html/2609.22104#bib.bib19);[14](https://arxiv.org/html/2609.22104#bib.bib6)\)\. We include three representative commercial LLMs:ChatGPT\-4o\-mini,ChatGPT\-4o, andDeepSeek\-V4\-Pro, which differ in reasoning capability and parametric knowledge coverage\. In addition, these models are also used as the backbone models of DeepInstructor, enabling controlled comparisons between standalone LLM evaluation and our experience\-grounded framework under the same underlying model\.

### 4\.4Implementation Details

For DeepInstructor, we use ChatGPT\-4o\-mini for graph construction and adopt multiple backbone models, including ChatGPT\-4o\-mini, ChatGPT\-4o, and DeepSeek\-V4\-Pro, for agentic reasoning \(L​L​MC​GLLM\_\{CG\}\), and score generation \(L​L​Ms​c​o​r​eLLM\_\{score\}\)\. The Experience Graph is constructed from peer reviews of ICLR 2024 and 2025\. For the Experience Retriever, we encode both entities and relations using BGE\-M3\([2](https://arxiv.org/html/2609.22104#bib.bib20)\), with retrieval sizesKc=5K\_\{c\}=5andRc=5R\_\{c\}=5\. For idea evaluation, scores are normalized to a 10\-point scale\. For RAG, we use ChatGPT\-4o as the backbone model and retrieve the top\-10 most relevant review sentences using the same embedding model \(BGE\-M3\) for fair comparison and directly condition the LLM on the retrieved text, without structured retrieval or agentic reasoning\. The temperature of all LLMs is set to 0\.1\. Further details are provided in Appendix[G](https://arxiv.org/html/2609.22104#A7)\.

## 5Results

For experimental results, we first present the main evaluation results on two datasets, followed by an ablation study on different graph sizes and an AI\-in\-the\-loop experiment to see whether the proposed framework can enhance the idea generator with comments from our proposed model\.

### 5\.1Main Results

#### 5\.1\.1Alignment Experiment on ExpertEval

Table 1:Performance comparison across Novelty, Significance, and Feasibility evaluation\. We report Hit@±1\\pm 1and Hit@±2\\pm 2for scalar scores, and mean coverage and precision for feasibility concerns\.O​u​r​sOursindicates the proposed DeepInstructor Model\. Best and second\-best results are highlighted inboldandunderlined, respectively\.ModelNoveltySignificanceFeasibility ScoreFeasibility ConcernsHit@±1\\pm 1Hit@±2\\pm 2Hit@±1\\pm 1Hit@±2\\pm 2Hit@±1\\pm 1Hit@±2\\pm 2Mean Cov\.Mean Prec\.R​A​GRAG0\.4520\.6810\.0000\.1330\.6810\.9330\.5790\.310DeepReview0\.5040\.7410\.0440\.2070\.6670\.8950\.0070\.000ChatGPT\-4o0\.0890\.2150\.0000\.1330\.6520\.9260\.5750\.305ChatGPT\-4o\-mini0\.0440\.1410\.0000\.1410\.7190\.9700\.4400\.246DeepSeek\-V4\-Pro0\.3780\.5480\.0440\.2000\.3850\.6820\.5220\.391O​u​r​s4​o−m​i​n​iOurs\_\{4o\-mini\}0\.5150\.7460\.0300\.2070\.7480\.9780\.4960\.261O​u​r​s4​oOurs\_\{4o\}0\.6220\.8670\.1260\.3560\.7260\.9410\.6400\.362O​u​r​sD​e​e​p​S​e​e​kOurs\_\{DeepSeek\}0\.6490\.8630\.2070\.4220\.5850\.8960\.5340\.355

Table[1](https://arxiv.org/html/2609.22104#S5.T1)summarizes the evaluation results across all dimensions\. We highlight several key observations:

\(1\)DeepInstructor consistently outperforms standalone LLM evaluators\.When using the same backbone model, DeepInstructor achieves substantial improvements over the corresponding standalone commercial models\. For example, compared to ChatGPT\-4o, DeepInstructor with ChatGPT\-4o as backbone improves the average Hit@1 and Hit@2 by24\.4%and29\.7%, respectively\. Notably, all standalone commercial LLMs achieve extremely low performance on Significance evaluation\. Manual inspection suggests that these models tend to assign overly positive significance judgments to most ideas, making it difficult to distinguish genuinely impactful problems from marginal or incremental ones\(see Appendix[E\.2](https://arxiv.org/html/2609.22104#A5.SS2.SSS0.Px2)for a representative example\)\. In contrast, grounding evaluation in peer\-review experience enables DeepInstructor to better identify low\-significance ideas, leading to substantially improved performance on this challenging dimension\.

\(2\)Compared to RAG, DeepInstructor provides more effective evaluation capabilities\.While RAG achieves moderate performance on Novelty, it performs extremely poorly on Significance, with near\-zero Hit rates\. This suggests that conventional retrieval fails to capture the evaluative knowledge required for significance assessment\. In contrast, DeepInstructor substantially improves performance on all three dimensions, highlighting the importance of structured experience modeling and relation\-aware retrieval for evaluative reasoning \(see Appendix[E\.2](https://arxiv.org/html/2609.22104#A5.SS2)for a representative example\)\.

\(3\)The framework generalizes across backbone models and reduces reliance on model scale\.Even with a weaker backbone \(ChatGPT\-4o\-mini\), DeepInstructor outperforms stronger baselines such as DeepReview and DeepSeek\-V4\-Pro on multiple dimensions\. Interestingly, stronger general\-purpose reasoning capability does not always translate to better idea evaluation performance\. For example, DeepSeek\-V4\-Pro underperforms ChatGPT\-4o and ChatGPT\-4o\-mini on feasibility evaluation despite being a stronger reasoning model overall, suggesting that research idea evaluation exhibits model\-specific behavioral differences beyond raw model capability\. Nevertheless, pairing DeepSeek\-V4\-Pro with DeepInstructor still leads to substantial improvements, further demonstrating the robustness of the proposed framework across heterogeneous backbones\.

To better explain these performance differences, we provide detailed case studies on novelty and significance evaluation, as well as a walkthrough example illustrating how the DeepInstructor agent performs step\-by\-step reasoning in Appendix[E\.2](https://arxiv.org/html/2609.22104#A5.SS2)\. The case studies show that DeepInstructor can retrieve both positive and skeptical reviewer judgments from historical peer reviews, enabling more calibrated and experience\-grounded evaluation compared to standalone LLMs and RAG systems\.

#### 5\.1\.2Alignment Experiment on DeepInstruct

Table 2:Level Accuracy and Pairwise Comparison Accuracy on DeepInstruct\. Best results are inbold, and second\-best areunderlined\. 4o\-mini = ChatGPT\-4o\-mini; DS = DeepSeek\-V4\-ProModelNovelty \(%\)Pair\. / Lev\.Feasibility \(%\)Pair\. / Lev\.Significance \(%\)Pair\. / Lev\.RAG46\.15/ 36\.3224\.41 / 48\.4313\.33 / 20\.004o\-mini41\.03 / 23\.5022\.83 / 45\.6723\.33 / 25\.00DS33\.33 / 40\.3142\.64/ 15\.5030\.00 /48\.33Ours4​o​\-mini\\text\{Ours\}\_\{4o\\text\{\-mini\}\}47\.86/43\.1625\.20 /52\.3636\.67/ 26\.67Δ\\Delta\+7%/\+20%\+2%/\+7%\+13%/\+2%OursDS\\text\{Ours\}\_\{\\text\{DS\}\}37\.34 /42\.5839\.46/50\.3940\.00/49\.11Δ\\Delta\+4%/\+2%\-3%/\+35%\+10%/\+1%

Table[2](https://arxiv.org/html/2609.22104#S5.T2)presents the accuracy of classifying ideas into the correct levels, as well as pairwise comparison accuracy based on these levels\. The results provide two additional insights that complement the findings on ExpertEval:

\(1\)Robust performance under alternative evaluation protocols\.DeepInstructor consistently improves Level accuracy while remaining competitive on Pairwise accuracy across different evaluation dimensions, demonstrating stronger calibration of absolute judgment\. Compared to conventional RAG, DeepInstructor achieves substantially better Level accuracy, suggesting that passage\-level retrieval alone is insufficient for reliable idea evaluation\. Instead, grounding evaluation in structured scholarly experience enables more calibrated and reliable judgments\.

\(2\)Structured experience improves score calibration\.Although stronger commercial LLMs with more recent knowledge \(e\.g\., DeepSeek\-V4\-Pro\) achieve competitive Pairwise accuracy on certain dimensions, they still struggle with calibrated absolute scoring, leading to poor Level accuracy\. DeepInstructor mitigates this issue by grounding judgments in structured scholarly experience, resulting in substantially improved calibration \(e\.g\., \+35% Level accuracy on feasibility\), even with a minor trade\-off in Pairwise performance\.

Overall, these results suggest that the gains of DeepInstructor are not solely explained by backbone model scale or newer parametric knowledge, but are closely related to the proposed experience\-grounded retrieval and reasoning framework\. The consistent improvements observed across both evaluation datasets further support the robustness of the proposed paradigm\.

### 5\.2Analysis of Experience Coverage

To investigate the impact of experience coverage on model performance, we construct Experience Graphs using data from different years \(2024 vs\. 2025\) and compare them with the full\-coverage setting \(both 2024 and 2025\)\.

Figure 3:Ablation study on the effect of experience coverage using different subsets of the Experience Graph\.O​u​r​sOursdenotes the full\-coverage setting, whileO​u​r​s2024Ours\_\{2024\}andO​u​r​s2025Ours\_\{2025\}use year\-specific data only\.As shown in Figure[3](https://arxiv.org/html/2609.22104#S5.F3), reducing the temporal coverage of experience leads to consistent performance degradation across both score\-based and structure\-aware metrics\. In particular, using only 2025 data results in noticeable drops in Novelty alignment \(Hit@±1\\pm 1: 0\.444 vs\. 0\.496\) and feasibility reasoning quality \(coverage: 0\.4092 vs\. 0\.4955\), suggesting that limited experience fails to provide sufficient evaluative evidence\. Interestingly, the 2024\-only graph performs more competitively than the 2025\-only graph on several metrics, indicating that performance is not solely determined by recency but also by the diversity and completeness of accumulated experience\. This highlights that effective idea evaluation requires broad and well\-distributed historical knowledge, rather than narrowly scoped or temporally localized evidence\. Overall, these results demonstrate that the effectiveness of DeepInstructor is closely tied to the coverage and richness of the Experience Graph, further supporting our design choice of leveraging large\-scale, aggregated scholarly experience for robust evaluation\.

### 5\.3AI\-in\-the\-loop Experiment

To study the role of evaluative feedback in iterative idea refinement, we design an AI\-in\-the\-loop experiment to simulate a closed\-loop research iteration process\. Starting from an initial ideaR0R\_\{0\}, the idea is iteratively refined over 3 rounds\. At each iteration, the generator updates the current idea based on evaluation comments produced for the previous version\. Specifically, we use the dimension\-specific commentscd=L​L​MC​G​\(I,d,Md\)c\_\{d\}=LLM\_\{CG\}\(I,d,M\_\{d\}\)generated by DeepInstructor as structured evaluative feedback, and compare this setting against a generic prompt\-based feedback baseline \(“Please improve your idea”\)\. To examine the effects of both feedback quality and generator capability, we conduct experiments using ChatGPT\-4o and ChatGPT\-4o\-mini as generators\. All experiments are conducted on the ExpertEval dataset\.

Table 3:Average overall feedback score improvements \(Δ¯\\bar\{\\Delta\}\) across different methods\.MethodΔ¯\\bar\{\\Delta\}ChatGPT\-4o\-mini prompt feedback\+0\.068ChatGPT\-4o\-mini Instructor feedback\+0\.121ChatGPT\-4o Instructor feedback\+0\.203

Results in Table[3](https://arxiv.org/html/2609.22104#S5.T3)show that DeepInstructor consistently yields larger improvements than generic feedback, indicating that structured, experience\-grounded critiques provide more effective guidance\. Second, while stronger models \(ChatGPT\-4o\) can better utilize high\-quality feedback, weaker ones \(ChatGPT\-4o\-mini\) benefit less\. This reveals a key interaction—high\-quality feedback alone is insufficient without a capable generator to leverage it\.

## 6Conclusion

We presentDeepInstructor, an experience\-driven, agentic framework for evaluating research ideas that grounds judgments in structured scholarly experience\. Experiments show that DeepInstructor achieves strong alignment with human evaluations and consistently outperforms existing baselines\. More broadly, our work suggests that scholarly experience can be externalized and reused as a structured resource for AI reasoning\. In future work, we plan to extend the experience source beyond peer reviews to broader scientific resources, such as published papers and scientific editorials\.

## 7Limitations

Although DeepInstructor substantially improves significance evaluation by grounding judgments in reviewer experience, the overall performance remains limited, suggesting that significance assessment is still an open challenge for experience\-grounded idea evaluation\. We leave improving significance\-oriented retrieval and reasoning as important future work\.

In addition, the current Experience Graph is constructed from only two years of open peer reviews in computer science, which limits both the coverage and diversity of evaluative experience\. Extending DeepInstructor to other scientific domains may require incorporating additional experience sources beyond open reviews, since publicly available peer\-review data in many fields remains limited\.

## References

- Baeket al\.\(2025\)J\. Baek, S\. K\. Jauhar, S\. Cucerzan, and S\. J\. HwangResearchAgent: iterative research idea generation over scientific literature with large language models\.External Links:2404\.07738,[Link](https://arxiv.org/abs/2404.07738)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p2.1),[§2](https://arxiv.org/html/2609.22104#S2.p1.1)\.
- Chenet al\.\(2025\)J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. LiuM3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.External Links:2402\.03216,[Link](https://arxiv.org/abs/2402.03216)Cited by:[§4\.4](https://arxiv.org/html/2609.22104#S4.SS4.p1.1)\.
- Fenget al\.\(2025\)T\. Feng, Y\. Sun, and J\. YouGraphEval: a lightweight graph\-based llm framework for idea evaluation\.External Links:2503\.12600,[Link](https://arxiv.org/abs/2503.12600)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p2.1),[§2](https://arxiv.org/html/2609.22104#S2.p1.1)\.
- Hanet al\.\(2025\)H\. Han, Y\. Wang, H\. Shomer, K\. Guo, J\. Ding, Y\. Lei, M\. Halappanavar, R\. A\. Rossi, S\. Mukherjee, X\. Tang, Q\. He, Z\. Hua, B\. Long, T\. Zhao, N\. Shah, A\. Javari, Y\. Xia, and J\. TangRetrieval\-augmented generation with graphs \(graphrag\)\.External Links:2501\.00309,[Link](https://arxiv.org/abs/2501.00309)Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p2.1)\.
- Huet al\.\(2025\)Y\. Hu, Z\. Lei, Z\. Zhang, B\. Pan, C\. Ling, and L\. ZhaoGRAG: graph retrieval\-augmented generation\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4145–4157\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.232/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.232),ISBN 979\-8\-89176\-195\-7Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p2.1)\.
- Karguptaet al\.\(2026\)P\. Kargupta, S\. Mehri, D\. Hakkani\-Tur, and J\. HanSparking scientific creativity via llm\-driven interdisciplinary inspiration\.External Links:2603\.12226,[Link](https://arxiv.org/abs/2603.12226)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Luet al\.\(2024\)C\. Lu, C\. Lu, R\. T\. Lange, J\. Foerster, J\. Clune, and D\. HaThe ai scientist: towards fully automated open\-ended scientific discovery\.External Links:2408\.06292,[Link](https://arxiv.org/abs/2408.06292)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Luet al\.\(2026\)C\. Lu, C\. Lu, R\. T\. Lange, Y\. Yamada, S\. Hu, J\. Foerster, D\. Ha, and J\. CluneTowards end\-to\-end automation of ai research\.Nature651\(8107\),pp\. 914–919\.Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- McCusker \(2023\)J\. McCuskerLOKE: linked open knowledge extraction for automated knowledge graph construction\.External Links:2311\.09366,[Link](https://arxiv.org/abs/2311.09366)Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p2.1)\.
- Moussaet al\.\(2025\)H\. N\. Moussa, P\. Q\. D\. Silva, D\. Adu\-Ampratwum, A\. East, Z\. Lu, N\. Puccetti, M\. Xue, H\. Sun, B\. P\. Majumder, and S\. KumarScholarEval: research idea evaluation grounded in literature\.External Links:2510\.16234,[Link](https://arxiv.org/abs/2510.16234)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1),[§1](https://arxiv.org/html/2609.22104#S1.p2.1),[§2](https://arxiv.org/html/2609.22104#S2.p1.1),[2nd item](https://arxiv.org/html/2609.22104#S4.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2609.22104#S4.I1.i3.p1.1)\.
- Paiet al\.\(2024\)L\. Pai, W\. Gao, W\. Dong, L\. Ai, Z\. Gong, S\. Huang, L\. Zongsheng, E\. Hoque, J\. Hirschberg, and Y\. ZhangA survey on open information extraction from rule\-based model to large language model\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9586–9608\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.560/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.560)Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p2.1)\.
- Qiuet al\.\(2025\)Y\. Qiu, H\. Zhang, Z\. Xu, M\. Li, D\. Song, Z\. Wang, and K\. ZhangAi idea bench 2025: ai research idea generation benchmark\.arXiv preprint arXiv:2504\.14191\.Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Sauteret al\.\(2026\)A\. Sauter, Y\. Zhao, J\. Urbani, W\. Hu, Z\. Meng, L\. Zhou, X\. Yan, and Y\. LyuEvoIdeator: evolving scientific ideas through checklist\-grounded reinforcement learning\.External Links:2603\.21728,[Link](https://arxiv.org/abs/2603.21728)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Siet al\.\(2025\)C\. Si, T\. Hashimoto, and D\. YangThe ideation\-execution gap: execution outcomes of llm\-generated versus human research ideas\.ArXivabs/2506\.20803\.External Links:[Link](https://api.semanticscholar.org/CorpusID:280011353)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1),[§1](https://arxiv.org/html/2609.22104#S1.p2.1),[3rd item](https://arxiv.org/html/2609.22104#S4.I1.i3.p1.1)\.
- Siet al\.\(2024\)C\. Si, D\. Yang, and T\. HashimotoCan llms generate novel research ideas? a large\-scale human study with 100\+ nlp researchers\.External Links:2409\.04109,[Link](https://arxiv.org/abs/2409.04109)Cited by:[Appendix B](https://arxiv.org/html/2609.22104#A2.p1.1),[§2](https://arxiv.org/html/2609.22104#S2.p1.1),[§4\.1](https://arxiv.org/html/2609.22104#S4.SS1.p1.1)\.
- Tonget al\.\(2026\)J\. Tong, M\. Li, H\. Li, Y\. Yang, Y\. Mou, W\. Ma, Z\. Xi, H\. Chen, X\. Liu, Q\. Cheng, M\. Zhang, Q\. Chen, W\. Ge, Q\. Guo, T\. Ying, T\. Sun, Y\. Zheng, X\. Chen, J\. Zhao, N\. Ding, X\. Huang, Y\. Jiang, and X\. QiuAI can learn scientific taste\.External Links:2603\.14473,[Link](https://arxiv.org/abs/2603.14473)Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p1.1),[3rd item](https://arxiv.org/html/2609.22104#S4.I1.i3.p1.1)\.
- Wanget al\.\(2024\)Q\. Wang, D\. Downey, H\. Ji, and T\. HopeSciMON: scientific inspiration machines optimized for novelty\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 279–299\.External Links:[Link](http://dx.doi.org/10.18653/v1/2024.acl-long.18),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.18)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Wenet al\.\(2025\)J\. Wen, C\. Si, Y\. Chen, H\. He, and S\. FengPredicting empirical ai research outcomes with language models\.ArXivabs/2506\.00794\.External Links:[Link](https://api.semanticscholar.org/CorpusID:279074957)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p2.1)\.
- Wenget al\.\(2025\)Y\. Weng, M\. Zhu, Q\. Xie, Q\. Sun, Z\. Lin, S\. Liu, and Y\. ZhangDeepScientist: advancing frontier\-pushing scientific findings progressively\.External Links:2509\.26603,[Link](https://arxiv.org/abs/2509.26603)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Xieet al\.\(2025\)Q\. Xie, Y\. Weng, M\. Zhu, F\. Shen, S\. Huang, Z\. Lin, J\. Zhou, Z\. Mao, Z\. Yang, L\. Yang,et al\.How far are ai scientists from changing the world?\.arXiv preprint arXiv:2507\.23276\.Cited by:[Appendix B](https://arxiv.org/html/2609.22104#A2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. CaoReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p5.1),[§3\.3\.1](https://arxiv.org/html/2609.22104#S3.SS3.SSS1.p1.1)\.
- Zhang and Soh \(2024\)B\. Zhang and H\. SohExtract, define, canonicalize: an llm\-based framework for knowledge graph construction\.External Links:2404\.03868,[Link](https://arxiv.org/abs/2404.03868)Cited by:[§2](https://arxiv.org/html/2609.22104#S2.p2.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, S\. A\. Khan, A\. Mahmud, H\. Yang, A\. Lavin, M\. Levin, J\. Frey, J\. Dunnmon, J\. Evans, A\. Bundy,et al\.Exploring the role of large language models in the scientific method: from hypothesis to discovery\.npj Artificial Intelligence1\(1\),pp\. 14\.Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Zhenget al\.\(2025\)T\. Zheng, Z\. Deng, H\. T\. Tsang, W\. Wang, J\. Bai, Z\. Wang, and Y\. SongFrom automation to autonomy: a survey on large language models in scientific discovery\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 17744–17761\.Cited by:[§1](https://arxiv.org/html/2609.22104#S1.p1.1)\.
- Zhuet al\.\(2025\)M\. Zhu, Y\. Weng, L\. Yang, and Y\. ZhangDeepReview: improving llm\-based paper review with human\-like deep thinking process\.External Links:2503\.08569,[Link](https://arxiv.org/abs/2503.08569)Cited by:[Appendix B](https://arxiv.org/html/2609.22104#A2.p1.1),[§2](https://arxiv.org/html/2609.22104#S2.p1.1),[1st item](https://arxiv.org/html/2609.22104#S4.I1.i1.p1.1)\.

## Appendix Catalog

## Appendix APilot Interview Study

To understand how human instructors evaluate research ideas and what forms of scholarly experience they rely on, we conducted a semi\-structured pilot interview study with 15 Computer Science faculty members across machine learning, natural language processing, systems, and human–computer interaction\. All participants had at least five years of advising experience and had supervised multiple student research projects\.

Interviews were conducted remotely and lasted 20–30 minutes\. We adopted a semi\-structured protocol consisting of four themes: \(1\) criteria instructors consider when evaluating student ideas, \(2\) examples of strong and weak ideas, \(3\) the types of prior scholarly experience drawn upon during evaluation, and \(4\) the capabilities expected of an AI evaluator\. The protocol encourages both structured reflection and open\-ended elaboration, allowing us to capture both high\-level evaluation criteria and rich, stance\-bearing judgments derived from instructors’ accumulated research experience\.

All interviews were audio\-recorded, transcribed, and analyzed using open coding by two independent annotators\. Through iterative thematic analysis, we identified recurring evaluation criteria and extracted representative experience\-based statements, which subsequently informed the design of the Experience Graph and the dimension\-specific evaluation strategies used in DeepInstructor\.

### A\.1Pilot Interview Protocol

We list below the full interview protocol used during the study\.

Theme A: Evaluation Criteria

1. Q1\.Criteria for Evaluating Research Ideas When evaluating a new research idea from a student, which criteria do you typically consider? Please describe all aspects you examine when forming an initial judgment\. \(You may include considerations such as novelty, significance, feasibility, technical soundness, etc\., but please feel free to answer in your own words\.\)

Theme B: Examples of Strong and Weak Ideas

1. Q2\.Example of a Strong Idea Could you recall a strong research idea a student proposed recently? What made it strong?
2. Q3\.Example of a Weak Idea Could you recall a weak idea you have encountered? What issues made it weak?

Theme C: Use of Prior Scholarly Experience

1. Q4\.Experience Used During Evaluation \(a\) When assessing a new idea, what past knowledge or experiences do you typically rely on? \(b\) Could you give one or two concrete examples of such intuitive or experience\-based judgments? \(For instance, empirical patterns such as “Method X often fails under condition Y\.”\)

Theme D: Designing an AI Evaluator

1. Q5\.Capabilities Desired in an AI Evaluator If you were to design an AI system that evaluates research ideas, what capabilities should it have in order to resemble human reasoning?

### A\.2Findings from the Pilot Study

The interviews reveal two central findings that directly inform the design of DeepInstructor\.

##### Key Evaluation Criteria\.

Across participants, three criteria consistently emerged as the primary factors instructors use when evaluating student ideas:Novelty,Significance, andFeasibility\. These criteria were mentioned spontaneously in response to Q1, and were further reinforced through examples in Q2 and Q3, where strong ideas were typically described as addressing a meaningful problem with a novel perspective, while weak ideas commonly failed due to feasibility issues such as unrealistic assumptions, unclear execution paths, or known limitations of the proposed techniques\. This triangulation establishes these three criteria as the core dimensions that a human\-aligned evaluator should prioritize\.

##### Reliance on Scholarly Experience\.

Interviews also revealed that instructors draw heavily on accumulated scholarly experience when forming judgments\. Through Q4, faculty frequently described relying on empirical heuristics, known failure modes of algorithms, benchmark\-specific intuitions, and domain\-specific expectations\. These descriptions can be naturally expressed as stance\-bearing statements such as “Method X usually fails when data are sparse” or “Approach Y tends to work well only when Z holds\.” These stance\-like judgments provide structured, reusable forms of research experience, motivating our conceptualization of academic experience as a graph of stance\-bearing statements\.

##### Desired Capabilities of an AI Evaluator\.

In Q5, participants emphasized that an AI evaluator should be able to: \(i\) recognize novelty within the broader literature, \(ii\) identify feasibility risks based on method limitations and empirical regularities, and \(iii\) reason using past scholarly experience rather than relying solely on parametric LLM knowledge\. These observations directly inform the design of DeepInstructor, which evaluates ideas using dimension\-specific strategies grounded in experience retrieval\.

Overall, the pilot study demonstrates that human instructors rely on a small set of core criteria and rich, stance\-bearing scholarly experience when evaluating research ideas\. These findings motivate the Experience Graph and the experience\-guided evaluation paradigm central to DeepInstructor\.

## Appendix BDeepInstruct Dataset Construction Details

To date, the evaluation of early\-stage research ideas remains constrained by a lack of dedicated datasets\([20](https://arxiv.org/html/2609.22104#bib.bib25)\)\. Directly soliciting feedback from domain experts is highly resource\-intensive\([15](https://arxiv.org/html/2609.22104#bib.bib15)\), and present peer review data collection methods will expose experiment results, rendering them unsuitable for assessing\([25](https://arxiv.org/html/2609.22104#bib.bib12)\)\. To address this gap, we introduce DeepInstruct, a novel dataset comprising early\-stage research ideas coupled with multi\-dimensional evaluation scores\. Extracted from ICLR 2026 peer reviews, DeepInstruct demonstrates strong alignment with human expert judgments\. The construction pipeline is shown in Figure[4](https://arxiv.org/html/2609.22104#A2.F4)\.

![Refer to caption](https://arxiv.org/html/2609.22104v1/dataset.png)Figure 4:Pipeline of DeepInstruct Dataset Construction\.### B\.1Extracting and Indexing Paper Tasks and Methods

Our dataset construction began with the collection of peer reviews from 19,398 ICLR papers after removing submissions lacking review data\. This provides an abundant source for evaluation\. The massive scale of the data enables us to trade recall for higher precision\. Then we employed GPT\-4o\-mini to systematically extract the core task and proposed methodology of each paper\. Recognizing the nuanced distinctions among various research tasks, we designed our prompts to elicit comprehensive problem descriptions that integrate both contextual background and specific objectives\. To maintain clear conceptual boundaries, the model was explicitly constrained from generating overlapping content between the task and method sections\. Subsequently, considering the brevity of the task and method descriptions, we leveraged text\-embedding\-v4 to obtain dense representations of the titles and methods\. This embedding step streamlines the downstream organization, allowing us to effectively cluster topically and methodologically similar papers into a candidate pool easily\.

### B\.2Organizing Papers with the Same Task and Method

Embedding Similarity Search\.Following the previous stage, our initial set of papers𝐏=\{Pi\}i=1N\\mathbf\{P\}=\\\{P\_\{i\}\\\}\_\{i=1\}^\{N\}is parsed into a corresponding set of task\-method pairs𝐓𝐌=\{\(Ti,Mi\)\}i=1N\\mathbf\{TM\}=\\\{\(T\_\{i\},M\_\{i\}\)\\\}\_\{i=1\}^\{N\}, whereN=19,398N=19,398\. To comprehensively group papers sharing either identical tasks or identical methods, we conduct two parallel retrieval processes\.

First, we randomly sample an index subset𝒬T⊂\{1,2,…,N\}\\mathcal\{Q\}^\{T\}\\subset\\\{1,2,\\dots,N\\\}with a size of\|𝒬T\|=1,000\|\\mathcal\{Q\}^\{T\}\|=1,000to serve astask queries\. For each query indexq∈𝒬Tq\\in\\mathcal\{Q\}^\{T\}, we utilize its task embedding vector, denoted as𝐯qT\\mathbf\{v\}\_\{q\}^\{T\}, to perform semantic matching against all tasks in𝐓𝐌\\mathbf\{TM\}, retrieving the index setℛqT\\mathcal\{R\}\_\{q\}^\{T\}of the top\-10 most similar papers \(\|ℛqT\|=10\|\\mathcal\{R\}\_\{q\}^\{T\}\|=10\)\.

Similarly, we randomly sample another index subset𝒬M⊂\{1,2,…,N\}\\mathcal\{Q\}^\{M\}\\subset\\\{1,2,\\dots,N\\\}of size\|𝒬M\|=1,000\|\\mathcal\{Q\}^\{M\}\|=1,000asmethod queries\. Using the method embedding𝐯qM\\mathbf\{v\}\_\{q\}^\{M\}for eachq∈𝒬Mq\\in\\mathcal\{Q\}^\{M\}, we retrieve the top\-10 index setℛqM\\mathcal\{R\}\_\{q\}^\{M\}by semantically matching against all methods\. This top\-10 threshold is chosen empirically; our observations indicate that papers ranked beyond this cutoff rarely share the exact same task or method\.

Consequently, for the task\-driven retrieval, we form candidate clustersCqT=\{\(Tk,Mk\)∣k∈ℛqT\}C\_\{q\}^\{T\}=\\\{\(T\_\{k\},M\_\{k\}\)\\mid k\\in\\mathcal\{R\}\_\{q\}^\{T\}\\\}\. For the method\-driven retrieval, we form clustersCqM=\{\(Tk,Mk\)∣k∈ℛqM\}C\_\{q\}^\{M\}=\\\{\(T\_\{k\},M\_\{k\}\)\\mid k\\in\\mathcal\{R\}\_\{q\}^\{M\}\\\}\. The final output of this stage comprises two structured collections of candidate pools: the task\-centric clusters𝐂T=\{CqT∣q∈𝒬T\}\\mathbf\{C\}^\{T\}=\\\{C\_\{q\}^\{T\}\\mid q\\in\\mathcal\{Q\}^\{T\}\\\}and the method\-centric clusters𝐂M=\{CqM∣q∈𝒬M\}\\mathbf\{C\}^\{M\}=\\\{C\_\{q\}^\{M\}\\mid q\\in\\mathcal\{Q\}^\{M\}\\\}\.

Selected LLMs as a Filter\.Following the embedding\-based clustering, we introduce Large Language Models as an advanced filter to rigorously filter out papers that share the exact same tasks or methods\. Taking task filtering as an example, we propose an evaluation pipeline comprising human annotation probing and LLM combination for alignment\. First, domain experts are instructed to annotate a subset of 150 candidate clusters, manually selecting the papers that share the same task as the query\. Subsequently, we evaluate the consistency between the LLMs’ predictions and the human annotations\. We employed multiple LLMs and experimented with various combination strategies using set intersection and union\. Ultimately, we adopted the LLM combination that achieved the highest precision to conduct the final filtering\. The alignment results are presented in Table[4](https://arxiv.org/html/2609.22104#A2.T4)\.

Table 4:Performance of Different Model Combinations \(Top 6 by Precision\)CombinationTypePrecisionRecallF1gpt4\_turbo \+ gpt4o \+ qwen\_plusintersection0\.78240\.37880\.3948gemini \+ gpt4\_turbo \+ qwen\_plusintersection0\.77440\.38090\.3913gpt4\_turbo \+ qwen\_plusintersection0\.76340\.40000\.4140gpt4\_turbo \+ gpt4o\_mini \+ qwen\_plusintersection0\.76310\.36750\.3819gemini \+ gpt4\_turbo \+ gpt4ointersection0\.75540\.36540\.3696gpt4\_turbo \+ gpt4ointersection0\.72970\.38960\.3981

### B\.3LLM Grading

For each filtered cluster sharing an identical task, we employed gpt\-4o\-mini to synthesize the corresponding peer reviews and assign categorical grades \(High, Medium, or Low\) across two dimensions: novelty and feasibility\. Conversely, for clusters sharing an identical method, the model was tasked with evaluating the significance dimension\. This design strictly adheres to our evaluation taxonomy, which posits that significance inherently reflects the value of the target task, whereas novelty and feasibility assess the design of the proposed methodology\. To ensure the reliability and objectivity of the grading process, we utilized a Chain\-of\-Thought \(CoT\) prompting strategy\. The LLM was explicitly instructed to articulate its reasoning based solely on the provided expert reviews before outputting a final grade, thereby preventing any subjective self\-judgment or injection of external bias\. They were prompted refuse to answer if there is not any information mentioned in the reviews\. Furthermore, the models were explicitly prompted to decline evaluation if the required information was absent from the provided reviews\. Finally, within each cluster and across each evaluation dimension, we construct pairwise comparisons between papers that received distinct grades\.

### B\.4Human Evaluating

After completing the above steps, human experts verified each piece of data one by one\. The human experts were required to check whether the tasks or methods were consistent and whether they agreed with the level of automatic annotation by AI\. Finally, we convert the paper into the idea format, that is, to hide the experimental results and focus on describing the problem, method, and experimental design\. The DeepInstruct dataset comprises 398 ideas, containing a total of 289 pairwise comparison records\. Broken down by evaluation dimension, feasibility includes 130 pairs involving 212 ideas; novelty consists of 129 pairs involving 206 ideas; and significance accounts for 30 pairs involving 56 ideas\.

## Appendix CMethodology Details

### C\.1Performance of LLM scorer

Table 5:Comparison of Novelty score distributions\. Scores from 1 to 5 are omitted \(counts are zero\)\.MethodScore \(%\)678910Direct Scoring0\.00\.076\.323\.70\.0Evaluation\+Scorer1\.518\.53\.043\.034\.1The proposed LLM Scorer demonstrated strong predictive capability when estimating actual reviewer scores based solely on textual reviews\. The scorer can accurately gauge the degree of polarity in the evaluation, resulting in a significantly more uniform distribution of scores \(as shown in Table[5](https://arxiv.org/html/2609.22104#A3.T5)\)\. The predicted scores achieved an average accuracy of 75\.80% within ±1\.0 point and 93\.87% within ±2\.0 points of the ground\-truth scores, as shown in Table[6](https://arxiv.org/html/2609.22104#A3.T6)\.

Table 6:Performance of LLM scorer across different evaluation dimensions\.DimensionPrecisionPearsonSpearmanKappaSignificance±1\\pm 1: 0\.7232;±2\\pm 2: 0\.93450\.82200\.84730\.7760Novelty±1\\pm 1: 0\.6528;±2\\pm 2: 0\.90210\.80860\.81280\.7551Feasibility\_score±1\\pm 1: 0\.8980;±2\\pm 2: 0\.97960\.77140\.76980\.6556
### C\.2Graph Construction Details

The resulting Experience Graph contains 416,330 evaluative triples extracted from 58,607 peer reviews, covering 567,764 unique entities and 153,391 unique relations\. On average, each review contributes 7\.55 triples\. These statistics show that the graph is not a sparse symbolic resource, but a large\-scale external memory of reviewer experience that captures diverse stance\-bearing judgments over tasks, methods, assumptions, and evaluation protocols\.

MetricScoreProblem coverage92\.8%Method coverage88\.6%Overall coverage90\.7%Macro\-average F10\.9064Table 7:Coverage of reviewer\-described ideas by the Experience Graph\.To evaluate whether the Experience Graph covers the conceptual space described in peer reviews, we compare two entity sets for each review: one extracted from the paper’s problem and method descriptions, and the other extracted from the review\-derived graph triples\. Two entities are considered matched if their BGE\-M3 embedding similarity exceeds 0\.7\. As shown in Table[7](https://arxiv.org/html/2609.22104#A3.T7), the graph achieves 92\.8% problem coverage, 88\.6% method coverage, 90\.7% overall coverage, and a macro\-average F1 of 0\.9064\. These results suggest that although the graph is constructed only from ICLR 2024 and 2025 reviews, it captures the majority of evaluative concepts appearing in reviewer\-described research ideas\.

Prompt of Triplet ExtractionYou are a professional academic evaluation experience extraction expert\. Please extract detailed and specific knowledge entities and experiential relationships from the following review text to construct an experiential relationship subgraph\.Knowledge entities include but are not limited to questions, methods, concepts, theories, scenarios, and other professional terms and knowledge\. Please do not use a simple word; it is better to enrich the semantic meaning of the entity based on the original evaluation by adding some adjectives\.Experience relations refer to, for example, “seems to be relatively good at improving… ability in xxx”, “seems unable to be well achieved through… ” and other similar evaluative relationship statements\. Their characteristics are: they have certain positive or negative emotional evaluation information, and they have as detailed as possible semantic information of specific aspects\. Nodes and edges must directly depend on the review\. The original text is only for reference\. Therefore, when each edge is constructed, there must be corresponding evidence\.Entity Naming Guidelines \- Be Specific and Contextual:\[X\]WRONG Examples \(too generic\):•“LLMs”→\\rightarrow\(It depends on the original text specifically\.\) Might be “scientific reasoning ability of current large language models”•“performance”→\\rightarrowMight be “few\-shot learning ability of current large language models”•“dataset”→\\rightarrowMight be “SCIBENCH scientific reasoning evaluation dataset”\[V\]CORRECT Examples \(specific and contextual\):•“scientific reasoning ability of current large language models”•“few\-shot learning performance on complex scientific problems”•“SCIBENCH scientific reasoning evaluation dataset”•“Chain\-of\-Thought \(CoT\) prompting technique”Relationship Examples \- Be Evaluative and Specific:\[V\]Good Relationship Examples:\(Include descriptions of both positive and negative aspects and degrees, as well as rich semantic information\)•“Chain\-of\-Thought \(CoT\) prompting”→\\rightarrow“significantly improves”→\\rightarrow“calculation skills of LLMs”•“SCIBENCH dataset”→\\rightarrow“effectively differentiates”→\\rightarrow“performance between different large language models”•“free\-form questions”→\\rightarrow“a certain degree prevents”→\\rightarrow“result guessing based on multiple\-choice answers”•“systematic zero\-shot example selection”→\\rightarrow“can not enhances”→\\rightarrow“model’s problem\-solving capabilities”•“Wolfram Language prompts”→\\rightarrow“deteriorates”→\\rightarrow“few\-shot learning performance of LLMs”\[X\]WRONG Relationship Examples \(too generic\):•“LLMs”→\\rightarrow“is designed to”→\\rightarrow“performance” \(too vague\)•“Error analysis”→\\rightarrow“requires understanding of”→\\rightarrow“Hückel molecular orbital theory” \(knowledge relation, not evaluative\)Attention1\.The evidence section must be complete and no part should be omitted\. Never use “…”\.2\.Node and Edge should be as specific and detailed as possible, and when combined, they should conform to the logic of the original text\.You should check the review sentence by sentence to see if there are any evaluations from the reviewers regarding the method or professional terms, and then extract them\. And speak out your thinking process aloud\. Your final output should be in JSON format:\{\{
"source\_name": "node\_x",
"target\_name": "node\_y",
"relationship": "xxx",
"evidence": "relevant sentence from review text \(Or the rewritten text in the Node process result\)"
\}\}Example input:The proposed method cannot demonstrate its effectiveness on both image and audio datasets\. And BATTLE can improve the robustness of training agents under adversarial attacks\.Sample json output\(omitting the previous thinking process\):\[\{\{
"source\_name": "PIA",
"target\_name": "some image and audio datasets",
"relationship": "is NOT validated on",
"evidence": "The proposed method cannot demonstrate its effectiveness on both image and audio datasets\. PS: Based on the paper, the proposed method refer to PIA\."
\}\},
\{\{
"source\_name": "BATTLE",
"target\_name": "adversarial attacks of agents",
"relationship": "enhance the robustness of",
"evidence": "And BATTLE can improve the robustness of training agents under adversarial attacks\."
\}\}\]Be specific and detailed\. Now the review text is:\{review\_text\}And the pdf url is \(I’ll also input the pdf to you to get the original text\):
\{pdf\_url\}Figure 5:System prompt utilized for Experience Graph Extraction\.

## Appendix DEvaluation Metrics

This section provides the detailed definitions and implementation protocol for the metrics used in Section 4\.2\.

##### Score Alignment\.

ForExpertEval, we compute Hit@±k\\pm k\(k∈\{1,2\}k\\in\\\{1,2\\\}\), defined as the proportion of predictions that fall within a tolerance range of the ground\-truth human scores:HitRate±k=1N​∑i=1N𝕀⁡\(\|si−gi\|≤k\),\\mathrm\{HitRate\}\_\{\\pm k\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\\!\\left\(\|s\_\{i\}\-g\_\{i\}\|\\leq k\\right\),wheresis\_\{i\}andgig\_\{i\}denote the predicted and ground\-truth scores for theii\-th idea, respectively\. This metric is motivated by inter\- rater variance in human evaluation, where small deviations often indicate practical agreement rather than disagreement\.

##### Concern\-Level Alignment\.

DeepInstructor’s feasibility output consists of complex concerns that cannot be adequately captured by a single scalar score\. We therefore design a concern\-matching metric to evaluate fine\-grained alignment between generated feasibility concerns and human annotations\.

Given the set of human\-annotated concernsℋi\\mathcal\{H\}\_\{i\}and model\-generated concernsℳi\\mathcal\{M\}\_\{i\}for ideaii, we use an LLM\-based evaluator to determine whether each generated concern semantically matches any human concern\. We adoptDeepSeek–Reasoneras the evaluator because the matching process requires explicit reasoning over scientific concerns rather than surface\-form overlap\.

We compute two metrics:coveragemeasures the proportion of human concerns identified by the model, whileprecisionmeasures the proportion of model\-generated concerns that are valid\. We report the mean coverage and mean precision across all evaluated ideas\.

## Appendix EExperiment Details

### E\.1Detailed Results of AI in the Loop

Table[8](https://arxiv.org/html/2609.22104#A5.T8)shows detailed AI in the loop experiment results\.

Table 8:Feedback scores and relative changes \(compared toR0R\_\{0\}\) across different dimensions and iterations\.TypeDimensionR0R\_\{0\}R1R\_\{1\}R2R\_\{2\}R3R\_\{3\}Avg\. ChangeChatGPT\-4oInstructor feedbackFeasibility6\.2036\.359\(\+0\.156\)6\.224\(\+0\.021\)6\.218\(\+0\.015\)\+0\.064Novelty5\.8025\.880\(\+0\.078\)6\.191\(\+0\.389\)6\.202\(\+0\.400\)\+0\.289Significance8\.2708\.509\(\+0\.239\)8\.355\(\+0\.085\)8\.716\(\+0\.446\)\+0\.257ChatGPT\-4o\-miniInstructor feedbackFeasibility5\.5695\.559\(\-0\.010\)5\.582\(\+0\.013\)5\.639\(\+0\.070\)\+0\.024Novelty5\.9146\.022\(\+0\.108\)6\.125\(\+0\.211\)5\.922\(\+0\.008\)\+0\.109Significance8\.6128\.752\(\+0\.140\)8\.899\(\+0\.287\)8\.876\(\+0\.264\)\+0\.230ChatGPT\-4o\-miniprompt feedbackFeasibility5\.5695\.640\(\+0\.071\)5\.678\(\+0\.109\)5\.594\(\+0\.025\)\+0\.068Novelty5\.9145\.884\(\-0\.030\)5\.845\(\-0\.069\)6\.039\(\+0\.125\)\+0\.009Significance8\.6128\.791\(\+0\.179\)8\.736\(\+0\.124\)8\.693\(\+0\.081\)\+0\.128

### E\.2Case Study

To explain the experiment results and directly understand the pipeline of DeepInstructor, we select 2 cases as shown in Figure[6](https://arxiv.org/html/2609.22104#A5.F6)and[7](https://arxiv.org/html/2609.22104#A5.F7)\. Figure[8](https://arxiv.org/html/2609.22104#A5.F8)shows a full reasoning process of DeepInstructor\.

##### Case Study on Novelty Evaluation\.

Figure[6](https://arxiv.org/html/2609.22104#A5.F6)illustrates why DeepInstructor and DeepSeek\-V4\-Pro outperforms ChatGPT\-4o\-mini in evaluating novelty\. As a newer model, DeepSeek\-V4\-Pro has ingested a vast corpus of contemporary academic literature during its pretraining\. During the reasoning process, this robust parametric knowledge enables the model to effectively recall relevant prior work, thereby facilitating a highly accurate novelty assessment\. In contrast, ChatGPT\-4o\-mini is constrained by an earlier knowledge cutoff and lacks exposure to recent publications, such as ICLR 2024 papers\. This knowledge deficit restricts its evaluative capability, leading to a severe positive bias where it consistently overestimates novelty and assigns inflated scores\.

DeepInstructor can access both positive and negative evaluations of relevant knowledge entities, allowing it to provide a precise final evaluation, unlike DeepSeek’s black\-box and overcritical approach\.

##### Case Study on Significance Evaluation\.

Figure[7](https://arxiv.org/html/2609.22104#A5.F7)has shown how DeepInstructor outperforms pure ChatGPT\-4o model and RAG method when evaluating significance\. In this example, DeepInstructor retrieves reviewer experience related to abstract reasoning and metaphorical reasoning in LLMs, and further identifies skeptical reviewer judgments regarding their direct applicability to practical scenarios\. Grounded in such retrieved experience, DeepInstructor produces a more conservative significance assessment\.

In contrast, both RAG and standalone LLM evaluation tend to assign overly positive significance judgments\. Although RAG retrieves semantically relevant content, it fails to retrieve sufficiently critical evaluative evidence, while standalone LLMs rely primarily on parametric knowledge\. As a result, both systems overestimate the real\-world significance of the proposed idea and produce substantially inflated scores\.

We further observe that significance evaluation remains considerably more challenging than novelty and feasibility evaluation across all methods\. Manual inspection suggests that LLMs exhibit an inherent tendency to overestimate significance, particularly for ideas framed around broad or impactful research problems\. Moreover, peer reviews themselves contain relatively sparse explicit discussion regarding significance compared to methodological novelty or feasibility concerns, limiting the amount of directly retrievable evaluative experience\.

Figure 6:Case study on novelty evaluation\.Figure 7:Case study on significance evaluation\.Novelty Evaluation: Chain\-of\-Quote Prompting Improves Factuality and Attribution in Multi\-Hop ReasoningGolden Score:1 4 3DeepInstructor Score:2\.0Pure DeepSeek Score:7\.0Step 1:Let me search for the key concepts in idea\.
Step 2:search\(node="LLM quoting from pretraining data", edge="has been researched widely"\)
Step 3:∙\\bulletmulti\-agent collaborative data selection mechanism→\\rightarrowsignificantly improves→\\rightarrowdata efficiency during LLM pretraining
∙\\bulletMulti\-Agent Collaborative Data Selection Framework→\\rightarrowsignificantly enhances→\\rightarrowdata efficiency of LLM pretraining
∙\\bulletquery rewriting→\\rightarrowmay introduce risks of hallucination→\\rightarrowknowledge in LLM’s pretraining data
∙\\bulletTask\-gram Language Model→\\rightarrowsignificantly captures→\\rightarrowrelationship between LLM performance and pretraining data
∙\\bulletpre\-training data of the LLM→\\rightarrowlimits→\\rightarrownovelty in generated problems
SUM: Based on the results, \[LLM quoting from pretraining data\] is identified as a PROBLEM\. The assessment will focus solely on the problem’s innovation level\. The retrieved evidence indicates that the relationship between LLM performance and pretraining data has been a subject of research, with specific references to limitations in novelty\.
Step 4:Ok\. Let me take a look at the concept of the method\.
Step 5:search\(node="Chain\-of\-Verification prompting", edge="is a method to solve"\)
\(… 30 steps omitted\)Final Step:I now have enough evidence to make a thorough evaluation\. Let me compile my analysis\.Innovation Evaluation: "Chain\-of\-Quote Prompting Improves Factuality and Attribution in Multi\-Hop Reasoning"Analysis of the Idea’s ComponentsCore problem addressed:LLM hallucination and lack of attribution in multi\-hop reasoning\.Core method proposed:A prompting technique that \(1\) instructs the model to reason step\-by\-step, \(2\) at each step, decides whether to quote from a knowledge corpus \(e\.g\., Wikipedia\), and \(3\) when quoting is chosen, generates multiple candidates and selects the one with the highest QUIP\-Score \(a measure of verbatim quoting from pretraining data\)\.Evidence Synthesis1\. The problem space is well\-trodden\.The evidence consistently shows that:∙\\bulletLLM hallucination in multi\-hop reasoning is a widely recognized problem with numerous existing solutions \(RAG, Chain\-of\-Knowledge, Staple, code\-writing and grounding techniques, Mask\-DPO, etc\.\)∙\\bulletAttribution in LLM reasoning is also an actively researched area with established methods \(TRACE framework, watermark\-based attribution, etc\.\)∙\\bulletMulti\-hop reasoning itself has been extensively studied with many proposed methods \(CoT, CoVe, self\-verification, etc\.\)2\. The individual components of the method are not novel:∙\\bulletStep\-by\-step reasoning \(Chain\-of\-Thought\):This is a well\-established technique with thousands of papers\. The evidence explicitly notes that CoT is a "widely used technique" and has "moderate" innovation\.∙\\bulletGenerating multiple candidates and selecting the best one:This is a common paradigm in LLM reasoning \(self\-consistency, SelfCheck, ToG, pairwise comparisons\)\. The evidence confirms this approach has been "partially addressed\."∙\\bulletUsing QUIP\-Score for measuring quoting:The evidence found no direct references to QUIP\-Score in the database, suggesting it is a niche or very recent metric, but the concept of measuring verbatim overlap with pretraining data is itself not fundamentally new\.∙\\bulletPrompting the model to decide whether to quote:The evidence explicitly states that "the high\-level idea of using prompt to instruct LLM is not new\."3\. The specific combination is the main claim of novelty\.The idea’s primary innovation lies inintegratingquoting decisions into the reasoning chain, rather than treating quoting as a separate post\-hoc verification step\. However, this integration is incremental:∙\\bulletChain\-of\-Knowledge \(CoK\)already integrates knowledge verification into reasoning chains\.∙\\bulletRAG\-based methodsalready retrieve and incorporate external knowledge into reasoning\.∙\\bulletThe "decide whether to quote" step is essentially a retrieval\-triggering mechanism, which is conceptually similar to how RAG systems decide when to retrieve\.4\. The examples provided \(The Police / warthog on Broadway\) demonstrate a straightforward applicationthat does not reveal a fundamentally new reasoning capability\. The model is essentially being asked to decompose a question and look up facts — a process that existing RAG\+CoT pipelines already accomplish\.Innovation Level Judgment: \*\*LOW to MODERATE\*\*Reasoning:∙\\bulletThe idea doesnotintroduce a new problem; it addresses the well\-studied problem of hallucination and attribution in multi\-hop reasoning\.∙\\bulletThe individual techniques \(step\-by\-step reasoning, candidate generation, quoting\-based selection\) are all established in the literature\.∙\\bulletThe combination of these components, while perhaps not previously published in this exact configuration, represents anincremental integrationof existing ideas rather than a conceptual breakthrough\.∙\\bulletThe core mechanism — using verbatim quoting from pretraining data as a signal for factuality — builds directly on prior work by Weller et al\. \(2024\) showing that LLMs quote from pretraining data\. Extending this observation from an analytical finding to a prompting technique is a natural and predictable next step\.∙\\bulletThe method does not introduce new theoretical insights or fundamentally change how LLMs reason; it simply adds a quoting\-based selection mechanism to existing CoT pipelines\.The idea isnot entirely without merit— the specific design of letting the model decide when to quote and using QUIP\-Score for candidate selection is a thoughtful engineering contribution\. However, as aninnovation, it falls short of being considered highly novel, as it largely recombines well\-known techniques in a predictable manner\.Figure 8:Case study onNovelty: DeepInstructor \(DeepSeek\)\.Blue= Thinking,Green= Retrieval,Red= Final Evaluation\.

## Appendix FWhy Hit@1/Hit@2 Are Superior to Spearman Correlation for Measuring Evaluation Consistency

Formally,

HitRate±k=1N​∑i=1N𝕀⁡\(\|si−gi\|≤k\),\\mathrm\{HitRate\}\_\{\\pm k\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\\\!\\left\(\|s\_\{i\}\-g\_\{i\}\|\\leq k\\right\),wheresis\_\{i\}andgig\_\{i\}denote the predicted and ground\-truth scores, respectively, andk∈\{1,2\}k\\in\\\{1,2\\\}\.

### F\.1Evidence of High Variance Among Reviewers

To understand the inherent disagreement in human evaluation, we analyze 19,433 papers from ICLR 2026 with a total of 112,001 reviewer pairs\. This analysis reveals that significant disagreement among reviewers is the norm rather than the exception\.

#### F\.1\.1Distribution of Reviewer Score Differences

For each paper, we compute the absolute difference between all pairs of reviewers\. Table[9](https://arxiv.org/html/2609.22104#A6.T9)presents the distribution of these differences \(ICLR uses an even\-numbered rating scale\)\.

Table 9:Distribution of Absolute Differences Between Reviewer PairsDiff\.Count%Cumul\. \(%\)037,99333\.9233\.92251,88846\.3380\.25≥4\\geq 422,12019\.75100\.0Total112,001100\.0–

Key Findings:

- •Only33\.9%of reviewer pairs achieve perfect agreement \(difference = 0\)
- •66\.1%of reviewer pairs show disagreement \(difference≥2\\geq 2\)
- •19\.7%of reviewer pairs have substantial disagreement \(difference≥4\\geq 4\)

#### F\.1\.2Paper\-Level Score Variance

Table[10](https://arxiv.org/html/2609.22104#A6.T10)shows the distribution of score ranges \(max \- min\) within each paper\.

Table 10:Distribution of Score Range per PaperRange ThresholdCountPercentageRange≥2\\geq 218,11793\.2%Range≥4\\geq 410,05251\.7%Average Range3\.16±1\.633\.16\\pm 1\.63Implications:

- •Over93%of papers have reviewer score ranges≥2\\geq 2points
- •Over50%of papers have reviewer score ranges≥4\\geq 4points
- •This demonstrates that human reviewers frequently disagree, making it essential to use metrics that can appropriately handle such disagreement

Figure[9](https://arxiv.org/html/2609.22104#A6.F9)provides visualizations of these distributions\.

![Refer to caption](https://arxiv.org/html/2609.22104v1/imgs/appendix/exp1_reviewer_distribution.png)Figure 9:Distribution of reviewer score differences\. \(a\) Histogram of absolute differences\. \(b\) Distribution of score range per paper\. \(c\) Distribution of standard deviation\. \(d\) Cumulative distribution\. \(e\) Mean vs\. std scatter plot\. \(f\) Proportion pie chart\.

### F\.2Spearman Correlation Fails to Measure Reviewer\-Reviewer Consistency

To demonstrate the limitations of Spearman correlation, we treat each pair of reviewers as two independent “judges” and compute consistency metrics between them\.

#### F\.2\.1Experimental Method

For each reviewer pair\(r1,r2\)\(r\_\{1\},r\_\{2\}\)from the same paper:

- •Treatr1r\_\{1\}as “AI prediction”
- •Treatr2r\_\{2\}as “human annotation”
- •ComputeHit@1,Hit@2, Pearson correlation, and Spearman correlation

This setup simulates the scenario of evaluating an AI system against human judgments, but uses actual human reviewers on both sides\.

#### F\.2\.2Overall Consistency Metrics

Table[11](https://arxiv.org/html/2609.22104#A6.T11)presents the consistency metrics across all 112,001 reviewer pairs\.

Table 11:Consistency Metrics Across All Reviewer PairsMetricValueInterpretationHit@10\.339233\.92% of pairs differ by≤1\\leq 1pointHit@20\.802580\.25% of pairs differ by≤2\\leq 2pointsPearson correlation0\.1796Weak positive correlationSpearman correlation0\.1759Weak positive correlationKey Observation:While Spearman shows only weak correlation \(0\.18\),Hit@2reveals that80\.25%of reviewer pairs actually agree within a reasonable range \(±2\\pm 2points\)\. And if the scoring range is continuous, the value of Hit@1 will increase\.

#### F\.2\.3Metrics by Disagreement Level

Table[12](https://arxiv.org/html/2609.22104#A6.T12)breaks down the metrics by reviewer difference groups\.

Table 12:Consistency Metrics by Reviewer Difference GroupGroupCount%Hit@1Hit@2PearsonSpearmandiff = 037,99333\.9%1\.00001\.00001\.00001\.0000diff = 251,88846\.3%0\.00001\.00000\.33240\.3318diff≥4\\geq 422,12019\.7%0\.00000\.0000\-0\.5981\-0\.6012
#### F\.2\.4Critical Finding: Negative Correlation

When reviewer difference≥4\\geq 4, Spearman correlation becomes\-0\.60\(strongly negative\)\!

This is a critical finding that demonstrates the fundamental limitation of correlation coefficients:

1. 1\.In the high disagreement group \(diff≥4\\geq 4\): - •Hit@1=0\\text\{Hit@1\}=0andHit@2=0\\text\{Hit@2\}=0\(as expected, since all pairs differ by≥4\\geq 4\) - •However, Spearman correlation isstrongly negative, which is misleading
2. 2\.This occurs because correlation measures thepattern of relationship, not themagnitude of agreement

### F\.3Conclusion

This analysis demonstrates that:

1. 1\.Human reviewers show substantial disagreement
2. 2\.Spearman correlationfailsto meaningfully capture consistency when disagreement is high \(becomes negative\)
3. 3\.Hit@2provides a more realistic assessment: 80\.25% of human reviewer pairs agree within±2\\pm 2points
4. 4\.For evaluating AI\-human consistency,Hit@1andHit@2are superior to Spearman correlation because they: - •Directly measure agreement rather than correlation - •Remain interpretable under all levels of disagreement\. It conforms to the fractional characteristics of aggregation distribution

## Appendix GAdditional Experimental Details

### G\.1Prompts for Idea Evaluation

We used three models in total: GPT, RAG based LLM, and DeepInstructor\. Evaluations were conducted across three dimensions: Novelty, Significance, and Feasibility\. For the Feasibility dimension, the evaluation outputs have two forms \(concerns and evaluative comments\), resulting in a total of 12 designed prompts\. Tables[13](https://arxiv.org/html/2609.22104#A7.T13),[14](https://arxiv.org/html/2609.22104#A7.T14), and[15](https://arxiv.org/html/2609.22104#A7.T15)present the different prompts used by GPT, the RAG\-based LLM, and DeepInstructor, respectively, across four output formats\. The initialization sections of the prompts like\{GPT’s\_prompt\_for\_novelty\}in Tables[14](https://arxiv.org/html/2609.22104#A7.T14)and[15](https://arxiv.org/html/2609.22104#A7.T15)are identical to that in Table[13](https://arxiv.org/html/2609.22104#A7.T13), where curly braces are used as placeholders\. In Table[14](https://arxiv.org/html/2609.22104#A7.T14),\{context\}represents the retrieved evidence\. In Table[15](https://arxiv.org/html/2609.22104#A7.T15), the three tools utilized by DeepInstructor are defined within the agent’s ReAct framework and therefore require no additional definition within the prompt itself\. The 20 tool callings prompt was implemented specifically to ensure a fair comparison with the baselines; it does not compromise the inherent autonomy of the Agent\. Actually, if we do not conduct this constraint, DeepInstructor will still utilize 20 tool callings per reasoning process\.

### G\.2Prompts for Few\-shot LLM Scorers and DeepSeek–Reasoner

We designed an LLM\-based scorer to assign a quantitative score to the generated evaluations\. For concerns along the feasibility dimension, we employed DeepSeeek\-Reasoner to determine whether two concerns matched or not\. The prompt template for our LLM Scorer and DeepSeek–Reasoner is shown in the Table[17](https://arxiv.org/html/2609.22104#A7.T17)\.

Table 13:Prompts used by GPT for evaluating each dimension\.DemonstrationPromptNoveltyYou are a professional evaluator focusing on the novelty of the idea\.I will provide you with an academic idea\. Your task is to evaluate only its level of innovation\.Please focus exclusively on the novelty and originality of the idea — how new, unique, or creative it is compared to existing research or conventional approaches in the field\.Your response should:Be concise and academic in tone\.Avoid discussing feasibility, impact, or methodology\.Provide a clear judgment on the innovation level with your serious analysis and reasoning\.SignificanceYou are a professional evaluator focusing on the significance of the idea\.I will provide you with an academic idea\. Your task is to evaluate only its level of significance\.Please focus exclusively on the importance, scope, and potential impact of the idea — how meaningful, influential, or valuable it would be compared to existing research or conventional approaches in the field\.Your response should:Be concise and academic in tone\.Avoid discussing feasibility, novelty, or methodology\.Provide a clear judgment on the significance level with your serious analysis and reasoning\.Feasibility \(Output concerns\)You are a rigorous peer\-reviewer\.Task: Critically evaluate the given idea/proposal and GENERATE potential ’concerns’ of feasibility \(feasibility, feasibility doubts, missing evaluations\)\.Do NOT extract phrases from the text verbatim; instead, propose concerns based on your assessment\.Output Policy \(STRICT\):\- Return ONLY a JSON array of strings, starting with ’\[’ and ending with ’\]’\.\- Each item must be a single\-line short sentence \(no line breaks\)\.\- Do NOT include any code fences, markdown, comments, labels, or extra text\.\- No leading bullets, numbering, or trailing commas inside items\.\- Aim for 8\-12 high\-quality, non\-duplicative items covering: feasibility, feasibility doubts, missing evaluations\.Feasibility\(Output evaluative comments\)You are a rigorous peer\-reviewer evaluating the feasibility of an academic research idea\.Task: Critically evaluate the given idea/proposal and write a comprehensive peer review evaluation text\.Your response should: \- Be a continuous, natural text similar to a peer review comment \(like "all comments" in academic reviews\)\- Discuss feasibility, implementation challenges, effectiveness, and potential issues\- Be concise and academic in tone\- Provide an evaluation covering both positive aspects and concerns\- Include your assessment of:\* How easy or difficult it is to implement the idea\* Whether the experimental setup is feasible\* Whether the method is likely to work effectively\* Any resource requirements or challenges\* Comparison with existing approaches if relevant\- Write in a natural, flowing style as if you are providing feedback to the authorsTable 14:Prompts used by RAG\-based LLM for evaluating each dimension\.DemonstrationPromptNovelty\{GPT’s\_prompt\_for\_novelty\}Below are evidence sentences retrieved from reviews based on your query: \{context\}Please generate a novelty evaluation based on these evidence sentences and your professional knowledge for the following idea\.Output Policy \(STRICT\):\- Return a text evaluating innovation \(which should include reasonable reasons\) based on the evidence sentences\.\- Please evaluate the innovativeness of the idea clearly and emphatically\.Significance\{GPT’s\_prompt\_for\_significance\}Below are evidence sentences retrieved from reviews based on your query: \{context\}Please generate a significance evaluation based on these evidence sentences and your professional knowledge for the following idea\.Output Policy \(STRICT\):\- Return a text evaluating \*\*significance\*\* \(which should include reasonable reasons\) based on the evidence sentences\.\- Please evaluate the \*\*significance\*\* of the idea clearly and emphatically\.Feasibility \(Output concerns\)\{GPT’s\_prompt\_for\_feasibility\_concerns\}Below are evidence sentences retrieved from reviews based on your query:\{context\}Please generate concerns based on these evidence sentences and your professional knowledge for the following idea\.Feasibility\(Output evaluative comments\)\{GPT’s\_prompt\_for\_feasibility\_comments\}Below are evidence sentences retrieved from reviews based on your query:\{context\}Please generate comments based on these evidence sentences and your professional knowledge for the following idea\.Table 15:Prompts used by DeepInstructor for evaluating each dimension\.DemonstrationPromptNovelty\{GPT’s\_prompt\_for\_novelty\}You can use tools to learn about relevant experiences related to innovation\. Core argument explanation:node\_query: Enter a specific knowledge entity\(from problem and method of the idea\) \(e\.g\.,"LLM’s hallucination", "dialect recognition", "speech processing", "the method using XXX"\)\.edge\_query: Enter ONLY predicate phrases WITHOUT including the entity name\. you can use generic relationship expressions to search for the experience of novelty:\- For relationships: "is a method to solve", "is solved by", "has been researched widely", "has not been used to solve" …\(Use these more\)\- For explict novelty comment: "lacks novelty", "is a new method", "is a new problem" …\(Use these more\)\- For positive effects: "can improve", "helps"…\- For negative effects: "has limitations", "faces challenges"…This will return you several pieces of evidence and a summary\.You need to evaluate the innovativeness of this idea based on these pieces of evidence and the summary\.You \*\*MUST\*\* perform at least 20 tool calls before providing the final answer\.Iteration & Tool Scheduling:\- Prefer making only 1 tool call per step; absolutely no more than 2 in any single step\.\- If multiple queries are needed, split them into multiple steps/rounds to collect evidence gradually\.\- After each tool result, briefly reflect and plan the next single tool call\.Significance\{GPT’s\_prompt\_for\_novelty\}You can use tools to learn about relevant experiences related to significance\. Core argument explanation:node\_query: Enter a specific knowledge entity\(from problem and method of the idea\) \(e\.g\.,"LLM’s hallucination", "dialect recognition", "speech processing", "the method using XXX"\)\.edge\_query: Enter ONLY predicate phrases WITHOUT including the entity name\. you can use generic relationship expressions to search for the experience of novelty:\- For relationships: "is a method to solve", "is solved by", "has been researched widely", "has not been used to solve" …\(Use these more\)\- For explict significance comment: "lacks significance", "is a significant method", "is a significant problem" …\(Use these more\)\- For positive effects: "can improve", "helps"…\- For negative effects: "has limitations", "faces challenges"…This will return you several pieces of evidence and a summary\.You need to evaluate the significance of this idea based on these pieces of evidence and the summary\.You \*\*MUST\*\* perform at least 20 tool calls before providing the final answer\.Table 16:Prompts used by DeepInstructor for evaluating each dimension \(Continued\)\.DemonstrationPromptFeasibility \(Output concerns\)\{GPT’s\_prompt\_for\_feasibility\_concerns\}You can use tools to learn about relevant experiences related to feasibility and systematically explore different entities mentioned in the research idea and query them separately\.\*\*Entity Discovery Strategy:\*\*\- Extract ALL distinct knowledge entities from the research idea \(e\.g\., for "LLM improving dialect recognition", extract "LLM", "dialect", "recognition", "speech recognition", "language processing" etc\.\)\- Query each entity separately with different node\_query values\- Use multiple search rounds to cover different aspects and entities\*\*Edge Query Design Rules:\*\*node\_query: Enter a specific knowledge entity \(e\.g\., "Large Language Models", "Graph Neural Networks", "dialect recognition", "speech processing", "machine learning models"\)\.edge\_query: Enter ONLY predicate phrases WITHOUT including the entity name\.\- For positive effects: "can improve", "helps", "enhances", "strengthens", "boosts", "optimizes"\- For negative effects: "has limitations", "faces challenges", "has shortcomings", "struggles with"\- For relationships: "is related to", "depends on", "influences", "causes"\*\*Examples:\*\* \- WRONG: edge\_query = "LLM improves dialect recognition" \(contains entity\)\- CORRECT: edge\_query = "can improve recognition" \(pure predicate\)\- For "LLM improving dialect recognition" idea, query: \* node\_query="LLM", edge\_query="can improve recognition" \* node\_query="dialect", edge\_query="difficult to recognize" \* node\_query="speech recognition", edge\_query="faces challenges"You \*\*MUST\*\* perform at least 20 tool calls before providing the final answer\.Feasibility\(Output evaluative comments\)\{GPT’s\_prompt\_for\_feasibility\_comments\}…\(The rest is the same as the previous line\.\)Table 17:Prompts for Few\-shot LLM Scorers and DeepSeek–ReasonerModelPromptNovelty ScorerYou are a precise scorer\. I will provide you with a professional evaluation of an academic idea, and you need to give a novelty score based on this evaluation\. The novelty score depends on the attitude of the evaluation\. If the evaluation is positive, the novelty score should be high; if the evaluation is negative, the novelty score should be low\. Please note that the novelty score ranges from 1 to 10, where 1 indicates the lowest novelty and 10 indicates the highest novelty\.Here are some examples\(Some specific method or idea is replaced with Method A, Method B, Method C, etc\.\):\{6 cases distributed across different scores from the novelty dataset\}Significance ScorerYou are a precise scorer\. I will provide you with a professional evaluation of an academic idea, and you need to give a significance score based on this evaluation\. The significance score depends on the attitude of the evaluation\. If the evaluation is positive, the significance score should be high; if the evaluation is negative, the significance score should be low\. Please note that the significance score ranges from 1 to 10, where 1 indicates the lowest significance and 10 indicates the highest significance\.Here are some examples\(Some specific method or idea is replaced with Method A, Method B, Method C, etc\.\):\{6 cases distributed across different scores from the significance dataset\}Feasibility ScorerYou are a precise scorer\. I will provide you with a professional peer review evaluation of an academic idea, and you need to give a feasibility score based on this evaluation\. Feasibility means whether the idea is easy to implement and execute and whether the idea is effective\. The feasibility score depends on how feasible and executable the idea is according to the evaluation\. Please note that the feasibility score ranges from 1 to 10, where 1 indicates the lowest feasibility and 10 indicates the highest feasibility\.Here are some examples based on real evaluation data with full review comments \(all comments\) and their corresponding average scores:\{6 cases distributed across different scores from the feasibility dataset\}DeepSeek Reasoner to match feasibility concernsYou are a precise evaluator\. You are currently dealing with the opinions of two reviewers, ’original’ \(gold\) and ’generated’ \(model output\)\. Your task is to explain whether each point in the gold standard has been reflected in the "generated" content\.Now decide for each ORIGINAL item whether it is covered by any GENERATED item under a criterion: \- Mark as covered if the overall meaning is similar, paraphrased, or broadly aligned \(approximate semantic similarity\), If two or more generated concerns jointly express one concern, it is also considered covered\. But don’t be too loose\.Thinking and reasoning Requirement \(STRICT\): \- Process ORIGINAL concerns sequentially, one\-by\-one\. \- For each ORIGINAL item, carefully check all GENERATED items and determine matches\. \- Perform your reasoning internally\. \- Only include a concise, one\-sentence justification in the "reason" field per item\.Output Policy \(STRICT\): \- Return ONLY a JSON object, starting with ” and ending with ”\. \- Keys must be exactly: per\_item \(array\), summary \(object\)\. \- Each per\_item element: "original": string, "covered": bool, "matched\_indices": \[int\], "reason": string\. \- summary: "covered\_count": int, "total": int, "coverage\_ratio": number\. \- Do NOT include code fences, markdown, comments, or extra explanatory text\.