On the Limits of LLM-as-Judge for Scientific Novelty Assessment

Hugging Face Daily Papers Papers
llm novelty-assessment research-questions benchmark human-evaluation scientific-ideation ai-judgment

Summary

This paper introduces RQ-Bench, a benchmark to evaluate LLMs' ability to assess the novelty of scientific research questions. It finds that LLM judges consistently rate generated questions as more novel than human experts do, raising concerns about the reliability of using LLMs for scientific novelty evaluation.

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.
Original Article
View Cached Full Text

Cached at: 06/11/26, 01:41 PM

Paper page - On the Limits of LLM-as-Judge for Scientific Novelty Assessment

Source: https://huggingface.co/papers/2606.12071

Abstract

Research questions generated by large language models exhibit inconsistent novelty assessments when compared to human experts, highlighting concerns about relying on LLMs for scientific novelty evaluation.

LLMs are increasingly used to generate and judge scientific ideas. This makesnovelty evaluationa central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, abenchmarkbuilt from recentarXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They areauthor-anchored reference pointsfor testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, andhuman expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; incomparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictorynovelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty ofresearch questions.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.12071 in a model README.md to link it from this page.

Datasets citing this paper1

#### declare-lab/rq-bench Updatedabout 12 hours ago • 1.31k

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.12071 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

The problem with LLMs as judges

Reddit r/AI_Agents

The article highlights the inefficiency and rationalization issues with using LLMs as judges, introduces Jev as a tool that leverages structured data for more reliable results, and questions the potential for open-source alternatives.