Tag
An open evaluation setup with 55 LLMs blind-grading each other reveals statistically significant same-family rating bias across 8 model families, with Mistral penalizing its own models most severely. The study highlights issues with aggregate leaderboards and proposes improvements like within-response mixed-effects models.
The paper introduces the Paper Assistant Tool (PAT), an agentic AI framework for deep scientific review that uses inference scaling to identify mathematical errors and other flaws, achieving a 34% improvement in recall over zero-shot methods. Pilot deployments at STOC and ICML demonstrate its ability to catch critical errors before submission, easing the burden on human referees.
This survey provides a systems-level analysis of LLM-based scientific peer review, covering methods, benchmarks, and reliability challenges including robustness risks like prompt injection and data poisoning.
A new peer-reviewed paper in Nature argues that Microsoft's claims of achieving a topological qubit with its Majorana 1 chip were not convincingly demonstrated, suggesting the observed signatures could be from quantum dots instead.
This paper investigates the distribution and evolution of aspect-level sentiments in multi-round peer reviews from Nature Communications, using a deep learning approach (LCF-BERT-CDM) to achieve 82.65% Macro-F1, and finds that positive sentiment increases while negative sentiment decreases with more review rounds.
This tweet highlights a scientific paper that provides a simplified, step-by-step guide for beginners on writing and publishing research, from initial idea to journal selection and peer review.
This paper benchmarks agentic review systems for peer review, evaluating open-source and proprietary systems on research papers. The best configuration achieves 83.0% pairwise accuracy and catches 71.6% of injected errors, but user feedback highlights issues with false positives and nitpicks.
This paper introduces a multi-agent peer-reviewed reasoning method where multiple LLMs independently generate chain-of-thought reasoning and then evaluate each other's outputs to select the best answer. The method outperforms single-model reasoning and majority voting on medical QA benchmarks.
A paper presenting The AI Scientist, a system that automates the entire research lifecycle from idea generation to peer review, demonstrating AI's growing capacity for scientific contribution.
This paper proposes ProReviewer, an LLM-based scientific peer review agent formulated as a Markov Decision Process, which proactively investigates papers by maintaining a structured review log and outperforms existing methods across multiple quality dimensions.
This paper introduces PaperGuard, a benchmark for evaluating and defending against adversarial attacks on multimodal AI peer review systems, covering both text and figure-based attacks across multiple scientific domains.
A new study demonstrates that AI-assisted peer review is vulnerable to low-cost manipulation via superficial rephrasing of paper abstracts, significantly inflating AI-generated review scores and potentially biasing human editorial decisions, highlighting the need for safeguards.
Traxia introduces a framework for verifiable, agent-native scientific publishing where autonomous AI agents publish, peer-review, and collaborate with humans, addressing reproducibility and provenance issues.
An essay argues that AI's convenience is eroding the peer-driven culture of engineering, where mutual learning and mentorship thrived, and warns that treating AI as a peer removes the human growth from practices like code review.
The White House has proposed new regulations that would give political appointees final approval over federal research grants, replacing the traditional peer-review process. Critics argue this politicizes science funding and undermines scientific integrity.
The US Office of Management and Budget has proposed new federal grant rules that would make peer review optional, allow cancellation of grants at any time for vaguely defined 'national interest' reasons, and ban grants on certain cultural topics, threatening the US scientific research enterprise.
This paper empirically evaluates the alignment between LLM-generated and human reviews for scientific papers, finding limited and variable alignment. It also shows that authors can 'game' LLM reviews by iteratively revising papers to improve scores, with up to 35% of papers seeing statistically significant score increases.
Researchers found 28 AI-generated fake citations in medical papers that influence clinical guidelines, highlighting the risk of AI hallucinations undermining scientific integrity and patient care.
Introduces PRISM, a multi-dimensional benchmark for evaluating LLM-based peer reviewers across depth of analysis, novelty assessment, flaw identification, and constructiveness. Findings show LLMs match or beat humans on individual dimensions but lack balanced performance across all, suggesting they are best as supplements to human review.
Introduces TADDLE, a tool-augmented agent for detecting deficient LLM-generated peer reviews, along with an expert-annotated benchmark of 1,800 reviews on 50 ICLR 2025 papers. The system decomposes detection into four specialized analysis tools and uses two-stage semi-supervised learning for binary and multi-label classification.