@yoheinakajima: more researchers should use @exscienceAI to review their papers
Summary
Yohei Nakajima highlights research by Jasper Dekoninck showing that GPT-6.1 Sol can detect errors in arXiv papers' main theorems, finding ~6% of pre-August 2026 papers contain false main theorems versus only 1.5% in recent months. The post advocates for broader use of AI tools like exscienceAI in academic peer review.
View Cached Full Text
Cached at: 10/16/26, 02:41 PM
more researchers should use @exscienceAI to review their papers
Jasper Dekoninck (@j_dekoninck): How often do ArXiv papers contain errors in the main theorems they claim to prove? Using GPT-6.1 Sol to identify such mistakes, it finds:
- 6% of papers before 08/2026 contain main theorems that are false.
- Only 1.5% contain similar mistakes in the last two months.
Similar Articles
On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists
A study evaluating AI reviewers (GPT-5.2, Claude Opus 4.5, Gemini 3.0 Pro) against 45 expert human reviewers on Nature-family papers found that AI reviewers can exceed top-rated humans in aggregate review quality, though they are less correct but raise more significant issues.
Gaming AI-Assisted Peer Reviews Poses New Risks to the Scientific Community
A new study demonstrates that AI-assisted peer review is vulnerable to low-cost manipulation via superficial rephrasing of paper abstracts, significantly inflating AI-generated review scores and potentially biasing human editorial decisions, highlighting the need for safeguards.
Google's Agentic Peer-Reviewer Handled ~10K Papers at ICML/STOC — Formal Research Paper Now Out [R]
Google deployed an agentic AI peer-reviewer at ICML and STOC conferences, reviewing ~10,000 papers with 30-minute turnaround. The formal paper shows it catches 34% more mathematical errors than zero-shot prompting, setting a precedent for AI-automated scientific review at scale.
Towards Automating Scientific Review with Google's Paper Assistant Tool
The paper introduces the Paper Assistant Tool (PAT), an agentic AI framework for deep scientific review that uses inference scaling to identify mathematical errors and other flaws, achieving a 34% improvement in recall over zero-shot methods. Pilot deployments at STOC and ICML demonstrate its ability to catch critical errors before submission, easing the burden on human referees.
AI research tools are still too eager to turn public signals into certainty
The author critiques AI research tools for overconfidence in weak signals, praising Komo AI's rapid discovery and source-attached summaries but highlighting the need for better uncertainty and contradiction handling. They describe a workflow that splits discovery, verification, and structured checking across multiple AI tools.