NEW paper worth reading.

(bookmark it)

Autonomous research systems usually prove themselves on cherry-picked wins, human-framed topics, or a handful of preset tasks.

FARS runs the full loop at scale instead. Stage-specific agents handle ideation, planning, experimentation, and writing over a shared workspace that records proposals, code, logs, results, and manuscripts.

Its first public deployment produced 166 complete papers across 67 fine-grained AI/ML topics, and it kept the failures in the corpus rather than curating a highlight reel.

Why it matters.

282 volunteer reviews over 140 papers give an honest read. FARS can produce review-worthy artifacts, while the same reviews expose recurring failure modes in narrow scope, methodology, and integrity.

Paper: https://arxiv.org/abs/2606.31651

Learn to build effective AI agents in our academy: https://academy.dair.ai

FARS: A Fully Automated Research System Deployed at Scale

Source: https://arxiv.org/html/2606.31651 Authors Qiong Tang†, Xiangkun Hu†, Xiangyang Liu†, Yiran Chen†, Yunfan Shao†Analemma{qtang,xkhu,xyliu,yrchen,yfshao}@analemma.ai Reviewers‡\ddagger Bobo Li1, Changze Lv2, Cheng Xu3, Chengsong Huang4, Chunyang Li5, Dizhan Xue6, Hao Bai7, Haodong Duan8, Hengquan Guo9, Hongyang He10, Hongyi Chen11, Hui Shen12, Jiahao Yuan13, Jiankai Sun14, Jikang Cheng15, Jinfeng Xu16, Jingqi Tong2,17, Jingye Chen5, Jinxiu Liu18, Jixuan Leng11, Junchi Yu19, Kaixun Jiang2, Kun Xiang20, Kunpeng Yao21, Lang Feng22, Liangqi Yuan23, Longsen Gao24, Meng Li25, Qi Jia26, Qiushi Sun16, Shengyuan Ding2, Shizhan Gong27, Siru Zhong28, Terry Jingchen Zhang29, Tianle Gu30, Tianyi Liang17,31, Weijie Liu15, Weikai Yang28, Weizhi Fei30, Xiangkun Hu32, Xiangyang Liu32, Xin Wang33, Xinpeng Liu34, Xuanwen Ding2, Yihong Tang35,36, Yuanli Wang37, Yukun Jiang38, Yuming Yang2, Zhengbao He34, Zhikai Chen39, Zhikun Xu40, Zhuang Li41, Zihao Huang8, Anonymous, Anonymous1National University of Singapore,2Fudan University,3University College Dublin,4Washington University in St. Louis,5The Hong Kong University of Science and Technology,6Institute of Automation, Chinese Academy of Sciences,7University of Illinois at Urbana-Champaign,8ByteDance,9ShanghaiTech University,10University of Warwick,11Carnegie Mellon University,12University of Michigan, Ann Arbor,13East China Normal University,14Stanford University,15Tencent,16The University of Hong Kong,17Shanghai Innovation Institute,18Nex-AGI Team,19University of Oxford,20Sun Yat-sen University,21University of Leeds,22Nanyang Technological University,23Purdue University,24University of New Mexico,25Nanjing University,26Shanghai Artificial Intelligence Laboratory,27The Chinese University of Hong Kong,28The Hong Kong University of Science and Technology (Guangzhou),29Vector Institute,30Tsinghua University,31OpenMOSS,32Analemma,33The Ohio State University,34Shanghai Jiao Tong University,35McGill University,36ServiceNow AI Research,37Boston University,38CISPA - Helmholtz-Zentrum für Informationssicherheit gGmbH,39Michigan State University,40Arizona State University,41RMIT University

Abstract

Recent automated research systems show that language-model agents can generate hypotheses, run experiments, and write complete manuscripts, but most evidence still comes from selected examples, human-framed topics, or a few pre-defined research tasks. We present FARS (Fully Automated Research System), a fully automated AI-for-AI research system designed to operate across research topics at scale. FARS autonomously generates and advances projects through ideation, planning, experimentation, and writing, using stage-specific agents coordinated through a shared workspace that records proposals, code, logs, results, and manuscripts. In its first public deployment, FARS produced 166 complete research papers spanning 67 fine-grained AI/ML topics while preserving intermediate artifacts as an auditable corpus rather than a curated set of successes. We evaluate this corpus with 282 structured reviews from volunteer reviewers covering 140 papers, including overall ratings, sub-scores, integrity checks, and LLM-use disclosure. The reviews indicate that FARS can produce review-worthy and occasionally strong AI/ML research artifacts in a large-scale public deployment, while also exposing recurring failure modes in narrow experimental scope, methodological limitations, and integrity issues.

22footnotetext:Equal contribution; human authors listed in alphabetical order by first name.33footnotetext:Named reviewers are listed alphabetically.## 1Introduction

Recent progress in autonomous research systems suggests that language-model agents can now perform substantial portions of the scientific workflow, including literature review, hypothesis generation, experiment execution, reviewing, and paper writing(Asaiet al.,2024; Baeket al.,2025; Schmidgallet al.,2025; Liet al.,2025). Systems such as The AI Scientist(Luet al.,2024), CycleResearcher(Wenget al.,2025a), Zochi(Intology,2025), AI Scientist v2(Yamadaet al.,2025), AIGS(Liuet al.,2024), and AI-Researcher(Tanget al.,2025)demonstrate increasingly complete research pipelines, while DeepScientist(Wenget al.,2025b)shows that autonomous search can make measurable progress on human-defined frontier tasks. However, the existing evidence base remains limited in scope, relying largely on selected demonstrations, human-framed topics, user-provided references, narrowly defined state-of-the-art improvement targets and benchmark tasks. These settings provide useful signals about what autonomous research agents can do, but they do not yet show how such systems behave when they run continuously, select and execute work at scale, and produce large volumes of outputs. This leaves open a central systems question: what happens when a fully automated AI research system is deployed continuously at scale, and how should its outputs be evaluated?

We introduce FARS, an end-to-end AI research system that operates at scale. It is designed to autonomously perform the complete research workflow without human intervention during execution. The guiding principle of FARS is to efficiently and reliably expand the frontier of knowledge. Under this view, the unit of automated research is a focused contribution: a clearly articulated hypothesis paired with an empirical or theoretical validation attempt. Narrowly scoped findings and negative results remain valuable as long as they are observable, well motivated, and verifiable. In its current instantiation, FARS is applied to AI research, positioning it within an emerging AI-for-AI research paradigm. AI research provides a practical deployment domain for autonomous research systems: many hypotheses can be operationalized as executable code and evaluated against shared benchmarks, while advances in AI have high intrinsic value and can also accelerate research in other fields.

FARS organizes research into four sequential stages, namely Ideation, Planning, Experiment, and Writing, coordinated through a shared workspace that serves both as persistent project memory and as an auditable artifact store. In its first continuous public deployment, FARS produced 166 complete papers spanning 67 fine-grained AI/ML topics. This scale contrasts with prior end-to-end systems, whose evaluations typically rely on selected papers, workshop submissions, human-scoped benchmark tasks, or a small number of pre-defined frontier tasks. Our deployment of FARS was designed as a live, large-scale experiment: FARS outputs vary across topics, methods, and quality, making evaluation through a handful of selected examples necessarily limited. We therefore retained successes, weak outputs, negative results, execution artifacts, and failure modes for inspection rather than presenting a curated sample, while external dissemination required human review and explicit labeling.

We evaluate the papers produced during this public deployment based on 282 structured reviews from carefully recruited volunteer reviewers covering 140 generated papers.111The review cycle ran from March 21 to April 12, 2026; 140 of the 166 generated papers received at least one completed review, while the remaining 26 were not reviewed and are excluded from the statistics.The reviews include overall ratings, scores for soundness, presentation, and contribution, reviewer confidence, integrity checks, and disclosure of LLM-assisted review. The results reveal substantial quality variation: FARS can produce review-worthy and sometimes strong research artifacts at a public-deployment scale, while recurring weaknesses include narrow experimental scope, methodological limitations, and integrity issues. These findings suggest that scalable automated research is already feasible, while the reliability of its scientific contribution is still bounded by the substantive value of the contribution, the sufficiency of experimentation, and the faithfulness of writing. Our contributions are as follows:

• •A fully automated research system, FARS, that turns research directions into complete papers through dedicated Ideation, Planning, Experiment, and Writing agents coordinated by a shared auditable workspace.
• •A public deployment of FARS that produced 166 complete papers spanning 67 fine-grained AI/ML topics while preserving an auditable corpus of intermediate artifacts, including hypotheses, plans, code, logs, results, and manuscripts.
• •A large-scale human review of deployment outputs, measuring paper quality, recurring failure modes, and the gap between scalable generation and reliable scientific contribution.

2Related Work

Autonomous Research Systems.

Autonomous research systems have progressed from assisting isolated research tasks toward automating larger portions of the scientific workflow. The AI Scientist(Luet al.,2024)introduced an end-to-end pipeline for idea generation, code generation, experimentation, visualization, paper writing, and automated review, while AI Scientist v2(Yamadaet al.,2025)reduced template dependence and demonstrated workshop-level AI-generated manuscripts. Subsequent systems extend this direction through benchmark-based autonomous scientific innovation(Tanget al.,2025), goal-oriented frontier improvement(Wenget al.,2025b), evidence-grounded claim verification(Menget al.,2026), automated research-review cycles(Wenget al.,2025a), falsification-driven discovery(Liuet al.,2024), and human-verified high-autonomy scientific discovery(Intology,2025). Collectively, these works show that agents can automate substantial parts of the research lifecycle, but they are typically evaluated within human-scoped domains, tasks, templates, benchmark corpora, starting methods, or submission and verification procedures. FARS differs by treating automated AI research as a deployed system for fully autonomous research project progression: from idea generation through planning, experimentation, and paper writing, each project proceeds without human intervention during execution.

Large-Scale Automated Research.

Scale in automated research can mean large search spaces, many experiments, or many generated artifacts. DeepScientist(Wenget al.,2025b)reports thousands of generated ideas and over a thousand experimental validations, but concentrates this search on three frontier tasks initialized from human SOTA methods. AI-Researcher(Tanget al.,2025)evaluates across 22 curated benchmark papers and open-ended variants, but the benchmark is constructed around selected human-authored papers, references, and datasets. AutoSOTA(Liet al.,2026)maps recent top-tier AI papers to executable repositories and optimizes them into improved models, while CodeScientist(Jansenet al.,2025)runs hundreds of code-based experiments, returning 19 candidate discoveries of which 6 pass external and internal validation. Robin(Ghareebet al.,2026)demonstrates an iterative lab-in-the-loop discovery workflow in experimental biology. These systems demonstrate scale in search, optimization, and empirical evaluation within structured settings. FARS instead examines scale through a public deployment of the idea-to-paper pipeline, combining complete paper generation with preserved intermediate artifacts.

Evaluation of AI-Generated Research.

Evaluation remains difficult because research quality depends on soundness, contribution, evidence, and integrity, not only on fluent presentation. LLM-as-a-Judge methods(Zhenget al.,2023; Guet al.,2025)scale cheaply, but they may miss field-specific weaknesses; related work studies overlap between LLM feedback and human review(Lianget al.,2023)and develops standardized automated review protocols(Yuet al.,2024). Idea quality is also insufficient on its own: LLM-generated ideas can be rated as novel while remaining less feasible(Siet al.,2024), and this ideation–execution gap widens when ideas are implemented(Siet al.,2025). Benchmarks such as MLAgentBench(Huanget al.,2024), EXP-Bench(Konet al.,2025), PaperBench(Staraceet al.,2025), and MLR-Bench(Chenet al.,2025)evaluate experimentation, replication, and open-ended ML research, finding that agents still struggle with execution and validation. Recent assurance-oriented systems further emphasize claim auditing, verifiable reporting, and integrity safeguards(Yanget al.,2026; Liuet al.,2026; Gupta and Pruthi,2025; Zhuet al.,2025; Resniket al.,2026). FARS instead evaluates complete papers produced by a deployed autonomous research system using structured human reviews, sub-scores, integrity checks, LLM-use disclosure, and preserved artifacts for inspection.

3FARS

FARS is a multi-agent research system that turns open-ended research directions into complete hypothesis-and-validation papers. It consists of four specialized stages, namely Ideation, Planning, Experiment, and Writing that sequentially transform candidate research hypotheses into executable plans, experimental evidence, and manuscripts. All stages coordinate through a shared workspace, which serves as both project memory and an artifact store for downstream inspection. Figure1provides an overview of the pipeline.

Figure 1:Overview of the FARS architecture. The Ideation stage generates multiple research proposals through iterative refinement, and accepted proposals proceed through Planning, Experiment, and Writing stages. Multiple projects can run independently while sharing a common pipeline structure. Completed manuscripts are paired with code repositories and supporting artifacts. All stages coordinate through a shared workspace, which serves as persistent project memory and an artifact store for downstream inspection; agents also access supplementary tools and infrastructure, including network resources, curated skills, a GPU cluster with 160 NVIDIA GPUs, and model inference endpoints.### 3.1Design Principles

FARS follows three design principles. First, the unit of production is a focused research contribution rather than a maximally polished conference paper. A project should state a hypothesis and make an empirical or theoretical validation attempt. FARS does not treat a negative result as a failed paper by default: when an experiment clearly tests the hypothesis and the limitations are explicit, the trajectory can become a negative-result manuscript rather than being suppressed. Second, the system is artifact-centered: agents exchange stage outputs through a shared workspace rather than hidden state alone, leaving both successful and failed trajectories inspectable. These artifacts provide the interface between stages and the evidence base for downstream inspection. Boundary checks validate stage outputs and trigger retry or feedback when supported. Third, FARS separates ideation, planning, experimentation, and writing into stages with clear inputs, outputs, and checks, allowing many projects to run independently while preserving a common structure for downstream review.

3.2Ideation Stage

Ideation is the first stage of FARS, responsible for turning open-ended research directions into concrete, executable proposals. In the live deployment, it operates without any interactive user input: rather than responding to individual requests, the agent is seeded with nine suggested research topics and is encouraged to explore freely beyond them. From this starting point, it autonomously performs literature review, direction exploration, proposal drafting, and quality assessment. Each proposal that clears automated review is organized around a single, well-scoped hypothesis, covering motivation, approach, related work, experimental design and success criteria. The generated research proposal serves as the input to the Planning stage and is also carried over to other downstream stages of Experiment and Writing.

Internally, Ideation is a collaboration among four specialized agents. A Lead Agent that orchestrates the process—searching open-access papers and public code repositories, curating a three-tier knowledge base (raw text, summaries, and a synthesized domain survey), identifying research gaps, and authoring the proposal; a Summarization Agent that distills the methods, results, and limitations of retrieved papers; a Peer Discussion Agent that adversarially pre-reviews candidate directions to prune weak ideas early; and an Evaluation Agent that conducts the accept/revise/reject judgment and provides revision feedback to the Lead Agent. Together, they drive a six-stage pipeline in an agentic manner: literature review, generation of candidate directions, peer discussion and filtering, proposal drafting, an iterative review-and-revision loop, and finalization.

Since every accepted proposal is forwarded downstream to produce a paper, the live system runs end-to-end: a proposal is admitted only once it clears an automated quality gate, with a circuit breaker that selects the best available draft after repeated failures to prevent unbounded iteration. Candidate directions are further constrained to remain within the AI/ML domain and to yield fully automated experiments that require no human annotation and stay within a fixed compute budget, with detailed planning and implementation deferred to downstream modules.

3.3Planning Stage

The Planning stage is tightly coupled with the Experiment stage, and its design is driven by the demands of long-horizon research execution. Unlike the software-engineering tasks at which coding agents typically excel, a complete research study may take hours or even days to run, yet FARS is intended to carry it out without human intervention. On such long-horizon tasks, current models exhibit two failure modes that are fatal for research: they tend to attempt the entire task in a single pass, giving insufficient thought to each sub-task, and they often declare a task complete after only a shallow attempt(Anthropic,2025). Research projects, however, demand fine-grained control over every detail and a complete, reliable record of experimental procedures and results. To mitigate these failure modes, the Planning stage does not hand the Experiment stage a free-form goal; instead, it converts the accepted proposal into a machine-readable experiment contract that decomposes the study into discrete, ordered units of work.

Concretely, the planning agent first reads the upstream proposal to understand its hypothesis, candidate method design, baselines to compare against, and target benchmarks, and then surveys the experimental setups of related work as a reference for what a rigorous study in this area should contain. Guided by this context, it constructs a structured experiment plan using a two-level hierarchical structure: each entry is an experiment item assigned to one of five task categories, and every item is further decomposed into ordered steps with concrete, executable instructions. The Experiment stage then consumes this structure and completes the steps one item at a time, so that each sub-task receives dedicated attention rather than being rushed in a single sweep.

The five task categories are environment configuration, baseline experiments, main experiments, effectiveness evaluation, and analysis experiments. Together, they encode the intended evidential structure of the study: the environment is prepared first, baselines establish comparison points, main experiments test the proposed method, effectiveness evaluation determines whether the evidence supports continuing, and analysis experiments deepen the interpretation only after the main claim has passed. Automated validation checks the JSON format, required fields, category names, and ordering constraints. Failed validation produces targeted feedback for regeneration, so the plan handed to the Experiment stage is both executable and faithful to the proposal’s experimental logic.

3.4Experiment Stage

The Experiment stage validates whether the hypothesis in the proposal can be realized. It must faithfully execute every task in the validated plan and organize the results so that the Writing stage can reference them programmatically. The central design constraint is that, for autonomous research, execution success alone is not sufficient: the system must preserve enough context for the external observer to determine what was run, what was measured, and whether the results support the claim. This auditability requirement shapes every mechanism in the stage.

The Experiment Loop

An autonomous agent given a multi-task experiment plan may deviate in subtle ways, such as adding unrequested experiments, modifying success criteria, or declaring success on incomplete results. Because the agent controls both execution and self-reporting, such deviations are difficult to detect after the fact. The stage therefore separates orchestration from execution, in which the harness decomposes the validated plan into sequential experiment items and executes them one at a time. Between items, the harness maintains a progress record and accumulates result summaries so that each subsequent agent understands what has been accomplished and can build on prior findings. An effectiveness evaluation gates the transition from main experiments to analysis, ensuring that compute is not spent investigating unsupported hypotheses while preserving negative outcomes as first-class results(Chenet al.,2025). Within each item, execution follows a structured loop. The experiment agent first produces an execution plan by examining the current environment, searching for available models and datasets and activating relevant skills. An independent review agent evaluates the fidelity of the execution plan to the proposal and iterates until approved. The experiment agent then carries out the approved work with full tool permissions. Upon completion, dual verification combines the agent’s semantic self-assessment with deterministic harness checks on output completeness, and results are written to a pre-defined location as experimental evidence which can be used in the paper writing.

Layered Environments and Curated Skills

ML software stacks are fragile due to tightly coupled CUDA builds, compiled extensions, and interdependent library versions, and an agent that must assemble them from scratch wastes most of its effort on setup failures rather than experimentation. FARS addresses this with a layered environment: a pre-configured base provides heavy dependencies (PyTorch, vLLM, flash-attention, Transformers, etc.) that the agent cannot modify, while the agent works inside an isolated environment where it can freely install additional packages. Pre-trained model weights and common datasets are cached on shared storage, eliminating repeated downloads. A related challenge is that language models possess broad but shallow knowledge of ML tooling and conventions: they know the concepts behind distributed training or evaluation frameworks but frequently use deprecated APIs or miss known pitfalls. To bridge this gap, FARS maintains a curated skill library covering 19 categories of ML research practice, organized as structured reference files with verified code examples and troubleshooting guides. Rather than loading all skills into context at once, the agent is given only a category-level index and retrieves individual skills on demand, first scanning the index, then reading a brief description to judge relevance, and loading full content only when needed.

Managed Compute and Unified Model Access

Experiments require GPU-accelerated training and access to diverse model APIs for tasks such as LLM-based evaluation, synthetic data generation, and embedding computation. Exposing raw infrastructure interfaces to the agent risks uncontrolled resource allocation, orphaned processes, and operations untraceable to the plan. FARS instead provides high-abstraction tool interfaces through which the agent declares its intent (model, GPU count, training configuration) while the platform handles resource provisioning and lifecycle management. Each task is tagged with the corresponding plan item for attribution, and ownership isolation ensures agents can only access their own resources. All compute is automatically reclaimed upon completion or interruption. For model API access, a unified service proxy presents multiple open-source and proprietary providers behind a single compatible endpoint, supporting chat completion, embedding, reranking, and image generation with centralized authentication and usage tracking, so that the agent focuses on experimental logic rather than provider-specific integration.

Checkpoint-Based Fault Tolerance

A full experiment may span hours or days, so the stage is designed for robustness under long, unattended operation. Each task’s outputs are committed to version control immediately upon completion, so that finished results survive even if a later task or the run itself fails. Checkpoint-based recovery resumes from the last unfinished item, preserving the agent’s full context for continuity. Automated scanning runs before each commit to prevent credential leakage, and a termination handler stops all associated compute tasks to prevent orphaned processes.

3.5Writing Stage

The Writing stage transforms the complete upstream project trajectory into an academic paper. FARS separates evidence analysis from manuscript drafting: theAnalysis Agentdetermines what can be claimed from the proposal, plan, experiments, and related work, while theWriting Agentturns this evidence plan into a polished manuscript. This decomposition is intended to prevent claims from being introduced opportunistically during prose generation and to ensure that writing remains grounded in the artifacts produced by earlier stages.

Before drafting begins, the upstream materials are audited and organized into a paper blueprint. The blueprint is a structured intermediate representation that links each central claim to its supporting experiments, source artifacts, candidate figures and tables, and relevant literature. It also defines the section-level writing plan used by the downstream drafting process. In addition to organizing evidence, the Analysis Agent prepares visual materials: analytical plots are generated by executing plotting scripts over saved experimental outputs, while methodological diagrams are synthesized from detailed figure descriptions and refined with vision-language model feedback. In our implementation, the diagram synthesis step is instantiated with NanoBanana-Pro(Google,2025).

Guided by the blueprint, each section is decomposed into concrete writing tasks that inherit the blueprint’s evidence links, keeping drafting tied to the planned claims and supporting artifacts. The agent drafts the LaTeX manuscript section by section and then refines it through automatic compilation feedback, citation and reference checks, figure and table checks, numerical verification against source artifacts, and paper-level review for unsupported or overstated claims. By placing evidence organization before prose generation and tracing numerical claims, tables, and plots back to source artifacts during refinement, the Writing stage reduces hallucinated results and unsupported interpretations. The final paper is therefore maintained together with the proposal, plan, code, logs, results, and supporting evidence needed to inspect how its claims were produced.

4Public Deployment

The first public instantiation of FARS produced 166 complete research papers. The live deployment ran for 417 hours, consumed 21.6 billion model tokens, and reported a total cost of approximately​186,000\\mathdollar 186\{,\}000\(including GPU cluster usage fees and token consumption costs\)\. Amortized over the completed corpus, this corresponds to 2\.51 deployment hours, approximately 130 million model tokens, and about​1,120\mathdollar 1{,}120per paper. The generated papers were made available through a public deployment page.222https://analemma.ai/fars/The purpose of the deployment was not to present a curated set of selected successes, but to expose the output distribution of a continuously running AI-for-AI research system, including weak projects, negative results, and failure modes.

FARS was seeded with nine initial AI research directions, summarized inTable1. These directions were intended to provide starting points rather than a closed taxonomy: the Ideation agent was allowed to explore beyond them during deployment.

Table 1:Initial research directions provided to FARS at the start of the public deployment.The resulting 166 papers span 67 fine-grained topics rather than remaining within the seed set; AppendixAprovides the full topic distribution. Overall, 72 papers fall under the nine seed topics, while 94 papers occupy emergent topics discovered during autonomous exploration. The largest seed-topic clusters are Reinforcement Learning from Verifiable Rewards (13 papers), Automated Evaluation of Frontier LLMs (10), Memory in AI Agents (10), and Continual Learning (9). The largest emergent cluster is AI Safety and Alignment (13 papers), followed by Vision-Language Models (7) and Code Generation (6). This corpus is the basis for the human review study reported inSection5.

The run was streamed publicly, and intermediate artifacts including hypothesis documents, plans, code, logs, draft sections, and final papers were committed to a dedicated GitLab namespace,333https://gitlab.com/fars-aoperated by the system itself. This artifact trace made the operation of the system observable beyond finished manuscripts: observers could inspect how each hypothesis-and-validation contribution was produced. The deployment therefore exposes both the final research outputs and the process evidence needed to audit them.

Submitting selected papers to archival venues such as arXiv was handled separately from the public deployment. While the deployment page makes all completed live-run papers available, only a small subset was additionally submitted to arXiv. Before submission, each selected paper underwent a minimal human integrity review covering factual accuracy, citation validity, artifact consistency, safety-relevant claims, and explicit AI-generation disclosure; the research idea, experiments, reported results, and claimed contribution were left unchanged. These safeguards were informed by recent analyses of research-integrity risks in AI-generated scholarship(Gupta and Pruthi,2025; Zhuet al.,2025; Resniket al.,2026). AppendixClists the arXiv-submitted subset.

5Human Review Evaluation

Assessing whether FARS produces genuine research contributions requires evaluation that goes beyond automated metrics. A growing number of publicly available AI-based reviewing systems can generate surface-level feedback, but they rarely deliver the deep, expert judgment needed to assess the hypothesis quality, experimental rigor, and actual contribution of a paper. The problem is sharper for AI-generated papers, which can additionally contain fabricated or hallucinated content such as nonexistent citations, unverifiable results, or methods that diverge from their implementation. Current automated reviewers are not designed to detect these issues. We therefore recruit qualified human experts to provide deep and comprehensive reviews of FARS-generated papers, and pair their judgment with an integrity audit grounded in each paper’s source artifacts. This section describes the review protocol and reports what the resulting reviews reveal about the quality and integrity of the FARS-generated papers.

5.1Review Protocol

Reviewer Recruitment

We posted an open call for volunteer reviewers across several social media platforms and collected volunteer information through a questionnaire that recorded each applicant’s affiliation, academic title, country, primary research areas, and prior peer-review experience. The call attracted 583 applications, of which 503 (86%) reported prior reviewing experience, yielding a large and experienced pool from which to select.

Reviewer Screening

From this pool we selected 88 reviewers, screening primarily on three criteria: the match between an applicant’s research areas and the topic distribution of the generated papers (Section4), academic standing, and prior peer-review experience. All 88 selected reviewers had reviewed before, and roughly 82% (72/88) had reviewed more than 20 papers. The panel spans career stages—predominantly PhD students, alongside postdoctoral researchers, faculty, and industry research scientists—and multiple countries, with the largest contingents from China and the United States. Because the reviewers are recruited volunteers rather than an independent venue program committee, the evaluation should be read as a structured expert analysis rather than an independent conference peer-review process.

Review System and Standards

We built a dedicated review system together with a review standard tailored to AI-generated papers,444https://docs.google.com/document/d/1Lebt2dUVR4igwKO1TeFZVgTA0_2aDDXRB-hEyBNzgpM/edit?usp=sharingmodeled on the ICLR review form but adapted to FARS’s hypothesis-and-validation outputs. The standard retains the ICLR scoring dimensions: overall rating, soundness, presentation, contribution, and reviewer confidence, and adds FARS-specific principles: short papers are not penalized for length, non-standard structure is permitted, and a well-motivated hypothesis that is rigorously falsified is treated as a first-class contribution rather than a failed paper. Because AI-generated papers cannot be reviewed under the usual assumption of good faith, the standard further introduces anAI Integrity Audit. To support it, reviewers are given access to each paper’s full artifact trace: code repository, experiment logs, and raw output data. They are required to verify the manuscript’s claims against these sources along eight integrity failure modes: hallucinated citations, fabricated or unverifiable results, spurious novelty, hallucinated methods or baselines, mathematical and logical errors, internal inconsistencies, experimental-design pathologies, and language pathologies. Details for each mode are provided inAppendixB.

Review Form and Scoring

Each review follows a fixed structure: summary, strengths, weaknesses, optional suggestions, an ethics and reproducibility assessment, the integrity audit, and a mandatory disclosure of any LLM assistance used while writing the review. The scoring is aligned with ICLR 2026: an overall rating on the discrete scale{0,2,4,6,8,10}\{0,2,4,6,8,10\}, ranging from strong reject (0) to strong accept (10) with 6 marking the acceptance threshold, together with sub-scores for soundness, presentation, and contribution on a 1–4 scale and reviewer confidence on a 1–5 scale; integrity issues are flagged through a separate field. While the numeric scale is identical to ICLR, the rubric is calibrated for short, focused papers, so that a high score does not require exhaustive long-paper breadth but does require a clear, bounded central claim supported by the available evidence.

Assignment

Given the total number of papers and the size of the reviewer panel, each reviewer was asked to review roughly 5–10 papers. Rather than assigning papers centrally, we let reviewers self-claim papers on the review system according to their own research areas and interests, with each paper claimable by at most three reviewers.

5.2Quantitative Review Analysis

Reviewer Participation

The review cycle ran from March 21 to April 12, 2026. Before analysis, we audited the 284 submitted reviews and found three paper-review mismatches: one review assigned to FA0027 discussed FA0045 instead, one review assigned to FA0045 duplicated an FA0020 review, and one review assigned to FA0297 referred to a paper outside FARS’s papers. We corrected these mismatches by reassigning the FA0027 review to FA0045 and excluding the two invalid reviews, leaving 282 valid reviews for analysis. Of the 88 selected reviewers, 55 submitted at least one review, together producing the 282 valid reviews analyzed here across 140 of the 166 generated papers: 45 papers (27.1%) received one review, 48 (28.9%) received two, and 47 (28.3%) received three, while the remaining 26 (15.7%) papers were not reviewed and are excluded from all statistics below.

Overall Rating Distribution

Figure2reports the score distribution over all 282 reviews. The overall ratings are left-skewed and concentrated below the acceptance threshold, with a mean of 3.17 and a median of 2.0. At the review level, 8.2% of reviews assign a rating of 0, 42.9% a 2, 31.2% a 4, and 17.7% a 6; no review assigned an 8 or a 10, so 6 is the highest score observed in the corpus. Although roughly half of all reviews (48.9%) place a paper at 4 or above, only 17.7% reach the ICLR acceptance threshold of 6. Aggregating reviews to the paper level gives a more conservative picture (Figure3): among the 140 reviewed papers (mean 3.23, median 3.0), 39.3% attain a mean rating of at least 4, 31.4% are rated≥\geq4 by every reviewer, and 69.3% receive at least one rating of 4 or higher, yet only 11.4% (16 papers) reach a mean rating at or above the acceptance threshold. This last proportion should be interpreted with care, as it is inflated by papers with sparse reviews: 14 of these 16 papers were assessed by a single reviewer, and only 2 of the 95 papers with two or more reviews attain a mean rating of 6. Overall, the corpus exhibits substantial variation in assessed quality alongside a comparatively thin tail of strong outputs.

Figure 2:Distribution of human review scores across 282 reviews of 140 FARS-generated papers: (a) overall rating, (b–e) sub-scores for soundness, presentation, contribution, and reviewer confidence, and (f) a box-plot summary across all dimensions.Figure 3:Paper-level distribution of mean overall rating across the 140 reviewed papers, binned by integer-width intervals (rightmost bar is exactly 6).

Sub-score Distributions

The four sub-scores (Figure2b–e) show where quality concentrates. Soundness (mean 2.15) and contribution (mean 2.00) sit at the lower end. More than half of reviews assign contribution a 2 (52.8%) and only 23.8% assign a 3. By contrast, presentation is comparatively strong (mean 2.46), with 48.9% of reviews assigning a 3. Reviewer confidence is high (mean 3.78) and clusters at 4 (54.3%), indicating that reviewers were generally comfortable with their assessments. Top sub-scores are rare: contribution never reaches 4, and soundness and presentation reach 4 in only 1 and 6 reviews, respectively.

Score Predictors

To identify which dimensions track overall quality, we compute Pearson correlations between each quality sub-score and the overall rating (Table2) and compare sub-score means across rating tiers (Figure4a). Contribution is the strongest predictor (r=0.743r=0.743), followed by soundness (r=0.652r=0.652) and presentation (r=0.451r=0.451), all highly significant. This ordering suggests that the main bottleneck is not manuscript polish but whether a paper delivers a meaningful, well-supported contribution. Reviewer confidence measures certainty rather than quality, so we do not treat it as a predictor. We note that confidence runs counter to rating, being higher on low-rated papers than on high-rated ones (mean 4.30 at rating 0 vs. 3.46 at rating 6; Figure4a), indicating that reviewers are more certain when identifying weak work than when endorsing strong work.

Table 2:Pearson correlations between the three quality sub-scores and overall rating. Contribution shows the strongest association with overall quality.Figure 4:Quality stratification and integrity findings for FARS papers. (a) Sub-score means by rating tier show monotonic improvement across soundness, presentation, and contribution, while confidence decreases for higher ratings. (b) Integrity issue prevalence shows a strong inverse relationship with quality, from 43.5% at rating 0 to 2.0% at rating 6. Error bars in (a) denote standard errors of the mean; bars in (b) show point estimates with 95% Wilson confidence intervals; the number of reviewsnnin each rating tier is annotated on the horizontal axis.

Integrity Audit Outcomes

The AI Integrity Audit flagged problems in 47 of 282 reviews (16.7%), affecting 39 of 140 reviewed papers (27.9%). These flags concentrate almost entirely in low-rated papers (Figure4b): 43.5% of rating-0 reviews report integrity issues, falling to 22.3% at rating 2, 10.2% at rating 4, and just 2.0% at rating 6. The steep gradient indicates that integrity violations behave as a distinct failure mode concentrated in the weakest outputs rather than a smoothly varying quality dimension, consistent with benchmark evidence that open-ended ML research agents can generate plausible papers while struggling to validate experimental results(Chenet al.,2025).

Reviewer Agreement

For the 95 papers with two or more reviews, we quantify how far reviewers’ overall ratings diverge using the per-paper spread (highest minus lowest rating). Since adjacent rating levels differ by two points, a spread of 2 corresponds to one level of disagreement. Reviewers assign identical ratings for 28.4% of these papers, differ by a single level (spread of 2) for 47.4%, by two levels (spread of 4) for 20.0%, and by the maximum possible spread of 6 (e.g., 0 versus 6) for 4.2%. The mean spread is 2.00 points—about one rating level. This degree of divergence is in line with the substantial reviewer disagreement documented for top machine-learning venues: in the NeurIPS 2021 consistency experiment, the average paper scores of two independent committees correlated at onlyr≈0.58r\approx 0.58and the committees disagreed on accept/reject for 23% of papers(Beygelzimeret al.,2023), while a large-scale analysis of ICLR 2024/2025 reports a per-paper reviewer disagreement of roughly1.51.5points (mean pairwise score difference before rebuttal) on a comparable ten-point scale(Kargaranet al.,2025). Our spread is thus of the same order, situating the FARS review within known peer-review variability rather than indicating anomalously noisy judgments. Because the reviewers are recruited volunteers rather than an independent venue program committee, these agreement statistics should be read as evidence of structured-review consistency for this deployment rather than as a substitute for independent venue review.

LLM-Assisted Reviewing

In their mandatory disclosures, 74.5% of reviews (210/282) reported using an LLM while preparing the review (GPT/ChatGPT: 81, Claude: 40, Gemini: 30), predominantly for language polishing and summarization rather than substantive evaluation. We record this because LLM-assisted reviewing may influence phrasing, consistency, and attention allocation even when the final judgments and scores remain human-authored.

5.3Qualitative Findings

The scores inSection5.2say how good the papers are; the free text of the reviews sayswhy. To characterize what reviewers reward and penalize, we label every review’s strengths and weaknesses against the category taxonomy ofKargaranet al.(2025)and map reported integrity problems onto the eight integrity failure modes of the AI Integrity Audit (Sections5.1andB), using a large language model annotator (Claude Opus 4.6, temperature 0). Each strength or weakness point is assigned a main category and subcategory; the percentages below are the share of the 282 reviews whose text was assigned a given category, and a single review may map to several.

What Reviewers Reward

Strengths cluster on execution rather than originality. The most frequently praised dimensions are methodological soundness (63.5% of reviews), experimental design (61.0%), and clear motivation (57.4%), followed by results (48.9%), while novelty is rewarded least often (31.9%). At the subcategory level the dominant strengths are a well-motivated problem (48.2%), an elegant and efficient method (29.1%), thorough ablation studies (25.2%), and a clear and correct interpretation of results (23.4%). Notably, 18.1% of reviews credit a detailed error or failure analysis—a direct consequence of the FARS design choice to surface negative and null results rather than suppress them. The praise profile sharpens with quality (Table3): recognition of novelty rises monotonically from 21.5% of low-rated reviews to 54.0% of high-rated ones, with the same upward trend for experiments (52.8%→\to78.0%) and results (38.9%→\to64.0%), indicating that what distinguishes a strong FARS paper is not competent execution alone but a contribution reviewers judge to be genuinely new.

Recurring Weaknesses

Criticism is dominated by evidence quality. Experimental weaknesses appear in 92.9% of reviews and are essentially universal among the highest-rated papers (100%), confirming that even the best FARS outputs leave reviewers wanting stronger evidence. The leading complaints are narrow evaluation—too few datasets or limited domain coverage (46.8%) and poor generalization across settings (46.8%)—followed by missing ablations (31.9%) and weak baselines (30.1%). Part of this concentration reflects format rather than outright failure: the current FARS is configured to produce short, focused papers, whose experimental sections are necessarily less extensive than those of full-length submissions, and the review rubric was calibrated so that brevity alone is not penalized (Section5.1). Even under this calibration, however, experimental concerns remain the single most prevalent weakness, indicating that the problem is often not breadth per se butsufficiency—a number of papers fail to support their central claim even with the small but well-targeted set of experiments that a short paper warrants. Beyond experiments, reviewers frequently question methodological soundness (63.1%; chiefly weak theoretical justification, 27.3%, and unrealistic assumptions, 22.3%) and results (58.2%; contradictory or unexplained findings, 22.3%, marginal gains, 21.3%, and overinterpretation, 17.7%). Weaknesses follow the opposite quality gradient to strengths: novelty criticism falls from 42.4% of low-rated reviews to 18.0% of high-rated ones and writing complaints from 24.3% to 12.0%, while experimental criticism stays high across all tiers (Table3). This pattern—low-rated papers penalized for experimental and methodological flaws, high-rated papers set apart by novelty and soundness—mirrors the rating-tier gradient reported for human-written ICLR submissions(Kargaranet al.,2025), placing FARS outputs within familiar review dynamics rather than a separate regime. It also echoes the quantitative scores: FARS reliably produces a focused, well-motivated claim, but its supporting evidence is too narrow for reviewers to consider the result established.

Table 3:Reviewer-identified strengths and weaknesses by category, as a percentage of the 282 reviews whose text was assigned each category (multi-label).Allis the overall rate;Lo/Hirestrict to reviews with overall rating≤2\leq 2(n=144n{=}144) and≥6\geq 6(n=50n{=}50). Strengths rise and weaknesses fall with rating, except experimental concerns, which persist across all tiers.

Integrity Failure Modes

The AI Integrity Audit exposes problems that ordinary peer review cannot, because reviewers cross-checked every claim against each paper’s code, logs, and raw outputs. Whereas 16.7% of reviews recorded aformalintegrity violation (Section5.2), free-text analysis shows that 44.0% (124/282) describe at least one integrity failure mode (Table4). The most common are experimental-design pathologies (22.7%; e.g. confounded comparisons or suppressed unfavorable runs) and internal-consistency failures (13.1%; contradictory numbers across abstract, tables, and figures). Crucially, the failure modes differ sharply in how often a mention is escalated to a formal violation: hallucinated methods or baselines (20 of 22 mentions flagged), hallucinated citations (13/16), and fabricated results (16/21) are almost always treated as genuine integrity breaches, whereas language and reasoning pathologies (4 of 31) and many design concerns are recorded as weaknesses rather than violations—precisely the distinction the audit asks reviewers to draw between fabrication and ordinary methodological limitation. That the gravest, code-verifiable failures (code–paper mismatch, untraceable numbers, nonexistent references) are both present and reliably caught underscores why artifact-grounded human review remains necessary for deployment-scale automated research: these are exactly the errors that plausible-looking generated papers conceal and that score- or text-only evaluation would miss(Chenet al.,2025).

Table 4:Integrity failure modes across the 282 reviews. ”Mentioned” counts reviews whose weakness or audit text describes the failure mode; ”Flagged” counts how many of those reviews also recorded a formal integrity violation. Severe, code-verifiable failures are almost always escalated to violations, while softer pathologies are usually noted only as weaknesses.Taken together, the qualitative analysis converges with the quantitative scores: reviewers consistently reward FARS for well-motivated problems and competent execution, the dominant limitation is the sufficiency of experimental evidence rather than presentation, and the artifact-grounded audit reliably surfaces the code-verifiable failure modes that distinguish AI-generated papers from human-written ones.Table5lists representative reviewed papers that illustrate these patterns end to end, from a rewarded falsifiable negative result (FA0328) and a controlled compression study that reveals an accuracy–perplexity tradeoff (FA0029) to a well-posed paper that one reviewer flagged for artifact-level integrity concerns (FA0064). To support scrutiny and follow-up analysis, we release all reviewed papers together with their reviews555https://huggingface.co/datasets/analemma-ai/fars-a-reviews. Before releasing, we contacted all reviewers who participated in the review process, and since some of them preferred not to open-source their own reviews, we withheld these reviews from the public release; this affects six reviews, one each from FA0004, FA0028, FA0050, FA0052, FA0145, and FA0194, while all other reviews for these papers remain available.

Table 5:Representative reviewed FARS outputs from the public deployment. Ratings are individual overall review scores on the ICLR scale{0,2,4,6,8,10}\{0,2,4,6,8,10\}; S/P/C report mean soundness, presentation, and contribution sub-scores.

6Comparison with Prior Research Systems

The human evaluation inSection5characterizes the quality of FARS outputs in isolation; it does not place them relative to other autonomous research systems. A like-for-like human comparison is impractical, because the human panel reviewed only FARS papers and the prior systems release only small numbers of generated papers. To obtain a common yardstick for paper quality, we instead apply a single automated reviewer, the Stanford Agentic Reviewer (SAR)666https://paperreview.ai/, uniformly to the public paper outputs of every system, including FARS. Concretely, we collect the generated papers released by six prior end-to-end research systems: The AI Scientist(Luet al.,2024)777The AI Scientist releases its generated papers in a public Google Drive grouped by base foundation model (Claude 3.5 Sonnet, GPT-4o, DeepSeek Coder, and Llama-3.1 405B). We use only the Claude 3.5 Sonnet subset, which the authors identify as their highest-quality outputs and explicitly recommend for qualitative inspection., The AI Scientist-v2(Yamadaet al.,2025), CycleResearcher(Wenget al.,2025a), DeepScientist(Wenget al.,2025b), AutoResearchClaw(Liuet al.,2026), and ScientistOne(Menget al.,2026). We then score these papers, together with the FARS-generated papers, using this same reviewer under an ICLR-style rubric that returns an overall rating on a0–1010scale. Because the identical reviewer and rubric are applied to all systems, the resulting scores are comparable across systems even though their absolute calibration differs from human review. The Stanford Agentic Reviewer may fail to produce a review. Issues arise when papers exceed the input size limit or errors occur during review generation. We present statistics exclusively for papers that receive valid reviews across all systems including FARS. Accordingly, the paper counts inTable6represent only successfully reviewed subsets, not the full set of released papers from each system.

FARS attains the highest automated rating

Under this shared reviewer (Table6), FARS achieves the highest mean rating,5.005.00over165165papers, ahead of every prior system: the strongest baselines, DeepScientist (4.134.13) and ScientistOne (4.004.00), trail by roughly one point, and the pooled mean of all110110non-FARS papers is2.742.74, so FARS leads the field by about+2.3+2.3points. FARS is also the only system with papers reaching the automated acceptance threshold (1.8%1.8\%scored≥6\geq 6), whereas no baseline paper does. This advantage holds despite FARS being scored on by far the largest and least curated corpus, whereas several baselines contribute only a handful of papers.

Automated scores are a relative signal, not an acceptance rate

The automated reviewer is on average more lenient than the expert panel: on the same FARS corpus it assigns a mean of5.005.00, against the human mean of3.233.23(Section5). The absolute automated scores should therefore be read as a relative ranking across systems rather than as a probability of acceptance, and the stricter human evaluation inSection5remains the more conservative measure of FARS quality. Reassuringly, the two measures agree in direction: FARS leads under the automated reviewer, and its human ratings exceed the automated scores that most prior systems obtain.

Scope and caveats

This comparison is for reference only and does not constitute a controlled benchmark. The baseline corpora have small and unbalanced sample sizes ranging from 3 to 71. Calculated means for individual systems contain noise, especially those with just three papers. The systems address diverse problem domains and benchmarks rather than a consistent task set, and each adopted its own criteria for paper release. Finally, the automated reviewer is itself an LLM-based proxy with its own biases. We therefore treatTable6as evidence that FARS outputs are competitive with or stronger than those of prior systems under a uniform automated standard, while relying on the human study for the more demanding, integrity-aware assessment.

Table 6:Cross-system comparison under the Stanford Agentic Reviewer, applied uniformly to each system’s public paper outputs. Ratings are on a0–1010scale;≥6\geq 6is the share of papers reaching the automated acceptance threshold. FARS additionally underwent expert human review (last row, fromSection5), shown for calibration; the human rating is markedly stricter than the automated one.

7Conclusion

We introduced FARS, a fully automated AI-for-AI research system that turns research directions into complete papers through a shared auditable workspace. Its first public deployment produced 166 papers across 67 fine-grained AI/ML topics while preserving the proposals, code, logs, results, and manuscripts behind each output. This corpus lets us evaluate automated research as a deployment-scale distribution rather than a selected set of demonstrations. Across 282 expert reviews of 140 papers, FARS produces reviewable and occasionally strong research artifacts; a uniform automated-review comparison further places FARS competitively relative to prior systems. Together, these results suggest that end-to-end automated research systems can already generate broad, inspectable research output at scale.

The same evidence also defines the limits of current automated research. The strong-output tail remains thin, and the dominant bottlenecks are not surface presentation but contribution, experimental sufficiency, and claim faithfulness. Progress will therefore depend less on producing more plausible manuscripts than on building agents that design decisive experiments, preserve provenance, audit claims against artifacts, learn from failed trajectories, and withhold outputs that are not scientifically supported. FARS is not evidence that automated research is solved; it is an early systems demonstration of what becomes possible when generation, experimentation, provenance, and evaluation are treated as a single auditable process.

References

• Effective harnesses for long-running agents.Note:https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agentsAnthropic Engineering BlogCited by:§3.3.
• A. Asai, J. He, R. Shao, W. Shi, A. Singh, J. C. Chang, K. Lo, L. Soldaini, S. Feldman, M. D’arcy, D. Wadden, M. Latzke, M. Tian, P. Ji, S. Liu, H. Tong, B. Wu, Y. Xiong, L. Zettlemoyer, G. Neubig, D. Weld, D. Downey, W. Yih, P. W. Koh, and H. Hajishirzi (2024)OpenScholar: synthesizing scientific literature with retrieval-augmented lms.External Links:2411.14199,LinkCited by:§1.
• J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang (2025)ResearchAgent: iterative research idea generation over scientific literature with large language models.External Links:2404.07738,LinkCited by:§1.
• A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan (2023)Has the machine learning review process become more arbitrary as the field has grown? the neurips 2021 consistency experiment.External Links:2306.03262,LinkCited by:§5.2.
• H. Chen, M. Xiong, Y. Lu, W. Han, A. Deng, Y. He, J. Wu, Y. Li, Y. Liu, and B. Hooi (2025)MLR-bench: evaluating ai agents on open-ended machine learning research.External Links:2505.19955,LinkCited by:§2,§3.4,§5.2,§5.3.
• A. E. Ghareeb, B. Chang, L. Mitchener, A. Yiu, C. J. Szostkiewicz, D. Shved, G. J. Gyimesi, J. M. Laurent, S. M. Wright, M. T. Razzak, A. D. White, S. C. Finnemann, M. M. Hinks, and S. G. Rodriques (2026)A multi-agent system for automating scientific discovery.Nature.External Links:Document,LinkCited by:§2.
• Google (2025)Nano Banana Pro: Gemini 3 Pro Image model from Google DeepMind.Note:https://blog.google/innovation-and-ai/products/nano-banana-pro/Accessed: 2026-03-19Cited by:§3.5.
• J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, and J. Guo (2025)A survey on llm-as-a-judge.External Links:2411.15594,LinkCited by:§2.
• T. Gupta and D. Pruthi (2025)All that glitters is not novel: plagiarism in AI generated research.InProceedings of ACL 2025,Cited by:§2,§4.
• Q. Huang, J. Vora, P. Liang, and J. Leskovec (2024)MLAgentBench: evaluating language agents on machine learning experimentation.External Links:2310.03302,LinkCited by:§2.
• Intology (2025)Zochi technical report.arXiv.Cited by:§1,§2.
• P. A. Jansen, O. Tafjord, M. Radensky, P. Siangliulue, T. Hope, B. Dalvi, B. P. Majumder, D. S. Weld, and P. Clark (2025)CodeScientist: end-to-end semi-automated scientific discovery with code-based experimentation.InFindings of the Association for Computational Linguistics: ACL 2025,Cited by:§2.
• A. H. Kargaran, N. Nikeghbal, J. Yang, and N. Ousidhoum (2025)Insights from the iclr peer review and rebuttal process.External Links:2511.15462,LinkCited by:§5.2,§5.3,§5.3.
• P. T. J. Kon, J. Liu, X. Zhu, Q. Ding, J. Peng, J. Xing, Y. Huang, Y. Qiu, J. Srinivasa, M. Lee, M. Chowdhury, M. Zaharia, and A. Chen (2025)EXP-bench: can ai conduct ai research experiments?.External Links:2505.24785,LinkCited by:§2.
• R. Li, T. Patel, Q. Wang, and X. Du (2025)MLR-copilot: autonomous machine learning research based on large language models agents.External Links:2408.14033,LinkCited by:§1.
• Y. Li, C. Shao, X. Liu, R. Zhao, P. Liu, H. Su, Z. Chen, Q. Yang, A. Xu, Y. Fang, Q. Zeng, T. Li, J. Xu, F. Xu, Y. Li, and T. Liu (2026)AutoSOTA: an end-to-end automated research system for state-of-the-art AI model discovery.arXiv preprint arXiv:2604.05550.Cited by:§2.
• W. Liang, Y. Zhang, H. Cao, B. Wang, D. Ding, X. Yang, K. Vodrahalli, S. He, D. Smith, Y. Yin, D. McFarland, and J. Zou (2023)Can large language models provide useful feedback on research papers? a large-scale empirical analysis.External Links:2310.01783,LinkCited by:§2.
• J. Liu, S. Qiu, M. Li, B. Li, H. Ji, S. Han, X. Ye, P. Xia, Z. Dong, M. Chen, C. Zhang, L. Zhang, G. Chen, H. Tu, X. Yang, L. Feng, X. Zhao, H. Chen, J. Zhou, X. Wang, W. Zhang, H. Zhu, Y. Li, J. Mei, H. Fei, J. Zhang, L. Li, L. Zhang, Y. Zhou, S. Wang, C. Xiong, J. Zou, Z. Zheng, C. Xie, M. Ding, and H. Yao (2026)AutoResearchClaw: self-reinforcing autonomous research with human-ai collaboration.External Links:2605.20025,LinkCited by:§2,Table 6,§6.
• Z. Liu, K. Liu, Y. Zhu, X. Lei, Z. Yang, Z. Zhang, P. Li, and Y. Liu (2024)AIGS: generating science from ai-powered automated falsification.External Links:2411.11910,LinkCited by:§1,§2.
• C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024)The ai scientist: towards fully automated open-ended scientific discovery.External Links:2408.06292,LinkCited by:§1,§2,Table 6,§6.
• R. Meng, B. D. Mishra, J. Chen, C. Li, P. Goyal, M. Parmar, Y. Song, Y. Song, R. Sinha, P. Ranganathan, B. Gokturk, J. Yoon, and T. Pfister (2026)ScientistOne: towards human-level autonomous research via chain-of-evidence.External Links:2605.26340,LinkCited by:§2,Table 6,§6.
• D. B. Resnik, M. Hosseini, and R. Hauswald (2026)Autonomous artificial intelligence, scientific research, and human values.AI and Ethics.External Links:DocumentCited by:§2,§4.
• S. Schmidgall, Y. Su, Z. Wang, X. Sun, J. Wu, X. Yu, J. Liu, M. Moor, Z. Liu, and E. Barsoum (2025)Agent laboratory: using llm agents as research assistants.External Links:2501.04227,LinkCited by:§1.
• C. Si, T. Hashimoto, and D. Yang (2025)The ideation-execution gap: execution outcomes of LLM-generated versus human research ideas.arXiv preprint arXiv:2506.20803.Cited by:§2.
• C. Si, D. Yang, and T. Hashimoto (2024)Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.External Links:2409.04109,LinkCited by:§2.
• G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan (2025)PaperBench: evaluating AI’s ability to replicate AI research.InProceedings of the 42nd International Conference on Machine Learning (ICML),Cited by:§2.
• J. Tang, L. Xia, Z. Li, and C. Huang (2025)AI-researcher: autonomous scientific innovation.External Links:2505.18705,LinkCited by:§1,§2,§2.
• Y. Weng, M. Zhu, G. Bao, H. Zhang, J. Wang, Y. Zhang, and L. Yang (2025a)CycleResearcher: improving automated research via automated review.InThe Thirteenth International Conference on Learning Representations,External Links:LinkCited by:§1,§2,Table 6,§6.
• Y. Weng, M. Zhu, Q. Xie, Q. Sun, Z. Lin, S. Liu, and Y. Zhang (2025b)DeepScientist: advancing frontier-pushing scientific findings progressively.External Links:2509.26603,LinkCited by:§1,§2,§2,Table 6,§6.
• Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha (2025)The ai scientist-v2: workshop-level automated scientific discovery via agentic tree search.External Links:2504.08066,LinkCited by:§1,§2,Table 6,§6.
• R. Yang, Y. Li, and S. Li (2026)ARIS: autonomous research via adversarial multi-agent collaboration.External Links:2605.03042,LinkCited by:§2.
• J. Yu, Z. Ding, J. Tan, K. Luo, Z. Weng, C. Gong, L. Zeng, R. Cui, C. Han, Q. Sun, Z. Wu, Y. Lan, and X. Li (2024)Automated peer reviewing in paper sea: standardization, evaluation, and analysis.External Links:2407.12857,LinkCited by:§2.
• L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023)Judging llm-as-a-judge with mt-bench and chatbot arena.External Links:2306.05685,LinkCited by:§2.
• K. Zhu, J. Zhang, Z. Qi,et al.(2025)SafeScientist: toward risk-aware scientific discoveries by LLM agents.arXiv preprint arXiv:2505.23559.Cited by:§2,§4.

Appendix ADeployment Topic Distribution

Figure5groups the 166 generated papers by whether they fall under the nine seed topics provided at launch or under emergent topics discovered during autonomous exploration.

Figure 5:Topic distribution of the 166 FARS-generated papers. Blue bars indicate the nine seed topics initially provided to FARS; gray bars indicate emergent topics explored autonomously during deployment.

Appendix BIntegrity Failure Modes

The AI Integrity Audit in our review standard (Section5.1) asks reviewers to verify each manuscript against its own source artifacts—the code repository, experiment logs, and raw output data—along eight integrity failure modes that AI-generated papers are prone to. We summarize each mode below; the full audit checklist, risk levels, and calibrated worked examples are given in the review standard. A guiding principle throughout is the distinction between a genuine integrity violation and an ordinary methodological weakness: only deceptive or artifact-contradicting cases (fabrication, misattribution, hidden manipulation) are treated as violations, whereas benign metadata artifacts—such as garbled author names for a paper that nonetheless exists, or sub-rounding numerical differences—are noted but not escalated.

1. 1.Hallucinated citations and references.References whose title, venue, and authors look plausible but that do not exist, or real papers cited in support of a claim they do not actually make. Author-name or year errors for an otherwise correctly identified paper are bibliography artifacts rather than violations.
2. 2.Fabricated or unverifiable experimental results.Reported tables, curves, or statistics that contradict the recorded logs, or that cannot be traced to any artifact at all; also selective reporting of a more favorable configuration than the one the paper describes.
3. 3.Spurious novelty and knowledge contamination.Reproducing established results while claiming novelty, overstating novelty relative to existing work, or asserting a literature gap that does not exist.
4. 4.Hallucinated methods and baselines.Methods or baselines that do not exist in the literature, or whose description in the paper materially diverges from the implemented code (e.g. the paper describes one model while the code runs another).
5. 5.Mathematical and logical errors.Proofs or derivations that read fluently but contain material flaws—unstated assumptions, unjustified steps, circular reasoning, or misapplied theorems—that invalidate the claim.
6. 6.Internal inconsistencies.Mutually contradictory statements across the abstract, body, tables, and figures, most often differing numbers reported for the same quantity, sometimes because values from different experiment runs are mixed without disclosure.
7. 7.Experimental-design pathologies.Designs that appear rigorous but bias the outcome: data leakage, compute- or data-unequal comparisons, cherry-picked seeds, post-hoc hypotheses, or suppressed unfavorable runs.
8. 8.Language and reasoning pathologies.Writing patterns that signal shallow understanding—stacked hedging, vacuous generalization, confident incorrectness, narrative in place of evidence, and templated phrasing—which are treated as presentation weaknesses unless coupled with verifiable fabrication from the modes above.

Appendix CarXiv-Submitted FARS Papers

Table7lists the FARS deployment outputs that were additionally submitted to arXiv. These submissions are provenance-preserving archival versions of the original FARS outputs rather than human-improved research variants: the research ideas, experiments, reported results, and claimed contributions remain those produced during the deployment.

Table 7:FARS papers submitted to arXiv after minimal human integrity review.