How much of AutoResearch is research, and how much is search?[D]
Summary
A part-time AutoResearch practitioner questions how much of autonomous research agent evaluation is genuine scientific inquiry versus optimization search within human-defined problem spaces, and asks what agents would need to demonstrate real research judgment beyond score improvement.
Similar Articles
How Far Are We From True Auto-Research?
This paper introduces ResearchArena, a scaffold for evaluating auto-research agents, and finds that while agent-generated papers appear competitive under manuscript-only review, artifact-aware review reveals severe failures in experimental rigor, with no paper meeting top-tier acceptance standards.
AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery
This survey examines the emerging field of AI-powered research automation (AutoResearch), analyzing how AI systems are moving from isolated task assistance to full workflow-level scientific discovery. It defines a spectrum from human-steered 'Vibe Research' to AI-led systems, and proposes five evaluation dimensions for scientific credibility.
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
This paper introduces AutoResearchEval, an evaluation framework for AI agents in automated scientific research, revealing a critical lack of metacognitive abilities as a recurring failure pattern across models.
AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery
A survey paper examining the transition of AI from task-specific assistants to workflow-level research automators, defining AutoResearch as the spectrum of AI-powered scientific workflow automation and analyzing challenges in autonomy, reproducibility, and accountability.
@sunweiwei12: What do auto-research agents actually do differently from top human researchers? Despite all the excitement around auto…
TraceML is a tool for trajectory-level analysis of ML research agents, comparing Codex agent runs with Kaggle Grandmaster notebooks and finding that human experts explore broader strategy sets while agents collapse into narrow loops. Distilling human research skills into a ~1,000-token planning prompt lifted Codex scores on five of seven competitions; the work will be presented at NeurIPS 2026.