Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Summary
The Discovery Certification Protocol audits AI research agents through executable recovery tests and controlled audits to validate outcomes and prevent false discovery claims, ensuring reproducibility in research.
View Cached Full Text
Cached at: 09/10/26, 06:10 AM
Paper page - Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
Source: https://huggingface.co/papers/2609.09219
Abstract
The Discovery Certification Protocol validates AI research agent outcomes through executable recovery tests, controlled audits, and deterministic verification with finite-sample recovery bounds.
AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. TheDiscovery Certification Protocol(DCP) turns claims about these results into executablerecovery and feedback tests. Gate 1 validates useful improvement onsealed evaluation. Gate 2 gives matched agents the registered starting information and observed Web content while withholding the target research history. Every valid method reaching the numerical target supplies a recovery witness and triggers theCore veto. DCP Core requires adequate controls, zero observed recoveries, and afinite-sample boundon recovery in one fresh registered episode. Optional Gate 3 measures the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint. DCP Evidence adds this effect after independentnull calibrationand a registered effect margin. Two controlled audits exercise the complete protocol inSQLite optimizationandvirtual catalyst controlunder different models. Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468. Each paired study yielded 30 truthful recoveries and zero neutral recoveries, with passing 60-pair null studies. Additional cases exercise Core, recovered, and audit-incomplete decisions. A deterministic,LLM-free verifierreproduces the decisions fromfrozen evidence. DCP provides a common evidence language for useful outcomes, alternative routes, and feedback effects across AI research.
View arXiv pageView PDFProject pageGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.09219
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.09219 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.09219 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.09219 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DiscoveryAgent: A Framework for Scientist-centric Agentic AI
DiscoveryAgent is a scientist-centric agentic AI framework combining autonomous investigation with scientist-directed hypothesis steering and audit agents that trace claims and reproduce analyses. Across predictive modeling tasks and three domain case studies, it achieved state-of-the-art results while scientists collaborating with audit agents found 14x more valid errors than manual review.
When there is no answer key for scientific discovery how do we verify an ai hypothesis
Discusses the challenge of verifying AI-generated hypotheses in scientific discovery where no ground truth exists, and presents Apodex's multi-agent approach with independent verifier agents as a solution.
Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research
An arXiv paper stress-tests defenses against AI research sabotage via a red-team/blue-team game, finding human auditors barely outperform chance (AUROC 0.59) at detecting sabotaged experiments, while frontier models perform somewhat better.
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence
This paper introduces Apodex Discovery, a framework and reality benchmark (TRACES) for evaluating and building 'discoverative AI'—AI systems that investigate open-ended real-world problems. The proposed heavy-duty solvers outperform published state-of-the-art on AAV capsid design and improve drug repurposing prediction scores.
Where Scientific Search Agents Fail: Decision-Checkpoint Auditing of Exposure and Inspection Attempts
This paper introduces decision-checkpoint auditing, a trajectory-level measurement method that classifies scientific-search agent failures into target exposure, inspection, and answer correctness stages, finding that keyword search and read-first policies achieve identical 24.6% accuracy despite differing failure distributions on AutoResearchBench Deep.