Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Hugging Face Daily Papers Papers

Summary

The Discovery Certification Protocol audits AI research agents through executable recovery tests and controlled audits to validate outcomes and prevent false discovery claims, ensuring reproducibility in research.

AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. The Discovery Certification Protocol (DCP) turns claims about these results into executable recovery and feedback tests. Gate 1 validates useful improvement on sealed evaluation. Gate 2 gives matched agents the registered starting information and observed Web content while withholding the target research history. Every valid method reaching the numerical target supplies a recovery witness and triggers the Core veto. DCP Core requires adequate controls, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode. Optional Gate 3 measures the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint. DCP Evidence adds this effect after independent null calibration and a registered effect margin. Two controlled audits exercise the complete protocol in SQLite optimization and virtual catalyst control under different models. Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468. Each paired study yielded 30 truthful recoveries and zero neutral recoveries, with passing 60-pair null studies. Additional cases exercise Core, recovered, and audit-incomplete decisions. A deterministic, LLM-free verifier reproduces the decisions from frozen evidence. DCP provides a common evidence language for useful outcomes, alternative routes, and feedback effects across AI research.
Original Article
View Cached Full Text

Cached at: 09/10/26, 06:10 AM

Paper page - Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Source: https://huggingface.co/papers/2609.09219

Abstract

The Discovery Certification Protocol validates AI research agent outcomes through executable recovery tests, controlled audits, and deterministic verification with finite-sample recovery bounds.

AI research agents combine prior knowledge, public sources, and experimental feedback to produce useful results. TheDiscovery Certification Protocol(DCP) turns claims about these results into executablerecovery and feedback tests. Gate 1 validates useful improvement onsealed evaluation. Gate 2 gives matched agents the registered starting information and observed Web content while withholding the target research history. Every valid method reaching the numerical target supplies a recovery witness and triggers theCore veto. DCP Core requires adequate controls, zero observed recoveries, and afinite-sample boundon recovery in one fresh registered episode. Optional Gate 3 measures the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint. DCP Evidence adds this effect after independentnull calibrationand a registered effect margin. Two controlled audits exercise the complete protocol inSQLite optimizationandvirtual catalyst controlunder different models. Each produced zero recoveries in 96 episodes, with an upper bound of 0.0468. Each paired study yielded 30 truthful recoveries and zero neutral recoveries, with passing 60-pair null studies. Additional cases exercise Core, recovered, and audit-incomplete decisions. A deterministic,LLM-free verifierreproduces the decisions fromfrozen evidence. DCP provides a common evidence language for useful outcomes, alternative routes, and feedback effects across AI research.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.09219

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.09219 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.09219 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.09219 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DiscoveryAgent: A Framework for Scientist-centric Agentic AI

arXiv cs.AI

DiscoveryAgent is a scientist-centric agentic AI framework combining autonomous investigation with scientist-directed hypothesis steering and audit agents that trace claims and reproduce analyses. Across predictive modeling tasks and three domain case studies, it achieved state-of-the-art results while scientists collaborating with audit agents found 14x more valid errors than manual review.