Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Hugging Face Daily Papers Papers
scientific-forecasting research-agents evaluation-protocol predictive-credit llm-benchmarks uncertainty-quantification

Summary

本文提出一种配对预测协议，用于衡量科学解释对实验预测的额外价值，并在 Tox21 与 OpenML 基准上测试；结论是自然解释的预测信用在测试分辨率下仍未得到确认，而 DeepSeek V4 系列模型的解释卡可降低部分次级漂移。

Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive. Tox21's preregistered ROC AUC interval-score harm test was unmet (D-M=-.0026, 95 percent interval [-.0174, .0104]); OpenML's joint formation, point-equivalence, and repeatability rule was unmet. Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero. Under requested DeepSeek V4 Pro, matched and donor cards reduced secondary Tox21 drift by 64.5 and 59.1 percent. A DeepSeek V4 Flash replay raised matched point MAE from .01823 to .02020 and missed matched-donor interval-score equivalence. OpenML full-card assignment widened nominal 80 percent intervals by 21 percent, with 49.3 percent coverage versus 51.4 percent for description and content in 66/144 cards. Direct-text Flash delivered all 144 notes without detectable matched point-accuracy gain. A researcher-authored mechanism positive control lowered point MAE by 2.60 percentage points versus description. The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:27 AM

Paper page - Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

Source: https://huggingface.co/papers/2610.00314

Abstract

Researchagentsexplainplannedexperiments.Wemeasurepredictivecreditwithpairedforecastssharinganintervention,forecaster,andoutcomewhilevaryingdescription,matchedexplanation,anddonorcontext.Fivecheckstrackcommitment,delivery,predictivegain,alignment,andknown-signaluptake.Across336prospectivestatesincontrolledlearning,12Tox21endpoints,and24OpenMLtasks,v5’sfrozencreditdecisionwasinconclusive.Tox21’spreregisteredROCAUCinterval-scoreharmtestwasunmet(D-M=-.0026,95percentinterval[-.0174,.0104]);OpenML’sjointformation,point-equivalence,andrepeatabilityrulewasunmet.Matchedpoint-accuracygainsoverdescriptionremainedunconfirmed,andTox21/OpenMLseed-donorintervalsspannedzero.UnderrequestedDeepSeekV4Pro,matchedanddonorcardsreducedsecondaryTox21driftby64.5and59.1percent.ADeepSeekV4FlashreplayraisedmatchedpointMAEfrom.01823to.02020andmissedmatched-donorinterval-scoreequivalence.OpenMLfull-cardassignmentwidenednominal80percentintervalsby21percent,with49.3percentcoverageversus51.4percentfordescriptionandcontentin66/144cards.Direct-textFlashdeliveredall144noteswithoutdetectablematchedpoint-accuracygain.Aresearcher-authoredmechanismpositivecontrolloweredpointMAEby2.60percentagepointsversusdescription.Theprotocolmeasurespredictivecreditforresearch-agentbenchmarksandscientificforecasting;natural-explanationcreditremainedunconfirmedatthetesteddonorresolutions.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.00314 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.00314 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.00314 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Evaluating Explanation Methods by the Predictors They Induce

arXiv cs.LG

The paper proposes a new evaluation method for machine learning explanations by converting explanations into predictors and testing their ability to reproduce model predictions. It demonstrates that the effectiveness of explanation methods like SHAP and PDP depends on the independence of features in the data.

Can LLMs predict the outcomes of machine learning experiments?

arXiv cs.LG

该论文构建了包含2882个LLM提出的ML实验想法的数据集，评估五个前沿大模型预测实验结果的准确性，发现最佳模型Claude Opus 5.5仅解释约一半方差（R²=0.49），且所有模型都系统性低估了变化幅度，表明当前LLM对ML想法的直觉有用但有限。