Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Summary
The paper proposes prediction-powered smoothing and validation methods for disaggregated AI evaluation, enhancing accuracy in point and interval estimates across domains with limited labeled data.
View Cached Full Text
Cached at: 09/22/26, 03:28 PM
Paper page - Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Source: https://huggingface.co/papers/2609.20758
Abstract
EvaluatinganAIsystemrequiresdisaggregatedassessment,asperformancevariesacrossdomainssuchasbenchmarktasktypesorconversationtypesindeployedagents.Exhaustivetestingisexpensive,soevaluationrestsonasampleoflabeledunits.Wetreattheevaluationsetasafinitepopulationandseekaccuratepointandintervalestimatesofeachdomainmean.Directestimators,includingprediction-poweredinference(PPI),useonlyadomain’sownlabelsandareimprecisewherelabelsarefew.Smallareaestimationaddressesthisproblem,andwebuildonittodevelopanintegratedworkflowforestimationandvalidation.Forestimation,weproposeprediction-poweredsmoothing(PP-S),aBayesianmodelfittoeachdomain’sprediction-poweredestimate,withanextensionthatborrowsstrengthacrossareportingtaxonomy(PP-TS).Forvalidation,wederiveanew,approximatelyunbiaseddesign-basedcross-validationscoreforchoosingamongdirectandsmoothedestimators.Westudyacuratedbenchmarkwithverifiablegradinganddeployedagenttrafficgradedbyhumans,eachwitheveryoutcomeobserved.Inboth,theproposedestimatorsimproveonthedirectestimatorsinpointandintervalestimation,withnear-nominalcoverage.Atthesamesamplingbudget,ourscoreselectsaswellasanindependentvalidationsampledoesandestimatestheselectedestimator’serrorfarmoreaccurately.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2609\.20758
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.20758 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.20758 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.20758 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Skill-based Agentic Evaluation for Real-time Data Science Tasks
This paper introduces a ground-truth-as-code framework for evaluating data-science agents on continuously updated data, achieving improved agreement with human evaluators and reduced token consumption.
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
This paper introduces GenAI Evaluation, a governed and configuration-driven pipeline for scalable multi-dimensional evaluation of retail conversational agents. It achieves high accuracy using LLM-as-a-judge scoring with selective re-evaluation, validated against human-labeled data.
How do you keep local AI evaluations useful when the model keeps changing?
Discusses strategies to maintain the usefulness of local AI evaluations as models are frequently updated, focusing on adaptability and consistency.
High validation accuracy can conceal production risk: Using SHAP to expose and block proxy bias at runtime [P]
This article demonstrates how high validation accuracy can conceal proxy bias in AI models, using SHAP for explainability and a runtime governance wrapper to enforce fairness policies and prevent biased decisions in production.
Accuracy and Robustness of Model Cascades Under Data Perturbations
This research paper investigates how data perturbations impact the accuracy and robustness of model cascades for efficient AI inference, identifying key failure modes and stressing the need for evaluation under distribution shift.