MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

Hugging Face Daily Papers Papers
multi-agent confidence-calibration foundation-models ensemble-methods code-generation runtime-adaptation

Summary

MARGIN提出了一种运行时置信度校准方法，无需重训练或预留校准集即可从观测结果中学习模型特定的置信度修正，用于异构基础模型的多智能体协调决策，在代码生成和问答任务上相比未校准的置信度加权提升了答案选择准确率。

When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:19 AM

Paper page - MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

Source: https://huggingface.co/papers/2605.22949

Abstract

Whenacoordinatorcomparesanswersfromheterogeneousfoundationmodels,self-reportedconfidencemayhavedifferentmeaningsacrossrespondersandchangingworkloads.ThispaperpresentsMARGIN(Multi-AgentRuntimeGradingviaIncrementalNormalisation),aruntimecalibrationmethodthatlearnsmodel-specificconfidencecorrectionsfromobservedansweroutcomeswithoutretrainingthemodelsorrequiringaheld-outcalibrationset.MARGINtracksrecentaccuracyandstatedconfidencewithinconfidencebands,usestheirratiotocorrectreportedconfidence,andblendssparse-bandcorrectionstowardamodel-levelestimate.Thecorrectedscoresweightcandidateanswersinacollectivedecision.Evaluationcoverscodegeneration,questionanswering,andmathematics,usingan18-modelpoolandanine-modelsubsetfordistribution-shiftexperiments.OnBigCodeBench,model-meanconfidenceisnegativelyrelatedtoaccuracy;amongcorrect/incorrectresponsepairs,choosingthemoreconfidentresponderperformsbelowchance.Againstfiveonlinecalibrationbaselinesreceivingidenticalfeedbackandretainingtheirlearnedstateacrosseachtransition,MARGINachieveslowerpost-shiftexpectedcalibrationerrorthanallfiveintwocode-generationtransitionsandthanfourinaquestion-answeringtransition;theremainingquestion-answeringcomparisonisinconclusive.Inseparatecode-generationcoordinationexperiments,calibrationimprovestherankingofcorrectresponsesandincreasesanswer-selectionaccuracyby4.3and14.0percentagepointsontwoofthreebenchmarksrelativetouncalibratedconfidenceweighting.Theseresultssupportmodel-specificruntimecalibrationforcoordinationunderchangingworkloadswhencorrectnessfeedbackisavailablefortheparticipatingresponders.

View arXiv pageView PDFProject pageAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.22949 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.22949 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.22949 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles