Tag
Google's Gemma 4 model has been enhanced to recognize when it is incorrect, improving its calibration and reliability.
This paper investigates the calibration gap that arises when temperature scaling, a common post-hoc calibration method relying on one-hot labels, is applied to models trained with soft label distributions reflecting genuine human disagreement. Experiments across vision and language domains show that temperature scaling calibrated on hard labels consistently underperforms direct soft-label calibration, with larger gaps in language tasks, highlighting risks for safety-critical deployments.
Introduces DualEval, a framework that jointly calibrates model ability and item difficulty/sharpness to unify static benchmark and arena-style evaluation, enabling more reliable rankings and downstream applications like benchmark compression and anomaly detection.
This paper explores Large Language Models' inability to recognize their knowledge limits on structured clinical data, proposing a cross-model attribution divergence method to detect epistemic blind spots. The approach improves calibration and accuracy without training by combining few-shot examples and SHAP-derived feature evidence.
This paper proposes a distribution-aware training approach for modeling next-event predictions in concurrent Go programs, treating scheduler nondeterminism as a signal. Fine-tuning a 7B model on fewer than a thousand traces achieves 36.2% accuracy on production bugs, outperforming Gemini 3.5 Flash zero-shot.
This paper introduces Adaptive Tool Trust Calibration (ATTC), a framework that improves tool-integrated reasoning models by enabling them to adaptively decide when to trust or ignore tool results based on code confidence scores. The approach addresses the "Tool Ignored" problem where models incorrectly dismiss correct tool outputs, achieving 4.1-7.5% performance improvements across multiple models and datasets.