calibration

This paper studies how instruction tuning affects model confidence and lexical diversity in question answering, finding that it alters confidence and reduces rationale diversity without improving calibration.

This paper introduces CW-BASS v2, a saturation-aware pseudo-label selection method for semi-supervised semantic segmentation that adaptively switches between strict filtering and an adaptive confidence floor depending on the teacher's reliability. It shows improved results over baselines across several benchmarks with DINOv2 teachers.

This paper proposes CalibDCD, a calibration framework for feature-based LLM data contamination detection that mitigates feature shifts caused by post-training, improving detection performance by up to 7.0% AUC and 15.0% TPR@5%FPR.

MARCO is a Meta AI framework that decomposes clicks by intent to improve ads conversion prediction, correcting per-intent calibration bias and lifting conversions per click by +2.80% and topline metrics by +0.98% in production.

This paper identifies a 'representation confidence gap' in diffusion language models: internal states detect input noise accurately but reported confidence stays high and answer ranking degrades under noise. It introduces a lightweight, training-free extraction tool that leverages hidden states to improve ranking without modifying the base model.

This paper empirically studies token-probability-based confidence estimation and calibration for LLMs in mathematical question answering, comparing single-pass and multi-pass estimators and evaluating post-hoc calibration methods.

Ask-E is a new benchmark and training environment that evaluates and trains models on generating questions calibrated to specific skill levels, defined by the capabilities of two existing language models. Frontier models score below 50% on calibration, and training on Ask-E improves downstream math benchmarks without new math data or correctness-based rewards.

This paper introduces a simple Dirichlet-based forecaster that achieves optimal simultaneous multiclass U-calibration rates, closing the known dimension gap in regret bounds for bounded proper losses and removing extra additive terms for smooth losses.

This paper studies confidence estimation for financial vision-language models in chart and document understanding, evaluating seven estimators across five LVLMs. It finds that calibration, not ranking, is the scarce property, and only trained probes produce thresholdable scores for safe deferral to human reviewers.

This paper proposes a novel method to mitigate scoring bias in LLM-as-a-Judge by having LLMs randomly generate numbers to measure their latent numerical bias, then rectifying token generation probabilities accordingly. Experiments across four tasks show the method outperforms baselines and reveals that scoring bias varies across models, tasks, and score ranges.

This paper studies the sample complexity of multicalibration for a sequence of properties that are sequentially identifiable, establishing matching upper and lower bounds up to logarithmic factors.

This paper shows that apparent LLM self-correction gains often stem from format repair rather than improved reasoning. Across multiple model scales, format effects dominate content effects, with content margins near zero on capable models, suggesting the field has misattributed a minority of measured self-correction to actual content improvement.

The paper introduces DUD (Decoupled Update Dynamics), a framework that separates Feed-Forward Network and Attention contributions via causal interventions to improve uncertainty quantification and calibration in large language models, outperforming state-of-the-art baselines.

The author comments on FutureX's prediction evaluation framework, pointing out that it only scores final answers and cannot distinguish the quality of probability calibration, and discusses finer-grained evaluation methods such as Brier score and log loss.

This paper presents the first systematic study of calibration under unseen subtype shift, showing that models become overconfident on novel subtypes within known coarse categories, and argues that subtype robustness should be evaluated with calibration metrics rather than accuracy alone.

This paper shows that knowledge distillation has asymmetric effects on bias in small language models: it improves context-following on unambiguous tasks but harms refusal calibration on ambiguous ones, and proposes PCCD, a protocol to diagnose such per-item harms that aggregate metrics miss.

Introduces CALCoDe, a post-hoc reliability layer for frozen medical vision-language models that mitigates class-tail undercoverage under clinical shift, achieving strong worst-class accepted coverage across multiple dermatology shifts and VLM backbones.

Introduces a montage-agnostic encoder for calibration-light cross-user gesture recognition from surface EMG, using shared weights and electrode coordinates to handle variable channel counts and reduce per-user calibration. It outperforms per-user baselines on some datasets and analyzes factors affecting cross-user transfer.

This paper introduces value-router, a simulation study for cost-aware routing between cheap heuristics and expensive LLM calls in recommender systems, showing that value-weighted routing improves precision and handles seasonal demand surges with adaptive budgets.

A developer built a testing harness that measures KL divergence per weight group during quantization, leading to three custom quantized builds of Qwen3.6-27B (Bedrock, Tightrope, Gambit) with optimized compression. Tool calling is identified as the first capability to degrade under quantization.