Tag
This paper analyzes selective prediction systems for rare-disease diagnosis, demonstrating that small open-weight LLMs have low recall on ultra-rare diseases and exploring the use of score margins for decision-making with limitations.
The paper diagnoses three failure modes in per-field selective risk control for document extraction systems and introduces a validity ladder of fixes, demonstrating improvements through experiments on real-world data with frontier AI models.
The paper introduces RA-DPO, a reliability-aware direct preference optimization method that combines annotator agreement, model confidence, and token-level uncertainty for sexism detection, improving training efficiency and enabling selective prediction.
The paper proposes A-CRC-QA, a post-hoc calibration framework for selective question answering that controls error rates among accepted answers via asymptotic risk calibration, demonstrating improved reliability-retention trade-offs on CoQA and MedMCQA.
This paper studies whether single-turn uncertainty quantification methods transfer to interactive LLM agent trajectories, evaluating white-box, black-box, and reflexive scorers across five LLMs and four tool-use datasets. Results show that transfer is uneven, with black-box self-consistency often strongest, and recommend revalidating UQ methods at the trajectory level.
This paper studies confidence estimation for financial vision-language models in chart and document understanding, evaluating seven estimators across five LVLMs. It finds that calibration, not ranking, is the scarce property, and only trained probes produce thresholdable scores for safe deferral to human reviewers.
Presents an evidence-grounded pipeline for constraint checking in construction documents, evaluating resolution-breadth tradeoffs in PDF evidence allocation on expert-referenced tasks.
This paper introduces Evidence Chain Evaluation (ECE), a selective fact-checking framework that allows LLM-based verification agents to abstain from giving verdicts when evidence is weak, sparse, or inconsistent. On ECE-Bench, ECE achieves 97.8% selective accuracy at 93.7% coverage, demonstrating a safety-oriented trade-off for handling epistemically weak evidence.
This paper introduces SEB-Cal, a method that augments output-space calibration with spectral features (band energy, entropy, peak dominance, phase stability) to improve selective reliability estimation in time-series classification, achieving higher Corr-AUROC and lower [email protected] across multiple datasets.
This paper studies which signals best predict correctness in text-to-SQL for selective prediction. It finds that verification-based signals from LLM judges outperform black-box statistical signals like self-consistency, and that a two-provider ensemble achieves 0.82 AUROC with well-calibrated probabilities.
This paper characterizes when conformal risk control can certify structured LLM outputs, proving impossibility bounds and analyzing certification hierarchies across different bounds. Empirical validation on six open-weight models shows that hard configurations are uncertifiable at low risk levels but practical certification is achievable at relaxed targets.
ExtractConf is a confidence estimation method for LLM-based document field extraction that uses two structurally different calls (field-guided and document-guided) to derive disagreement signals, achieving 0.928 ROC AUC on DocILE invoices and enabling reliable selective prediction for high-stakes automation.