calibration

This paper applies split-conformal claim filtering to multi-hop RAG, showing that at a 95% conformal target, the fraction of responses with fully supported claims rises to 95.80%-97.20% across HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini — but with strong abstention and low claim retention, highlighting that reliability gains trade off against answer coverage.

The paper presents OpenJev-RLCD, a working open-source implementation of Reinforcement Learning for Calibrated Decisions (RLCD) for reasoning models, using a 'calibrate, then reinforce' recipe with proper scoring rules that matches or beats SFT, RFT/STaR and GRPO with temperature scaling, especially in selective prediction (e.g., 81% of GSM8K items decided at ≤5% error versus 19% for GRPO).

The paper introduces SBERT2S1, which converts Sentence-Transformers retrieval encoders into calibrated typed decision models, along with the BIODECIDE benchmark and MEDLINE-S1 dataset (243k decisions from NLM indexing), showing that retrieval training helps prior-fused residual heads but often hurts cross-heads, and that RLCD reward training underperforms cross-entropy due to biased reward normalization.

A self-run evaluation of two small confidence-scoring classification models — TypeSafe AI's hosted Jev and the open-source Laya — finds that label descriptions phrased around input features, consistent time windows, and calibration (ECE) matter more than model scale for downstream thresholding performance.

PACT introduces a four-term training objective for single-token typed-decision models that exploits contrastive pairs and certificate-verified ablations, matching baseline accuracy while significantly reducing position bias, seed variance, and calibration error.

A paper showing that running statistics over raw LLM judge scores inflates false positives, introducing PPI-based calibrated hypothesis tests (including the first PPI Wilcoxon/Mann-Whitney corrections) and the open-source Python package evalstats, which automatically selects statistically sound methods for small-sample AI evaluations.

A paper introducing PersonaDose, a calibrated activation-steering approach that controls language model persona expression via trait descriptions and requested mean intensity, evaluated across Llama-3.1-8B, Qwen3-8B, and Gemma-3-4B.

The paper introduces CALIBRA, a calibration-first multimodal temporal framework for cross-cohort asthma risk forecasting that uses recurrent encoders, gradient-reversal domain adaptation, hierarchical logistic calibration, and conformal prediction to remain reliable under distribution shift and incomplete modalities, validated on a semi-synthetic three-cohort benchmark.

A comprehensive visual guide to language models for text classification, including hands-on experiments with RNNs, CNNs, transformers, and calibration techniques.

PixSim is an open-source simulator for modeling fraud recovery and interdiction in Brazil's Pix instant payment system, calibrated to real data and evaluating policies under analyst capacity constraints.

This paper audits a non-generative 'System-1' model on biosecurity-relevant benchmarks, measuring accuracy, calibration, and sensitivity to answer option order, with implications for AI safety pipelines.

The article explains how to address hallucination in large language models by using probability distributions and calibration to quantify uncertainty, rather than unreliable confidence scores.

The author benchmarked the Jev judge on a dataset and reduced its calibration error by 68.1% through learning from human-labelled examples, improving confidence alignment for production use without significantly changing classification accuracy.

The Intern-Decision multi-modal model family (0.8B, 2B, 4B) is released, with the 4B model outperforming Jev 1.13.0 in decision-making tasks, featuring fast local deployment and improved calibration on probability problems.

DSPy 3.4.0 is released with native support for Jev and System one models and a new optimizer, ReAnchor, for calibrating outputs with confidence.

PixelJev is introduced as a native-image decision interface using small open multimodal models to map images, instructions, and candidate sets to structured choices. The study demonstrates adaptation improves accuracy on benchmarks like Pets and highlights challenges in calibration and generalization.

CoRe-Stack+ is a meta-learning pipeline that improves deep stacking generalization by filtering redundancies and enhancing calibration, achieving better accuracy and efficiency on vision benchmarks.

The paper proposes LWCal, a CPU-only post-hoc calibration method for tabular classifiers that handles noisy calibration labels without requiring clean data, showing improved calibration error and scoring metrics.

本文总结了来自 Hamel Husain 和 Shreya Shankar 的 AI Evals 课程的 8 个核心技能，旨在指导工程师和产品经理构建有效的 AI 评测系统，涵盖错误分析、评测器设计、校准和监控等步骤。

This paper argues that calibration should be a first-class criterion in LLM evaluation to enhance trustworthiness and address miscalibration problems in deployment and research pipelines.