calibration

The author explores using small LLMs like Gemma 4 as classifiers by analyzing logit probabilities with high temperature and calibration, sharing code and findings on GitHub.

TypeSafe AI launched model Jev with claims of calibrated confidence in AI responses, but the absence of published calibration data has sparked skepticism despite strong early adoption metrics.

TypeSafe AI launched its Jev model claiming no hallucination and calibrated probabilities, but the article questions the lack of public evidence for calibration while noting rapid developer adoption.

This paper proposes a generative approach using neural temporal point processes for probabilistic forecasting in business process monitoring, demonstrating superior calibration and inference efficiency over existing discriminative models.

The paper introduces Calibrated On-Policy Distillation (Cal-OPD), a method that estimates the teacher's self-deviation to calibrate teacher-student discrepancies, improving on-policy distillation for mathematical reasoning tasks.

This paper proposes a multi-subject pretraining approach for surface EMG speech decoding that reduces calibration time while improving accuracy, achieving a character error rate of 21.7% with just three minutes of target-subject data.

JevBench is a new benchmark that evaluates AI models on bounded software decisions by combining intelligence, calibration, speed, and cost, as announced by @rohanpaul_ai.

TypeSafe AI introduces Jev, a new class of AI models called System One Models, optimized for automated decisions in software with faster processing and lower costs than traditional LLMs.

A project analyzing NHANES survey data to classify coronary heart disease risk, emphasizing data leakage audit and calibration checks while comparing machine learning models.

Laya is an open-source non-autoregressive decision model that provides typed answers with calibrated probabilities, designed for tasks like email triage and conversational AI, showing significant performance improvements over existing models.

The article discusses the limitations of using LLMs directly as classifiers and proposes treating LLM outputs as features in traditional machine learning models like logistic regression to achieve better calibration, interpretability, and performance.

The paper proposes OBC-Prune, a calibration method for pruning large reasoning models that identifies causally important reasoning circuits to improve accuracy and reduce inference overhead on benchmarks like MATH500 and LiveCodeBench.

This paper demonstrates that calibration content in attention-based reranking can degrade performance with detailed instructions and proposes interpolated null calibration as a training-free method to recover rankings on instruction-heavy tasks.

This paper explores calibrated ambiguity as a generative resource in human communication versus multimodal language models, using the Dixit game to show that AI exhibits ambiguity collapse and lacks cultural references compared to humans.

This paper introduces methods to calibrate confidence in agentic systems using internal representations, demonstrating improved performance over baselines in multi-turn benchmarks.

This paper demonstrates that verbalized confidence has become a superior soft-scoring mechanism for LLM-as-a-Judge on post-2025 proprietary models, challenging previous advice to prefer log-probabilities. It introduces an overconfidence advisory and self-debate to improve calibration and robustness.

Counterfactual Fragility Certificates (CFC) introduce a model-agnostic audit protocol to detect high-confidence brittleness in machine learning models under structured evidence failure scenarios, improving over existing methods.

This paper introduces ECCBench, a benchmark and evaluation protocol for assessing memory in Vision-Language Models beyond accuracy, focusing on efficiency, compression, and calibration axes.

This paper proposes a calibration method for product-of-experts Gaussian process models to correct overestimated posterior variances, enhancing uncertainty quantification and scalability for large datasets.

SpanCalib-VLM presents a hybrid system that combines a multimodal sequence tagger with a generative vision-language model to enhance hallucination span detection and calibration, achieving improved performance on the SHROOM-Visions task.