calibration

Fovea-4B is an open-weight 4B vision model that brings Jev-style typed decision contracts to images, leading ImageJev-style tasks with 80.6 chance-corrected accuracy and strong calibration (ECE 0.035), released under Apache-2.0.

The paper proposes PACE (Proxy Alignment and Curvature Expansion), a pre-optimization diagnostic that uses repeated human ratings to predict the human-gain curve induced by a fixed proxy, modeling proxy optimization as a KL-regularized exponential tilt with first-order alignment and second-order correction terms, and shows reduced held-out gain-prediction error across multiple benchmarks.

The paper investigates three calibration approaches—HiGPTQ, HiAutoRound, and HiFlatQuant—for quantizing LLMs with the HiFloat4 block floating-point format, reducing the accuracy gap to BF16 below 1% and outperforming NVFP4 across models and tasks.

The paper introduces FABench, a scalable backtesting environment for forecasting agents with an automated pipeline that discovers events, generates questions, and resolves outcomes to avoid training-cutoff leakage. Evaluating agents on eight recent models, it finds retrieval consistently improves Brier scores and lifts leading agents to Polymarket market consensus, while revealing recurring evidence-weighting failures.

The paper compares five LLM-based review paradigms across three input representations and three LLMs on 232 ICLR 2025 submissions, finding that direct rating matches comparative ranking after recalibration with far fewer API calls, abstract-only input beats full-paper summaries, and human scores still predict citations better than any LLM paradigm except Group Rating.

EDU-Jev introduces a family of single-pass 'System One' decision models (0.8B/2B/4B) that return probability distributions over fixed options for educational assessment tasks, achieving 67.6% average accuracy across twelve benchmarks and outperforming leading commercial decision APIs by 9.4 points.

The paper introduces EDB-Bio, a 96k-episode benchmark for scientific evidence deliberation across three biological domains, showing that frontier and open-weight models are systematically underconfidence-prone (Brier scores near a trivial base-rate forecaster) and that post-training with proper scoring rules improves calibration, yielding an Ibis0 model 2.5× more efficient than the best frontier model in sequential experiment selection.

ThinkJev is a training-free method that builds fast (System 1) and slow (System 2) answering paths from a single hybrid-thinking open model, reading calibrated option probabilities directly or after variable-length thought, so it only pays for thinking where it statistically guarantees a target error rate.

该论文构建了涵盖五种幻觉类型的二元基准（来自六个数据集的9,852个样本），比较了五种决策模型与生成式裁判gpt-6-sol在幻觉检测上的表现，发现决策模型在部分类型上以1-3%的成本匹敌或超越生成式裁判，但在逻辑与指令不一致类型上仍有差距且概率存在过度自信问题。

该论文精确计算了语言模型在被要求随机选数时对每个整数的完整概率分布，发现指令调谐使分布更不均匀：安装了避免轮数的启发式并强化了避免端点值的倾向，但未产生跨模型共享的偏好整数。

MedJev 是一个可在笔记本本地运行的开源 System-One 医疗模型，在十项医学基准上进行零样本评测，性能接近 GPT-4，并提出两参数再校准方法显著改善患者级概率校准。

该论文提出知识冲突的解决不应依赖固定的启发式规则，而应引入对上下文可靠性的事先概率先验；在 GRACE 与 ClashEval 基准上，基于先验偏移的双声部选择器在各种上下文混合比例下显著优于现有基线。

This paper proposes Error-Subspace Calibration (ESCal), a post-merging calibration framework that identifies low-dimensional error subspaces from weight differences between expert and merged models, using lightweight Bridge modules to recover task-specific behavior. Experiments on vision and NLU benchmarks show ESCal improves merged-model performance across diverse architectures and merging methods.

The paper introduces a controlled benchmark of French surface variants (apostrophe omission, diacritic removal, keyboard typos) across seven tasks with 12,600 inputs, showing that accuracy drops and Brier score/NLL rise with perturbation intensity, while calibration combined with input restoration improves reliability.

This paper audits a cheap frozen 'System One' classification model against 14 frontier LLMs on content moderation, finding it accuracy-competitive at far lower cost, and shows that 'confidence' conflates two distinct uncertainties—human disagreement and decision risk—that require opposite escalation signals.

研究比较了 System One（决策）模型与标准 reranker 微调在检索重排任务上的表现，发现决策微调在所有 9B 以下 Qwen 主干上均不如直接进行 reranker 微调，nDCG@10 差距达 5 到 29 分，结论是如果需要重排器，就应该直接微调重排器。

A behavioral case study across six frontier LLMs finds they confidently misattribute a 17-year-old's exam answer to an adult, with stated confidence uncorrelated or inversely correlated with correctness, and articulate retrospective bias recognition that fails to improve blind judgments.

该论文证明在基于正确性奖励的价值函数强化学习中，置信度估计与critic训练是等价问题，因此已有的RL critic可直接复用为LLM的置信度估计器，无需额外训练或数据，且在数学、代码、工具调用等多个领域中表现出最佳的置信度估计与校准性能。

论文提出 SearchHorizonBench，一个用于评估 AI 智能体从执行经验中估计搜索价值的基准数据集，包含 45,632 次执行记录，发现历史校准可将 Brier 误差降低 16.2%，但历史估计器仅能解释不到 2% 的搜索增益方差。

This paper identifies and formalizes 'inference-time model shift'—a change in conformal score distributions caused by varying inference configurations like quantization and temperature rather than data distribution changes—showing it breaks split-conformal exchangeability and overstates deployment-time coverage across 8 language models and 4 finite-label benchmarks.