calibration

ProbabilityBench introduces a benchmark with exactly known ground-truth class probabilities, allowing direct measurement of how well models, including decision models like Jev and Perplexity Decider, estimate true probabilities and enabling evaluation of calibration metrics and Bayes-error estimators.

ReCal 是一种即插即用的校准方法，在结构化剪枝前通过教师-剪枝探针间的前向 KL 识别受损预测并重加权校准统计，从而提升剪枝模型在在线策略蒸馏（OPD）后的恢复效果，在 AIME 上最高提升 16.7 个百分点。

TypedBench is a new benchmark for evaluating System One typed decision models on calibration, wording sensitivity, policy adherence, and decision cost under asymmetric cost matrices, comparing a hosted model, an open encoder, and open decoder families on identical items.

本文提出 RL-ARC，一个校准感知的训练框架，通过联合利用推理置信度与答案置信度来校准大型推理模型，缓解 RLVR 训练导致的过度自信问题，并在 ID 和 OOD 设置下提升校准表现而不显著牺牲推理性能。

The paper introduces NeSCal, a graph-based train-time regularizer that penalizes Jensen-Shannon divergence between predictive distributions of neighboring samples in representation space, showing improved calibration that complements post-hoc methods like temperature scaling across image and tabular benchmarks.

This arXiv paper introduces a role-aware Decider-Supervisor framework combining centralized ML, a FedAvg-trained federated meta-model, and Base/QLoRA LLM variants with blockchain auditability, finding that AI supervision only reduces fraud-decision errors under specific roles, calibration, and traffic conditions rather than by default.

Phocinae-Largha-150M-v1 是一个 144.3M 参数的决策模型（基于 mmBERT-small），接受结构化状态和类型化的是非/选择/评分问题，单次前向推理即可输出带置信度的校准答案，在自建英文评测集上取得 0.906 准确率，延迟低至 GPU fp16 21ms。

论文研究了在实时环境中，快速决策模型何时应将控制权交给慢速推理的视觉-语言模型的双系统智能体架构。通过 Doom 游戏闭环实验，发现决策模型的置信度校准与 AUROC 能有效预测延迟决策带来的收益，并开源了代码、提示词与数据。

The paper introduces Threshold-Aware Conformal Routing (TACR), a method that learns input-dependent interval scales to decide when a neural surrogate can replace an expensive scientific simulator, reducing simulator deferrals by 14–75% while preserving distribution-free marginal coverage guarantees.

The paper introduces Calibrated User Embeddings (CUE), a training-free framework that encodes observed sessions and samples continuous persona embeddings decoded into prompts for LLM user simulators, achieving better outcome calibration and failure-mode reproduction than existing persona-based simulation methods on tau^2-Bench.

HakemBench introduces an open Turkish benchmark (CC BY 4.0) for 'typed decisions'—calibrated probability outputs over fixed options—covering seven tracks such as fact-check triage, legal routing, and moderation. It evaluates decision quality (macro F1), calibration (Brier score), and selective automation (risk-coverage), while transparently disclosing weaknesses in its AI-generated gold labels and non-blind evaluation of the lab's own model.

MIT 的 Neil Thompson 提出 O*NET-BENCH 审计套件，评测 33 种 LLM judge 配置，发现排序一致性虽高但接受率估计偏差极大（3.0%–97.9% vs 工人实际的 61.1%），表明仅凭排序一致性选择 judge 不足以支撑职业层面的 AI 测量。

The paper evaluates twenty deepfake detectors against ten generators, finding detection accuracy degrading from 99.5% to 76% over time and collapsing below 2% under adversarial perturbations, then proposes a calibrated detection paradigm based on plausibly deniable authenticity via generator reconstruction, achieving at most 1% false certification.

SearchJev introduces a fast, calibrated System-1 model for search agents that scores legal decisions without autoregressive generation, using Soft-Label Learning for Calibrated Decisions (SLCD) to reduce latency and improve confidence reliability. A dual-system agent combining SearchJev with a System-2 model achieves up to 4.7x speedups on BrowseComp-Plus while raising answer accuracy from 45% to 54%.

A practitioner describes a small self-learning loop around an AI message-drafting assistant, pairing a banned-words/structural-tells checker with a nightly job that extracts style rules from edits and rejections, tightly constrained by a 6-rules-per-night cap and a 25-rule style-only limit with sanitisation to prevent the model from self-instructing beyond bounds.

OmniMed-Jev reframes medical LVLM outputs as distributions over a runtime-supplied candidate set, cutting calibration error by up to an order of magnitude and reliability error by two versus a generative baseline while keeping point-prediction performance comparable (except counting).

This paper benchmarks typed 'System One' decision models against supervised classifiers and generative language models for automated decision gates, finding that model rankings depend on label availability, calibration, and cost. It introduces condition-dependent design rules and shows cascaded models can match accuracy at a fraction of the cost.

IFM releases K2-Type-0.9B, an Apache-2.0 0.9B decision model that takes one state plus multiple typed questions (noul, choice, score) and returns calibrated probabilities in a single forward pass, scoring 0.762 on JevBench's public set with 20-60ms latency.

An independent blog analysis of TypeSafe's Jev 'System One' classifier model, testing how accurately its predicted probability distributions match known ground-truth distributions (e.g., the Maxwell-Boltzmann velocity distribution) using well-understood questions. The post concludes that Jev is poorly calibrated despite being extremely cheap to evaluate.

This technical blog post explores whether Jev, TypeSafe's fast structured-decision model, is secretly learning a value function, examining how calibration in System 1 decision-making relates to Q-function prediction in reinforcement learning and agentic browser tasks.