calibration

这篇博文论证了让 LLM 自行生成置信度分数是不可靠的做法：置信度本质上应是基于信息的概率计算与校准结果，而 LLM 给出的数字缺乏可验证的依据，无法被信任。

This paper studies learned post-training quantization calibration for 4-bit text embedders (e5-base-v2, BGE-M3, EmbeddingGemma-300m), finding that calibration outcome ranges from gains to catastrophic collapse depending on the 'dose' of update, and proposes a validation-gated abstention rule that never selects harmful checkpoints while retaining most beneficial ones.

VX-One 是一个小型多任务神经网络，输出带类型且经过温度校准的决策基元（校准门控、含显式未知选项的闭集选择、序数分数），用于实时边界流量分诊。论文在 UNSW-NB15、CIC-IDS2017、EMBER2024 上评测其与签名引擎的两阶段级联表现，发现其在时漂移下检测占优，但校准会崩溃且未知攻击拒识能力跨基准迁移性差。

DecisionPivot is a diagnostic benchmark with 48 tasks and 40,033 items evaluating decision models in LLM systems, revealing large gaps between aggregate accuracy, contrast-group performance, calibration, and reasoning effort that standard evaluations obscure.

这篇论文审计了三类非生成式决策模型（编码器打分器、解码器打分器和闭源 API）在孟加拉语习语上的表现，发现当选项列表中包含字面释义时，模型会强烈偏向字面含义，且这种"字面吸引"在孟加拉语测试中比英语高 1.9 到 6.4 倍，同时校准性全面下降。

该论文研究如何将一个模型表示上学到的量化校准增益复用到同一预训练模型的其他表示（如浮点与INT3量化）上，无需接收端优化即可保留大部分文本损失改善，量化了校准更新的可迁移性与接收端专属训练的剩余收益。

一篇博客文章系统分析了决策模型（如 Jev）的经济学，探讨其 choice/score/noul 输出类型、几乎免费的输出 token 成本，以及由此衍生的"决策层级"内存层级类比。

This paper evaluates whether a typed decision model (Jev 1.13) consulted at fixed decision points can make a Claude Sonnet 5.5 terminal agent cheaper without hurting performance on Terminal-Bench 2.1. The controller increased cost per attempt by 19% with no pass-rate improvement (73% vs 73%), because although it ranked finish requests well (AUROC 0.78), it was poorly calibrated and triggered unnecessary verification.

A hands-on benchmark of TypeSafe's Jev 1.14 image input and new Box (bounding-box) output across 48 labeled COCO photos, finding strong yes/no accuracy (98.4%) but notable pitfalls: Box never answers "not found", confidence thresholds don't transfer between model versions, and wording strongly affects probabilities. The author recommends pairing every Box with a presence check, per-question thresholds, and pinning the model version.

System1-Arena 是一个包含 92k 问题、105 项任务与 138 个闭环环境的基准，用于比较 System-1 概率模型与 System-2 推理 LLM 的离散决策能力。结果显示最大的 System-1 模型在非智能体任务上以四分之一延迟匹配快速闭源 LLM 且校准更好，而推理 LLM 精度最高但成本约为 70 倍；在闭环环境中大型开源 LLM 成功率最高（达 32.9%）。

The paper introduces Verdict-MQ, a 395M open encoder that encodes an input's state once and answers many typed questions about it via a structured attention mask, outperforming the 421M Laya cross-encoder on calibration and accuracy for typed decisions while trading off performance on RACE and BoolQ.

This paper shows that naive few-shot LoRA personalization on-device degrades out-of-scope (OOS) rejection AUROC across five intent-classification benchmarks while preserving in-scope accuracy, and proposes a training-free fix using Mahalanobis-distance-based rejection scoring that recovers most of the lost quality on 3 of 5 datasets.

Liquid AI 发布 d1-ultra，一款多模态决策模型，在 112 项评测中超越 Jev 1.14 及所有其他决策模型，输出经校准的概率答案，现已在 Liquid API 上线，价格为每百万输入 token 0.042 美元。

Liquid AI released d1-ultra, a multimodal decision model that processes text and images in a single forward pass to return calibrated probabilities without token generation, claiming top performance on the Artificial Analysis Decision Index and pricing at $0.042 per million input tokens.

作者实现了一个基于 conformal risk control 的写操作安全闸门 gate-calibrated，在 tau-bench retail 上验证了风险上界成立（α=0.02 时 1.7% 错误写入未询问，达标），但闸门在每错误写入避免率上并未优于简单地每次询问用户。诊断显示模型的 verbalized confidence 和自采样一致性信号均已饱和（错误步骤上置信度也≥0.9），真正起作用的特征只是对话步数位置，说明该方案在易基准上也难以产生有区分度的置信信号。

该研究探讨多智能体LLM系统中应如何通信以聚合分布式私有信息，证明在保持校准不确定性的情况下，通信概率信念比通信具体动作更能促进协作决策，并提出了一种事后对话校准干预方法。

该论文提出 SRCoD，一个结构-响应协同校准的框架，用于在条目级别诊断多模态基准的模态依赖性，识别可通过单一模态或答案规律作答的捷径题目。实验表明 SRCoD 在基准精化验证和人工标注验证两种协议下均优于现有基准诊断方法，代码与数据已开源。

Jev 1.14.0 achieves strong overall calibration on binary questions (ECE of 0.023), but a breakdown by difficulty reveals it is strongly over-confident on hard questions by about 22 percentage points, exceeding the calibration threshold for that slice.

Intern-Decision introduces a family of compact (0.8B/2B/4B) multimodal decision models using an autoregressive masked language modeling objective to output structured choices with probabilities in a single causal forward pass, achieving 90.02% average accuracy across seven benchmarks with low latency for interactive agents.

A technical blog post comparing causal-decoder (v19) and encoder-based designs for strands-decider-2B, finding that encoders like e1b offer lower latency with similar accuracy, while encoder-decoder models scale worse in prompt length.