metrology

An explainer about Vienna Standard Mean Ocean Water (VSMOW), the official international water standard used in precise measurements, and why its purity and isotopic composition make it extremely expensive.

This paper introduces a psychometric datasheet protocol for evaluating LLM judges as measurement instruments, measuring dark current, positional false preference, stable cross-sensitivity, and target sensitivity. A case study on three open-weight models reveals significant differences in judge quality and behavior.

Discusses the structural weakness of current evaluation methods for LLMs, which fail to anticipate qualitative shifts in capability, and argues that developing proactive evaluation infrastructure is the critical bottleneck for safe capability jumps.