Calibrated Preference Learning: The Case of Label Ranking

# Calibrated Preference Learning: The Case of Label Ranking
Source: [https://arxiv.org/html/2605.30447](https://arxiv.org/html/2605.30447)
Viktor BengsTimo KaufmannSebastian J\. VollmerEyke Hüllermeier

###### Abstract

Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision\-making\. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set\. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top\-k predictions\. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub\-rankings, and top\-k rankings\. We prove that full\-rank calibration implies the others but not conversely, and sub\-ranking and top\-k calibration are incomparable\. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub\-ranking and top\-k metrics\. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top\-1 accuracy\. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it\.

Machine Learning, ICML

## 1Introduction

Probabilistic label ranking \(ProLR\) models a probability distribution over the possible rankings of a set of items given a context, capturing both the probability and the certainty of each potential ranking\(Cheng\.2010;Cheng\.2009\)\. For such predictions to be trustworthy, the model must be*calibrated*: predicted probabilities should align with true outcome frequencies\(Filho\.2023\)\. While calibration has been extensively studied in classification\(Vaicenavicius\.2019\), regression\(Song\.2019\), and recommender systems\(DaSilva\.2025\), calibration for distributions over rankings remains unexplored\. One important application is Reinforcement Learning from Human Feedback \(RLHF\)\(Ouyang\.2022\), which aims to align LLMs to human preferences via reward models that learn underlying preference structures\. The alignment procedure in RLHF relies on pairwise preferences, which are substructures of full rankings as considered in label ranking\(Wirth\.2017\)\.

Label ranking aims to predict a complete ordering of a set of items given some context\(Furnkranz\.2008;Hullermeier\.2008\)\. Unlike standard label ranking, which predicts only the single most likely ordering\(Vembu\.2010\), ProLR exposes the distribution over rankings\. This information, in turn, is only reliable for downstream decisions if the model is well calibrated\(Filho\.2023\)\. In particular, calibration ensures that expressed uncertainty reflects true outcome variability, which is desirable for pluralistic alignment where consensus and contestation should be distinguished\.

In ranking\-related settings such as recommender systems\(DaSilva\.2025\), calibration has been viewed as aligning predicted item scores with underlying user scores\. These score\-based calibration methods\(Yan\.2022;Sculley\.2010\)then aim at aligning scores to correctly reflect how likely a user is in favor of an item to be ranked\. A different line of work defines calibration via a pivot point that separates relevant from non\-relevant items within a ranking\(Furnkranz\.2008\)\. In contrast, aligning a predicted distribution to the underlying distribution over rankings, as required in ProLR, remains unexplored\.

One can easily obtain a notion of calibration by treating the label ranking problem as a multi\-class classification problem, with each ranking considered a class\. However, the resulting notion of calibration has some disadvantages for practical applications\. First, there is a factorial number of \(ranking\) classes, which makes the quantitative measurement of this kind of calibration computationally difficult even for a moderate number of labels\. What complicates matters further is that, in general, the rankings observed reflect only a tiny fraction of all possible outcomes\. Furthermore, the metric structure of the ranking space is not reflected in the straightforward calibration notion for multi\-class classification\. For example, the rankingsi1≻i2≻i3i\_\{1\}\\succ i\_\{2\}\\succ i\_\{3\}andi1≻i3≻i2i\_\{1\}\\succ i\_\{3\}\\succ i\_\{2\}both share the common sub\-rankingi1≻i2i\_\{1\}\\succ i\_\{2\}, and, as such, calibration on the former may induce calibration on the latter \(see[Section 4](https://arxiv.org/html/2605.30447#S4)\), which remains unreflected in the naive classification viewpoint\. Finally, important modalities for practical applications, including pairwise and top\-k rankings, are not taken into account\.

Full Rank Calibration\(Definition[1](https://arxiv.org/html/2605.30447#Thmdefinition1)\)Rankwise Calibration\(Definition[2](https://arxiv.org/html/2605.30447#Thmdefinition2)\)Sub\-k Calibration\(Definition[4](https://arxiv.org/html/2605.30447#Thmdefinition4)\)Top\-k Calibration\(Definition[5](https://arxiv.org/html/2605.30447#Thmdefinition5)\)Rankwise Sub\-k Calibration\(Definition[6](https://arxiv.org/html/2605.30447#Thmdefinition6)\)Rankwise Top\-k Calibration\(Definition[7](https://arxiv.org/html/2605.30447#Thmdefinition7)\)Sub\-k FamilyTop\-k Family[Theorem 4\.1](https://arxiv.org/html/2605.30447#S4.Thmtheorem1)[Theorem 4\.2](https://arxiv.org/html/2605.30447#S4.Thmtheorem2)\(i\)[Theorem 4\.2](https://arxiv.org/html/2605.30447#S4.Thmtheorem2)\(ii\)[Theorem 4\.3](https://arxiv.org/html/2605.30447#S4.Thmtheorem3)\(i\)[Theorem 4\.3](https://arxiv.org/html/2605.30447#S4.Thmtheorem3)\(ii\)Figure 1:Overview of the calibration definitions and their relationships \(an outgoing arrow means implication, whereas dashed only hold for specific model classes\)\. Exclusion results are depicted inLABEL:fig:overview\_exclusive\_figure\.We therefore introduce the notion of calibration in ProLR, capturing calibration at different granularities, ranging from calibration over the full item set to calibration over selected subsets\. Building upon the two forms of strong and weak calibration in multi\-class classification, we develop a hierarchy of calibration notions for ProLR\. We theoretically investigate the relationships between these notions, showing that calibration on sub\-items does not imply calibration on all items\.

#### Contribution\.

Our main contributions are as follows:

1. 1\.Calibration in Label Ranking:We introduce calibration notions tailored to probabilistic label ranking, extending beyond what can be captured by viewing label ranking as a multi\-class classification problem\.
2. 2\.Unified theoretical picture \([Figure 1](https://arxiv.org/html/2605.30447#S1.F1)andLABEL:fig:overview\_exclusive\_figure\):We establish theoretical relationships between the proposed calibration notions, showing that calibration on sub\-rankings does not imply calibration on full\-rankings, also for widely used Plackett–Luce and Mallows models\.
3. 3\.Empirical Investigation:We empirically evaluate the calibration properties of popular label ranking learners, illustrating differences between sub\-ranking and full\-ranking calibration\. We also examine calibration of reward models in a popular RLHF benchmark\.

## 2Related Work

#### Probabilistic Calibration

Prior work on multi\-class calibration has established a hierarchy of increasingly strong notions, ranging from calibration of only the most likely class\(Guo\.2017\), through*class calibration*\(Zadrozny\.2001\), which decomposes into one\-vs\-rest binary tasks, to*multi\-class calibration*\(Widmann\.2019\), which requires calibrated distributions over all classes\. This hierarchy has been extended to the case of regression\(Song\.2019;Widmann\.2021\)and set\-based probabilistic classifiers\(Mortier\.2023;Juergens\.2025\)\. Our work introduces an analogous hierarchy of calibration notions for the label ranking setting, additionally considering special cases such as pairwise and top\-k calibration\. Calibration has been investigated for popular machine learning algorithms, such as decision trees\(Zadrozny\.2001\), random forests\(Shaker\.2025\)and neural networks\(Mukhoti\.2020;Wang\.2023\)\. In line with this, we investigate calibration of popular ranking models such as Plackett–Luce and Mallows models, showing that they are often poorly calibrated for large item sets\. For an extensive overview of calibration methods, we refer to the reviews byFilho\.2023andLane\.2025\.

#### Label Ranking and Calibration\.

The field of preference learning has contributed significantly to the rise of LLMs, particularly to their fine\-tuning\(Ouyang\.2022;Kaufmann\.2025\)\. This development was accompanied by new research impulses in subfields of preference learning, such as label ranking\. New learning methods\(korba2018structured;adam\_inferring\_2024;Thies\.2024;zhou\_heuristic\_2024\)and extensions of the standard setting have been introduced, such as the partial label ranking\(alfaro\_learning\_2021;alfaro\_pairwise\_2023\)or dyad ranking\(schafer2018dyad\)problem\. Further work has improved inference for probabilistic ranking models, such as the popular Mallow’s model\(Kenig\.2018;Ping\.2020\)\. Yet surprisingly, the concept of calibration has been completely neglected so far in label ranking, even though \(i\) calibration is of special interest to the emerging topic of uncertainty quantification in ML\(Huellermeir\.2021\)and \(ii\) a well\-calibrated label ranking model could be a useful tool to capture annotator differences and train models that reflect these nuances\.

#### Calibration in other Preference Learning Settings

In contrast to ProLR, calibration in other ranking\-based domains, particularly recommender systems, has received considerable interest\. It should be noted that, in this domain, calibration refers to aligning predicted scores with the user’s underlying latent scores\. In ProLR, however, we consider aligning probability estimates to the true probabilities of the data\-generating process\.Sculley\.2010were the first to consider calibrated scores alongside accurate rank predictions\.Li\.2015;Steck\.2018;Penha\.2021show that solely focusing on prediction quality in recommender systems deteriorates calibration, resulting in reduced user satisfaction\. Calibration of top\-k rankings based on recommender systems has been considered bySato\.2024\. Similarly,Yan\.2022;Zhang\.2024deal with increasing user satisfaction in recommender systems via a two\-component loss function, where the latter part encourages calibration\. To improve calibration, non\-parametric\(Menon\.2012\)and parametric methods\(Kweon\.2022\)have been introduced, generalizations of classification methods\. We refer toDaSilva\.2025for a broader overview\.

#### Calibration in RLHF\.

The term calibration appears in multiple senses within the RLHF literature\. One line of research studies calibration of LLM output confidence: whether models express well\-calibrated uncertainty in their answers\(Zhu\.2023;Tian\.2023;Stangel\.2025\)\. A related question for our work is whether reward models themselves are calibrated\.Halpern\.2025formalize pairwise calibration for pluralistic alignment: they train ensembles of reward functions such that the ensemble prediction is calibrated with respect to the empirical annotator distribution\.Kaufmann\.2025\.bfocuses on training a single reward model that captures utility differences well, thereby producing better\-calibrated Bradley\-Terry probabilities\. Our work is more closely aligned with the second direction, as we focus on calibrating reward models themselves\. It differs from prior work by formalizing calibration*notions*for ranking distributions, offering a framework for measuring and comparing calibration across prediction granularities and beyond RLHF, rather than proposing training methods\.

## 3Probabilistic Label Ranking

Here, we introduce the general notation of probabilistic label ranking and formalize the learning problem\. We then briefly review common probabilistic ranking models, including the Plackett–Luce and Mallows models\. Finally, we discuss the connection between the Plackett–Luce model and the widely adopted learning paradigm of ranking by pairwise comparison\. For a more general overview of label ranking, see\(Vembu\.2010;Zhou\.2014\)\.

#### Notation\.

Let𝒳\\mathcal\{X\}denote the feature space,ℐ\\mathcal\{I\}the set of items, andℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}a subset of items\. The set𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}consists of all rankings ofℐ\\mathcal\{I\}, withm=\|ℐ\|m=\|\\mathcal\{I\}\|denoting the number of items\. A rankingπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}is a mappingπ:ℐ→ℕ\\pi:\\mathcal\{I\}\\rightarrow\\mathbb\{N\}, which assigns each item its corresponding position in the ranking\. We write a ranking asπ=i≻j≻k\\pi=i\\succ j\\succ kinstead ofπ​\(i\)=1,π​\(j\)=2,π​\(k\)=3\\pi\(i\)=1,\\pi\(j\)=2,\\pi\(k\)=3\. We also use the≻\\succoperator to represent a rankingπ\\pithrough \(sub\-\)rankingsπ1,π2\\pi\_\{1\},\\pi\_\{2\}, e\.g\.,π=π1≻π2\.\\pi=\\pi\_\{1\}\\succ\\pi\_\{2\}\.For each rankingπ\\pi, we denote byπ−1:ℕ→ℐ\\pi^\{\-1\}:\\mathbb\{N\}\\rightarrow\\mathcal\{I\}the mapping assigning each position its corresponding item, e\.g\., ifπ=i1≻i3≻i2\\pi=i\_\{1\}\\succ i\_\{3\}\\succ i\_\{2\}, thenπ−1​\(1\)=i1\\pi^\{\-1\}\(1\)=i\_\{1\},π−1​\(2\)=i3\\pi^\{\-1\}\(2\)=i\_\{3\}, andπ−1​\(3\)=i2\\pi^\{\-1\}\(3\)=i\_\{2\}\. Furthermore, givenρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}withℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}, we writeρ⊆ℬπ\\rho\\subseteq\_\{\\mathcal\{B\}\}\\piif∀i,j∈ℬ:ρ​\(i\)<ρ​\(j\)⇔π​\(i\)<π​\(j\)\\forall i,j\\in\\mathcal\{B\}:\\ \\rho\(i\)<\\rho\(j\)\\Leftrightarrow\\pi\(i\)<\\pi\(j\)and omitℬ\\mathcal\{B\}when it is clear from context\. We denote random variables by capital lettersX,ΠX,\\Pi, sets by calligraphic or script letters such as𝒳,𝒮ℐ\\mathcal\{X\},\\mathcal\{S\}\_\{\\mathcal\{I\}\}, and realizations by lowercase lettersx,πx,\\pi\. The probability of an eventAAis denoted byP​\(A\)P\(A\), whileℙ​\(𝒜\)\\mathbb\{P\}\(\\mathcal\{A\}\)denotes the set of all probability distributions over a set𝒜\.\\mathcal\{A\}\.

#### Probabilistic Label Ranking\.

Let a hypothesis spaceℋ\\mathcal\{H\}be given, where a modelh∈ℋh\\in\\mathcal\{H\}is a mappingh:𝒳→ℙ​\(𝒮ℐ\)h:\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)\. For a rankingπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}and a given contextx∈𝒳x\\in\\mathcal\{X\},h​\(x\)h\(x\)denotes a distribution over𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}, andh​\(x\)​\[π\]h\(x\)\[\\pi\]denotes the probability assigned toπ\\pi\. A modelhhis learned by minimizing a chosen loss function on a dataset𝒟=\{\(xi,πi\)\}i=1n\\mathcal\{D\}=\\\{\(x\_\{i\},\\pi\_\{i\}\)\\\}\_\{i=1\}^\{n\}withxi∈𝒳x\_\{i\}\\in\\mathcal\{X\}andπi∈𝒮ℐ\\pi\_\{i\}\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\. Throughout this work, the data points are assumed to be independent and identically distributed\. Similarly to standard supervised learning, a common approach in probabilistic label ranking is to induce a parametrized ranking model of the formh​\(x\)=f​\(g​\(x\)\)h\(x\)=f\(g\(x\)\), wheref:Θ→ℙ​\(𝒮ℐ\)f:\\Theta\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)is a parametrized ranking distribution andg:𝒳→Θg:\\mathcal\{X\}\\rightarrow\\Thetapredicts the per\-instance parameters\. Sinceffis fixed onceθ∈Θ\\theta\\in\\Thetais given, learning reduces to finding the optimal functiongg\.

#### Plackett–Luce Model\.

A prominent class of models forffis latent utility models, which are parametrized byθ∈Θ⊂ℝ\+m\\theta\\in\\Theta\\subset\\mathbb\{R\}\_\{\+\}^\{m\}\. Widely adopted is the Plackett–Luce \(PL\) model, which assigns a probability to a rankingπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}via

q𝜽​\[π\]=∏i=1m𝜽π−1​\(i\)∑j=im𝜽π−1​\(j\)​\.q^\{\\bm\{\\theta\}\}\[\\pi\]=\\prod\\nolimits\_\{i=1\}^\{m\}\\tfrac\{\\bm\{\\theta\}\_\{\\pi^\{\-1\}\(i\)\}\}\{\\sum\_\{j=i\}^\{m\}\\bm\{\\theta\}\_\{\\pi^\{\-1\}\(j\)\}\}\\,\\text\{\.\}\(1\)The parameters are typically learned by minimizing the negative log\-likelihood of the PL model on the training data\. As the PL model is identifiable only up to a positive scaling factor\(Alvo\.2014\), the parameters are commonly constrained such that∑i=1mθi=1\\sum\_\{i=1\}^\{m\}\\theta\_\{i\}=1for allx∈𝒳x\\in\\mathcal\{X\}111For stability,ggoften outputs log scores:θi=exp⁡\(g​\(x\)i\)\\theta\_\{i\}=\\exp\(g\(x\)\_\{i\}\)\.\. We denote the resulting hypothesis space byℋP​L\.\\mathcal\{H\}\_\{PL\}\.For two\-item rankings, i\.e\., pairwise comparisons, PL corresponds to the Bradley–Terry\(Bradley\.1952\)model \(seeHunter\.2004\) defined asP​\(i≻j\)=θiθi\+θj​\.P\(i\\succ j\)=\\tfrac\{\\theta\_\{i\}\}\{\\theta\_\{i\}\+\\theta\_\{j\}\}\\,\\text\{\.\}

#### Mallows Model\.

In contrast to latent utility models, another popular class of models forffis distance\-based models\. The most prominent example is the Mallows model\(Mallows\.1957\), parametrized by\(τ,λ\)∈Θ=𝒮ℐ×ℝ\+\(\\tau,\\lambda\)\\in\\Theta=\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\times\\mathbb\{R\}\_\{\+\}\. The probability of a rankingπ\\piunder the Mallows model\(τ,λ\)\(\\tau,\\lambda\)is

q\(τ,λ\)​\[π\]=1C​\(λ,τ\)​exp⁡\(−λ​d​\(π,τ\)\)​,q^\{\(\\tau,\\lambda\)\}\[\\pi\]=\\tfrac\{1\}\{C\(\\lambda,\\tau\)\}\\exp\(\-\\lambda d\(\\pi,\\tau\)\)\\,\\text\{,\}\(2\)whereC​\(λ,τ\)=∑r∈𝒮ℐexp⁡\(−λ​d​\(r,τ\)\)C\(\\lambda,\\tau\)=\\sum\_\{r\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\}\\exp\(\-\\lambda d\(r,\\tau\)\)is a normalization constant andd:𝒮ℐ×𝒮ℐ→ℝd:\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\times\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\rightarrow\\mathbb\{R\}is a distance defined on rankings\. In practice, both parameters are learned by alternating between optimizingλ\\lambdaand updatingτ\\tauuntil convergence\(Busse\.2007\)\. Throughout this work, we use the Kendall distance fordd\(seeAlvo\.2014\)and denote the resulting hypothesis space byℋM​M\.\\mathcal\{H\}\_\{MM\}\.

#### Ranking by Pairwise Comparison\.

Finally, we consider the approach of*ranking by pairwise comparison*\(RPC\)\(Hullermeier\.2004\)\. In this approach, a ranking model of the formh​\(x\)=f​\(g​\(x\)\)h\(x\)=f\(g\(x\)\)is learned, wheref:\[0,1\]m×m→ℙ​\(𝒮ℐ\)f:\[0,1\]^\{m\\times m\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)is a mapping from pairwise preference probabilities to a predictive ranking distribution andg:𝒳→\[0,1\]m×mg:\\mathcal\{X\}\\rightarrow\[0,1\]^\{m\\times m\}predicts the pairwise preference probabilities for each instancex∈𝒳x\\in\\mathcal\{X\}\. The idea is \(i\) to learn pairwise preference probabilities viaggfor every item pair by reducing the label ranking problem to\(m2\)\\binom\{m\}\{2\}binary classification problems – one for each pair, and \(ii\) to aggregate these pairwise preference probabilities to a predictive ranking \(distribution\) viaff\. For each item pair\(i,j\)\(i,j\), the dataset𝒟i,j=\{\(x,𝟙π⊇i≻j\)∣\(x,π\)∈𝒟\}\\mathcal\{D\}\_\{i,j\}=\\\{\(x,\\mathbbm\{1\}\_\{\\pi\\supseteq i\\succ j\}\)\\mid\(x,\\pi\)\\in\\mathcal\{D\}\\\}is considered for training a binary classifiergi,j:𝒳→\[0,1\]g\_\{i,j\}:\\mathcal\{X\}\\rightarrow\[0,1\]\. This leads to a pairwise probability matrix𝐌​\(x\)=\[gi,j​\(x\)\]i,j∈ℐ\\mathbf\{M\}\(x\)=\[g\_\{i,j\}\(x\)\]\_\{i,j\\in\\mathcal\{I\}\}, withgj,i​\(x\)=1−gi,j​\(x\)g\_\{j,i\}\(x\)=1\-g\_\{i,j\}\(x\)\. While there are several options available forff\(seeFahandar\.2017\), they often yield degenerate predictive distributions\. In ProLR, aggregations forffare of interest that can produce non\-degenerate predictive distributions\. One way to achieve this is by interpreting the entries of𝐌​\(x\)\\mathbf\{M\}\(x\)as Bradley–Terry probabilities and recovering Plackett–Luce weightsθ1,…,θm\\theta\_\{1\},\\dots,\\theta\_\{m\}such thatθiθi\+θj≈gi,j​\(x\),\\frac\{\\theta\_\{i\}\}\{\\theta\_\{i\}\+\\theta\_\{j\}\}\\approx g\_\{i,j\}\(x\),e\.g\., by using the improved Zermelo algorithm\(Newman\.2023\)\.

## 4Calibration in ProLR

To introduce meaningful calibration concepts in label ranking, we follow the very principle of calibration\. We will briefly examine this idea in its two common variants for multi\-class classification, then apply it to label ranking and adapt it to pairwise and top\-k rankings\. All proofs can be found inLABEL:appendix:proofs\.

#### Calibration in Multi\-class Classification

In multi\-class classification, we are given a finite set of outcomes𝒴\\mathcal\{Y\}, for which a modelh:𝒳→ℙ​\(𝒴\)h:\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{Y\}\)is learned\. The outcomes are distributed according to some random variableYY\. We call a model*strongly calibrated*if it holds for ally∈𝒴y\\in\\mathcal\{Y\}that:

∀p∈ℙ​\(𝒴\):P​\(Y=y\|h​\(X\)=p\)=p​\[y\]​\.\\forall p\\in\\mathbb\{P\}\(\\mathcal\{Y\}\):P\(Y=y\\ \|\\ h\(X\)=p\)=p\[y\]\\,\\text\{\.\}\(3\)In words, if a modelhhis strongly calibrated, the probability of classyygiven the predicted distributionh​\(X\)=ph\(X\)=pis exactlyp​\[y\]p\[y\]\. For example, in weather forecasting with𝒴=\{sun,rain,snow\}\\mathcal\{Y\}=\\\{\\text\{sun\},\\text\{rain\},\\text\{snow\}\\\}, strong calibration requires that among all days assigned prediction\(0\.5,0\.3,0\.2\)\(0\.5,0\.3,0\.2\), outcome frequencies match it\.

One can weaken this notion naturally by restricting the condition to the respective entry ofh​\(X\)h\(X\)\. Formally, a classifier is called*weakly calibrated*if it holds for ally∈𝒴y\\in\\mathcal\{Y\}that

∀α∈\[0,1\]:P​\(Y=y\|h​\(X\)​\[y\]=α\)=α​\.\\forall\\alpha\\in\[0,1\]:P\(Y=y\\ \|\\ h\(X\)\[y\]=\\alpha\)=\\alpha\\,\\text\{\.\}\(4\)In the weather example, among days where rain is predicted with probabilityα\\alpha, it rains on anα\\alpha\-fraction \(and analogously for sun and snow\)\. These can diverge because calibration pools by the outcome’s probability: predictions\(0\.6,0\.3,0\.1\)\(0\.6,0\.3,0\.1\)and\(0\.4,0\.3,0\.3\)\(0\.4,0\.3,0\.3\)\(perhaps reflecting different seasonal patterns\) both assign0\.30\.3to rain, so weak calibration pools them\. If rain frequencies are0\.20\.2and0\.40\.4respectively, they average to0\.30\.3\(weak holds\), but neither full prediction matches its true distribution \(strong fails\)\.

#### Calibration for Full Rankings in ProLR

Translating the concepts to label ranking can be easily done by viewing the problem as a multi\-class classification problem, where each ranking is interpreted as a class\. Thus, for an item setℐ\\mathcal\{I\}with\|ℐ\|=m\|\\mathcal\{I\}\|=m, there existm\!m\!different rankings, which correspond tom\!m\!different classes\. With this, we introduce the notion of full\-rank calibration \([Definition 1](https://arxiv.org/html/2605.30447#Thmdefinition1)\), which is the label ranking equivalent to strong calibration in the sense of multi\-class classification\.

###### Definition 1\.

A modelh∈ℋh\\in\\mathcal\{H\}is*full\-rank calibrated*if and only if it holds for allπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}that

∀p∈ℙ​\(𝒮ℐ\):P​\(Π=π\|h​\(X\)=p\)=p​\[π\]\.\\forall p\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):\\;P\(\\Pi=\\pi\\ \|\\ h\(X\)=p\)=p\[\\pi\]\\,\.\(5\)

In a similar fashion, we introduce the notion of*rankwise calibrated*\([Definition 2](https://arxiv.org/html/2605.30447#Thmdefinition2)\), which considers conditioning only on the probability of a specific ranking, instead of a full probability distribution as in full\-rank calibration\.

###### Definition 2\.

A modelh∈ℋh\\in\\mathcal\{H\}is*rankwise*calibrated if and only if it holds for allπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}that

∀α∈\[0,1\]:P​\(Π=π\|h​\(X\)​\[π\]=α\)=α\.\\forall\\alpha\\in\[0,1\]:\\;P\(\\Pi=\\pi\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)=\\alpha\\,\.\(6\)

As an example, consider predicting musical taste with items*rock*,*pop*, and*classical*\. Rankwise calibration requires that among all people for whom a ranking likerock≻pop≻classical\\text\{rock\}\\succ\\text\{pop\}\\succ\\text\{classical\}is predicted with probabilityα\\alpha, it occurs for anα\\alpha\-fraction\. Full\-rank calibration requires that among all people assigned the same distribution over all six rankings, empirical frequencies match it\. As with weather, these can diverge because rankwise calibration pools by one ranking’s probability: different full predictions \(e\.g\., different age groups\) assigning the same probability torock≻pop≻classical\\text\{rock\}\\succ\\text\{pop\}\\succ\\text\{classical\}are pooled, allowing subgroups to average correctly while neither matches the full prediction\. One can easily show that full\-rank calibration implies rankwise calibration, as formulated in[Theorem 4\.1](https://arxiv.org/html/2605.30447#S4.Thmtheorem1)\.

###### Theorem 4\.1\.

For an item setℐ\\mathcal\{I\}and rankings𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}, it holds

- h​is full\-rank calibrated⇒h​is rankwise calibrated\.h\\ \\text\{is full\-rank calibrated \}\\Rightarrow h\\ \\text\{is rankwise calibrated\.\}

Intuitively, a model can be rankwise calibrated yet not full\-rank calibrated, as the latter is a more demanding property\. AsLABEL:th:rankwise\_not\_full\_rank\_calibshows, this intuition holds under some mild assumptions satisfied by PL and Mallows models\.

#### Calibration for Key Modalities in ProLR\.

As noted in the introduction, the classification viewpoint ignores relationships between rankings and important modalities such as pairwise and top\-k rankings\. To address this, we introduce calibration notions for sub\-rankings and top\-k rankings, projecting full rankings onto the sub\-rankings of interest and marginalizing the resulting distributions accordingly\.

Consider a subset of itemsℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}with\|ℬ\|≥2\|\\mathcal\{B\}\|\\geq 2for whichρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}is a partial ranking of interest\. Then we denote by

𝒫ℬ​\(ρ\)=\{π∈𝒮ℐ\|π⊇ρ\}​,\\mathcal\{P\}\_\{\\mathcal\{B\}\}\(\\rho\)=\\\{\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\ \|\\ \\pi\\supseteq\\rho\\\}\\,\\text\{,\}\(7\)the set of rankingsπ\\pithat containρ\\rhoas a partial ranking\. The set𝒫​\(ρ\)\\mathcal\{P\}\(\\rho\)contains all rankingsπ\\pifor which the items inℬ\\mathcal\{B\}are ranked according toρ\\rho, while the remaining items inℐ∖ℬ\\mathcal\{I\}\\setminus\\mathcal\{B\}can be ranked arbitrarily\.

Another special case of sub\-rankings are top\-k rankings, which only consider the items ranked in the firstkkpositions\. For these, we introduce

𝒯ℬ​\(ρ\)=\{π∈𝒮ℐ\|∃r∈𝒮ℐ∖ℬ:π=ρ≻r\}​,\\mathcal\{T\}\_\{\\mathcal\{B\}\}\(\\rho\)=\\\{\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\ \|\\ \\exists r\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\\setminus\\mathcal\{B\}\}:\\pi=\\rho\\succ r\\\}\\,\\text\{,\}\(8\)for eachℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}with\|ℬ\|≥1\|\\mathcal\{B\}\|\\geq 1andρ∈𝒮ℬ\.\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}\.In words, the set of rankings that coincide withρ\\rhoon the top positions\. Here, the length of the top positions is determined by the cardinality ofℬ\\mathcal\{B\}\.

As an example to illustrate both concepts, considerℐ=\{1,2,3\}\\mathcal\{I\}=\\\{1,2,3\\\}andℬ=\{1,2\}\\mathcal\{B\}=\\\{1,2\\\}withρ=1≻2\\rho=1\\succ 2\. Here, we obtain

𝒫ℬ​\(ρ\)\\displaystyle\\mathcal\{P\}\_\{\\mathcal\{B\}\}\(\\rho\)=\{1≻2≻3,1≻3≻2,3≻1≻2\},\\displaystyle=\\\{1\\succ 2\\succ 3,\\;1\\succ 3\\succ 2,\\;3\\succ 1\\succ 2\\\}\\,\\text\{,\}𝒯ℬ​\(ρ\)\\displaystyle\\mathcal\{T\}\_\{\\mathcal\{B\}\}\(\\rho\)=\{1≻2≻3\}​\.\\displaystyle=\\\{1\\succ 2\\succ 3\\\}\\,\\text\{\.\}Notice that𝒯​\(ρ\)⊆𝒫​\(ρ\)\\mathcal\{T\}\(\\rho\)\\subseteq\\mathcal\{P\}\(\\rho\)holds in general and𝒫​\(ρ\)=𝒯​\(ρ\)=\{ρ\}\\mathcal\{P\}\(\\rho\)=\\mathcal\{T\}\(\\rho\)=\\\{\\rho\\\}only in the case ofℬ=ℐ\\mathcal\{B\}=\\mathcal\{I\}\. In the remainder of this work, we omitℬ\\mathcal\{B\}from the subscripts of𝒫​\(ρ\)\\mathcal\{P\}\(\\rho\)and𝒯​\(ρ\)\\mathcal\{T\}\(\\rho\), whereℬ\\mathcal\{B\}is clear from context\. As a matter of fact, the cardinality ofℬ\\mathcal\{B\}will play a more central role in what follows, and we assume throughout that\|ℬ\|=k\|\\mathcal\{B\}\|=k\. Accordingly, we refer to𝒫\\mathcal\{P\}as the*sub\-k subsets*and𝒯\\mathcal\{T\}as the*top\-k subsets*\.

These two sets correspond to the respective modality, i\.e\., sub\-k rankings or top\-k rankings\. The latter are of particular interest, as the evaluation of predictions in downstream tasks often considers top\-rated items \(top\-k\) or comparisons of specific subsets of items \(sub\-k\), despite models in ProLR predicting full rankings\. Alternative subsets, such as worst\-k, are much less practically relevant and therefore not considered here\. Next, we introduce a probability distributionp∈ℙ​\(𝒮ℐ\)p\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)suitable marginalizations for both subsets𝒫\\mathcal\{P\}and𝒯\\mathcal\{T\}\.

###### Definition 3\.

For a set of itemsℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}with\|ℬ\|=k\|\\mathcal\{B\}\|=kand sub\-rankingρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}, the*sub\-k marginal distribution*of a distributionp∈ℙ​\(𝒮ℐ\)p\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)is

p\|\|𝒫​\[ρ\]=∑π∈𝒫​\(ρ\)p​\[π\]​,p\_\{\|\|\\mathcal\{P\}\}\[\\rho\]=\\sum\\nolimits\_\{\\pi\\in\\mathcal\{P\}\(\\rho\)\}\\;p\[\\pi\]\\,\\text\{,\}and the*top\-k marginal distribution*is given by

p\|\|𝒯​\[ρ\]=∑π∈𝒯​\(ρ\)p​\[π\]​\.p\_\{\|\|\\mathcal\{T\}\}\[\\rho\]=\\sum\\nolimits\_\{\\pi\\in\\mathcal\{T\}\(\\rho\)\}\\;p\[\\pi\]\\,\\text\{\.\}

Note that the sub\-k marginal distribution is an element ofℙ​\(𝒮ℬ\),\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\),i\.e\., a probability distribution on the set of rankings over the item setℬ,\\mathcal\{B\},while the top\-k marginal distribution is an element ofℙ​\(∪ℬ⊂ℐ:\|ℬ\|=k𝒮ℬ\),\\mathbb\{P\}\(\\cup\_\{\\mathcal\{B\}\\subset\\mathcal\{I\}:\|\\mathcal\{B\}\|=k\}\\mathcal\{S\}\_\{\\mathcal\{B\}\}\),i\.e\., a probability distribution on the set of all rankings of sizekkconsisting of elements inℐ\.\\mathcal\{I\}\.We now consider calibration for the above marginal distributions with respect to a modelhhthat outputs distributions on𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}\. First, we define the \(full\-rank\) sub\-k calibration again by conditioning on an entire probability distributionq∈ℙ​\(𝒮ℬ\)q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\)\. Here, we consider the sub\-k marginal ofh​\(X\)h\(X\)for all sub\-k subsets\.

###### Definition 4\.

A modelh∈ℋh\\in\\mathcal\{H\}is*sub\-k*calibrated if for everyℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}and\|ℬ\|=k\|\\mathcal\{B\}\|=kit holds for allρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}that

∀q∈ℙ​\(𝒮ℬ\):P​\(Π\|\|ℬ=ρ\|h\|\|𝒫​\(X\)=q\)=q​\[ρ\],\\forall q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\):\\;P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)=q\[\\rho\],\(9\)whereΠ\|\|ℬ\\Pi\_\{\|\|\\mathcal\{B\}\}denotes the projection ofΠ\\Pionto the items inℬ\\mathcal\{B\}\.

Note that fork=2k=2we recover the modality of pairwise rankings, which is of special interest in Reinforcement Learning from Human Feedback \(RLHF\)\. Similarly, we introduce*top\-k*calibration by considering the top\-k marginal ofh​\(X\)h\(X\)for all top\-k subsets\.

###### Definition 5\.

A modelh∈ℋh\\in\\mathcal\{H\}is*top\-k*calibrated if for everyℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}and\|ℬ\|=k\|\\mathcal\{B\}\|=kit holds for allρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}and allq∈ℙ​\(∪ℬ⊂ℐ:\|ℬ\|=k𝒮ℬ\):q\\in\\mathbb\{P\}\(\\cup\_\{\\mathcal\{B\}\\subset\\mathcal\{I\}:\|\\mathcal\{B\}\|=k\}\\mathcal\{S\}\_\{\\mathcal\{B\}\}\):

P​\(Π\|\|ℬ=ρ\|h\|\|𝒯​\(X\)=q\)=q​\[ρ\]\.P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)=q\[\\rho\]\\,\.\(10\)

In other words, the probability for all events, where the sub\-k \(top\-k\) marginalised observed ranking isρ\\rhoconditional on the sub\-k \(top\-k\) marginalised model predictingqq, is exactly the probability thatρ\\rhohas withinqq\. For example, in cases where the model predicts the probabilitypi,jp\_\{i,j\}for the pairwise comparisoni≻ji\\succ j\(e\.g\.,rock≻\\succpop\) and we consider these pairwise comparisons precisely, they occur in exactlypi,jp\_\{i,j\}of the cases\. Another way to interpret sub\-k or top\-k calibration of a modelhhis that its sub\-k or top\-k marginal distribution is full\-rank calibrated on every subsetℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}of sizekk\.

#### Relationship of strong calibration notions in ProLR\.

Intuitively, both sub\-k and top\-k calibration pose a weaker notion than full\-rank calibration, as they both consider sub\-rankings of sizekkinstead of full\-rankings of sizemm\. Indeed, one can show that full\-rank calibration implies both sub\-k and top\-k calibration, reflected in the following theorem\.

###### Theorem 4\.2\.

For every finite item setℐ\\mathcal\{I\}andk≤mk\\leq m:

- \(i\)h​is full\-rank calibrated⇒h​is sub\-k calibrated,h\\ \\text\{is full\-rank calibrated \}\\Rightarrow h\\ \\text\{is sub\-k calibrated,\}
- \(ii\)h​is full\-rank calibrated⇒h​is top\-k calibrated\.h\\ \\text\{is full\-rank calibrated \}\\Rightarrow h\\ \\text\{is top\-k calibrated\.\}

Considering the converse direction, we can show that neither sub\-k nor top\-k calibration implies full\-rank calibration\. Intuitively, this stems from the fact that both sub\-k and top\-k calibration consider only sub\-rankings of sizekk, hence ignoring information about the remaining items\. Therefore, modifications on the entire distributionh​\(X\)∈ℙ​\(𝒮ℐ\)h\(X\)\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)can cancel themselves out in the sub\-k or top\-k marginal distributions, yet lead to violations of full\-rank calibration\.

Table 1:Counterexample showing that sub\-2 calibration does not imply full\-rank calibration\. Sub\-2 marginals satisfy calibration for bothx1x\_\{1\}andx2x\_\{2\}, whereas the full ranking probabilities violate calibration for both predictions\.[Section4](https://arxiv.org/html/2605.30447#S4.SS0.SSS0.Px4)shows an example of a model that is sub\-kkcalibrated yet not full\-rank calibrated\. An example of a top\-kkcalibrated model that is not full\-rank calibrated is given inLABEL:tab:counterexample\_top1\_not\_rankwise\. More general results dealing with all values ofkkare Theorems[A\.4](https://arxiv.org/html/2605.30447#A1.Thmtheorem4)and[A\.5](https://arxiv.org/html/2605.30447#A1.Thmtheorem5)\. Note that neither sub\-k nor top\-k calibration implies the other, as the former considers all rankings containing a specific sub\-ranking, while the latter only considers rankings where the sub\-ranking appears in the top\-k positions\. Again, modifications on the entire distributionh​\(X\)∈ℙ​\(𝒮ℐ\)h\(X\)\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)can cancel themselves out in one of the marginal distributions, yet lead to violations in the other\. Thus, a model can be sub\-k calibrated yet not top\-k calibrated, and vice versa \(see Theorems[A\.2](https://arxiv.org/html/2605.30447#A1.Thmtheorem2)and[A\.3](https://arxiv.org/html/2605.30447#A1.Thmtheorem3)\)\. This supports the view that both notions cover different aspects of calibration\.

It remains to clarify how the weaker rankwise calibration relates to sub\-k or top\-k calibration\. One can show that neither sub\-k nor top\-k calibration implies rankwise calibration, and vice versa\. Thus, a model can be rankwise calibrated, yet not sub\-k calibrated \([TheoremA\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7)\) or top\-k calibrated \([TheoremA\.8](https://arxiv.org/html/2605.30447#A1.Thmtheorem8)\)\. Similarly, a model can be sub\-k calibrated but not rankwise calibrated \([TheoremA\.13](https://arxiv.org/html/2605.30447#A1.Thmtheorem13)\) or can be top\-k calibrated but not rankwise calibrated \([TheoremA\.14](https://arxiv.org/html/2605.30447#A1.Thmtheorem14)\)\. These results can be intuitively explained: rankwise calibration conditions on the probability of a specific ranking, while sub\-k and top\-k calibration condition on marginal distributions over sub\-rankings\. Thus, roughly speaking, the notions are orthogonal and therefore do not imply anything about one another\.

Table 2:Rankwise calibrated model that is neither sub\-2 nor top\-1 calibrated\. Sub\-2 marginals violate calibration forx2x\_\{2\}and top\-1 marginals violate calibration for bothx1x\_\{1\}andx2\.x\_\{2\}\.An example of a model being rankwise calibrated but neither sub\-k nor top\-k calibrated is illustrated in[Section4](https://arxiv.org/html/2605.30447#S4.SS0.SSS0.Px4)\.

#### Weaker calibration notions in ProLR\.

Similar to the transition from full\-rank calibration to rankwise calibration, we can further weaken the conditions of sub\-k and top\-k calibration by conditioning only on the probability of a specific sub\-ranking instead of the entire marginal distribution\.

###### Definition 6\.

A modelh∈ℋh\\in\\mathcal\{H\}is*rankwise sub\-k*calibrated if for everyℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}with\|ℬ\|=k\|\\mathcal\{B\}\|=kit holds for allπ∈𝒮ℬ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}that

∀α∈\[0,1\]:P​\(Π\|\|ℬ=π\|h\|\|𝒫​\(X\)​\[π\]=α\)=α\.\\forall\\alpha\\in\[0,1\]:\\;P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\pi\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)\[\\pi\]=\\alpha\)=\\alpha\\,\.\(11\)

###### Definition 7\.

A modelh∈ℋh\\in\\mathcal\{H\}is*rankwise top\-k*calibrated if for everyℬ\\mathcal\{B\}with\|ℬ\|=k\|\\mathcal\{B\}\|=kit holds for allπ∈𝒮ℬ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}that

∀α∈\[0,1\]:P​\(Π\|\|ℬ=π\|h\|\|𝒯​\(X\)​\[π\]=α\)=α\.\\forall\\alpha\\in\[0,1\]:\\;P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\pi\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)\[\\pi\]=\\alpha\)=\\alpha\\,\.\(12\)

Obviously, it holds that sub\-k calibration implies rankwise sub\-k calibration, and similarly for top\-k and rankwise top\-k calibration, as formulated in the following theorem\.

###### Theorem 4\.3\.

For every finite item setℐ\\mathcal\{I\}andk≤mk\\leq m:

- \(i\)h​is sub\-k calibrated⇒h​is rankwise sub\-k calibratedh\\ \\text\{is sub\-k calibrated \}\\Rightarrow h\\ \\text\{is rankwise sub\-k calibrated\}
- \(ii\)h​is top\-k calibrated⇒h​is rankwise top\-k calibratedh\\ \\text\{is top\-k calibrated \}\\Rightarrow h\\ \\text\{is rankwise top\-k calibrated\}

Both rankwise sub\-k and rankwise top\-k calibration pose a weaker notion than their full marginal counterparts, as they consider only conditioning on the probability of a specific sub\-ranking instead of the entire marginal distribution\. Due to this, one can show that neither rankwise sub\-k nor rankwise top\-k calibration implies \(full\-rank\) sub\-k or top\-k calibration, respectively\.

We chooseδ\\deltain such a way that the above adjustments again lead to valid probability distributions, i\.e\. max\(P\(Π=τ— X=x\_i\), P\(Π=π— X=x\_j\)\) ≤δ≤1 \- max\(P\(Π=π— X=x\_i\), P\(Π=τ— X=x\_j\)\)\.For all other data pointsl∈\{1,…,n\}∖\{i,j\}l\\in\\\{1,\\dots,n\\\}\\setminus\\\{i,j\\\}we haveP​\(Π\|X=zl\)=P​\(Π\|X=xl\)P\(\\Pi\\ \|\\ X=z\_\{l\}\)=P\(\\Pi\\ \|\\ X=x\_\{l\}\)\. Obviously,hhis still rankwise calibrated onz1,…,znz\_\{1\},\\dots,z\_\{n\}as in group𝒢\\mathcal\{G\}the changesδ\\deltaand−δ\-\\deltacancel themselves out\. Indeed, by writingPz1,…,znP\_\{z\_\{1\},\\ldots,z\_\{n\}\}andPx1,…,xnP\_\{x\_\{1\},\\ldots,x\_\{n\}\}to indicate the considered feature space, we obtain

Pz1,…,zn​\(Π=π\|h​\(X\)​\[π\]=α\)\\displaystyle P\_\{z\_\{1\},\\ldots,z\_\{n\}\}\(\\Pi=\\pi\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)=∑z∈𝒵h​\(z\)​\[π\]=αP​\(Π=π\|X=z\)​P​\(X=z\|h​\(X\)​\[π\]=α\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}z\\in\\mathcal\{Z\}\\\\ h\(z\)\[\\pi\]=\\alpha\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=z\)P\(X=z\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\(28\)=∑z∈𝒵∖\{zi,zj\}h​\(z\)​\[π\]=αP​\(Π=π\|X=z\)​P​\(X=z\|h​\(X\)​\[π\]=α\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}z\\in\\mathcal\{Z\}\\setminus\\\{z\_\{i\},z\_\{j\}\\\}\\\\ h\(z\)\[\\pi\]=\\alpha\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=z\)P\(X=z\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\+P​\(Π=π\|X=zi\)​P​\(X=zi\|h​\(X\)​\[π\]=α\)\\displaystyle\\quad\+P\(\\Pi=\\pi\\ \|\\ X=z\_\{i\}\)P\(X=z\_\{i\}\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\+P​\(Π=π\|X=zj\)​P​\(X=zj\|h​\(X\)​\[π\]=α\)\\displaystyle\\quad\+P\(\\Pi=\\pi\\ \|\\ X=z\_\{j\}\)P\(X=z\_\{j\}\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\(29\)=∑x∈𝒳∖\{xi,xj\}h​\(x\)​\[π\]=αP​\(Π=π\|X=x\)​P​\(X=x\|h​\(X\)​\[π\]=α\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\setminus\\\{x\_\{i\},x\_\{j\}\\\}\\\\ h\(x\)\[\\pi\]=\\alpha\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=x\)P\(X=x\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\+\(P​\(Π=π\|X=xi\)\+δ\)​P​\(X=xi\|h​\(X\)​\[π\]=α\)\\displaystyle\\quad\+\\bigl\(P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\+\\delta\\bigr\)P\(X=x\_\{i\}\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\+\(P​\(Π=π\|X=xj\)−δ\)​P​\(X=xj\|h​\(X\)​\[π\]=α\)\\displaystyle\\quad\+\\bigl\(P\(\\Pi=\\pi\\ \|\\ X=x\_\{j\}\)\-\\delta\\bigr\)P\(X=x\_\{j\}\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\(30\)=∑x∈𝒳h​\(x\)​\[π\]=αP​\(Π=π\|X=x\)​P​\(X=x\|h​\(X\)​\[π\]=α\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\\\ h\(x\)\[\\pi\]=\\alpha\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=x\)P\(X=x\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\(31\)=Px1,…,xn​\(Π=π\|h​\(X\)​\[π\]=α\)\\displaystyle=P\_\{x\_\{1\},\\ldots,x\_\{n\}\}\(\\Pi=\\pi\\ \|\\ h\(X\)\[\\pi\]=\\alpha\)\(32\)=α\\displaystyle=\\alpha\(33\)
Here, Steps \([28](https://arxiv.org/html/2605.30447#A1.E28)\) and \([32](https://arxiv.org/html/2605.30447#A1.E32)\) are by the law of total probability\. Steps \([29](https://arxiv.org/html/2605.30447#A1.E29)\) and \([30](https://arxiv.org/html/2605.30447#A1.E30)\) use straightforward rewrites, while \([31](https://arxiv.org/html/2605.30447#A1.E31)\) uses the assumption onxix\_\{i\}andxjx\_\{j\}\. Finally, using thathhis rankwise calibrated on𝒳\\mathcal\{X\}yields \([31](https://arxiv.org/html/2605.30447#A1.E31)\)\.

More interestingly,hhis not full\-rank calibrated, as theδ\\deltachanges do not cancel themselves\. Proceeding similarly as in the display before, one obtains

Pz1,…,zn​\(Π=π\|h​\(X\)=q\)\\displaystyle P\_\{z\_\{1\},\\ldots,z\_\{n\}\}\(\\Pi=\\pi\\ \|\\ h\(X\)=q\)=∑x∈𝒳∖\{xi,xj\}h​\(X\)=qP​\(Π=π\|X=x\)​P​\(X=x\|h​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\setminus\\\{x\_\{i\},x\_\{j\}\\\}\\\\ h\(X\)=q\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=x\)P\(X=x\\ \|\\ h\(X\)=q\)\+\(P​\(Π=π\|X=xi\)\+δ\)​P​\(X=xi\|h​\(X\)=q\)\\displaystyle\\quad\+\\bigl\(P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\+\\delta\\bigr\)P\(X=x\_\{i\}\\ \|\\ h\(X\)=q\)\(34\)=∑x∈𝒳h​\(X\)=qP​\(Π=π\|X=x\)​P​\(X=x\|h​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\\\ h\(X\)=q\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=x\)P\(X=x\\ \|\\ h\(X\)=q\)\+δ​P​\(X=xi\|h​\(X\)=q\)\\displaystyle\\quad\+\\delta P\(X=x\_\{i\}\\ \|\\ h\(X\)=q\)\(35\)=Px1,…,xn​\(Π=π\|h​\(X\)=q\)\+δ​P​\(X=xi\|h​\(X\)=q\)\\displaystyle=P\_\{x\_\{1\},\\ldots,x\_\{n\}\}\(\\Pi=\\pi\\ \|\\ h\(X\)=q\)\+\\delta P\(X=x\_\{i\}\\ \|\\ h\(X\)=q\)\(36\)=q​\[π\]\+δ​P​\(X=xi\|h​\(X\)=q\)\\displaystyle=q\[\\pi\]\+\\delta P\(X=x\_\{i\}\\ \|\\ h\(X\)=q\)\(37\)≠q​\[π\]\\displaystyle\\neq q\[\\pi\]\(38\)
Here, we use in \([37](https://arxiv.org/html/2605.30447#A1.E37)\) thathhis full\-rank calibrated on𝒳\\mathcal\{X\}and thatxi∈ℱx\_\{i\}\\in\\mathcal\{F\}for \([38](https://arxiv.org/html/2605.30447#A1.E38)\)\. ∎

###### Theorem A\.2\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k<m:∃h∈𝒳→ℙ​\(𝒮ℐ\):h​sub\-k calibrated∧h​nottop\-k calibrated\.\\forall k<m:\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{sub\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} top\-k calibrated\}\\text\{\.\}

###### Proof of Theorem[A\.2](https://arxiv.org/html/2605.30447#A1.Thmtheorem2)\.

Letℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}be a set of items,ρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}, and letp∈ℙ​\(𝒮ℐ\)p\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\)andq∈ℙ​\(𝒮ℬ\)q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\)be arbitrary but fixed\. Without loss of generality555Choosing differentτ1,τ2\\tau\_\{1\},\\tau\_\{2\}is possible by relabeling the items, such thatτ1=i1≻i2≻…\\tau\_\{1\}=i\_\{1\}\\succ i\_\{2\}\\succ\\dotsandτ2=im≻im−1≻…\\tau\_\{2\}=i\_\{m\}\\succ i\_\{m\-1\}\\succ\\dots\.letτ1=i1≻i2≻⋯≻im\\tau\_\{1\}=i\_\{1\}\\succ i\_\{2\}\\succ\\dots\\succ i\_\{m\}andτ2=im≻⋯≻i2≻i1\\tau\_\{2\}=i\_\{m\}\\succ\\dots\\succ i\_\{2\}\\succ i\_\{1\}\. Lethhbe full\-rank calibrated on𝒳\\mathcal\{X\}and such that there is a subset𝒳0=\{x1,…,xn\}\\mathcal\{X\}\_\{0\}=\\\{x\_\{1\},\\dots,x\_\{n\}\\\}of𝒳\\mathcal\{X\}with1<n<∞1<n<\\inftypoints such thath​\(x\)h\(x\)is non\-degenerated for eachx∈𝒳0x\\in\\mathcal\{X\}\_\{0\}\. Recall that sub\-k calibration \([9](https://arxiv.org/html/2605.30447#S4.E9)\) satisfies P\(Π\_——B=τ— h\_——P\(X\)=q\) = q\[τ\] = ∑\_x ∈Xh\_——P\(x\)=qP\(Π\_——B=τ— X=x\) P\(X=x — h\_——P\(X\)=q\) and top\-k calibration \([10](https://arxiv.org/html/2605.30447#S4.E10)\) satisfies P\(Π\_——B=ρ— h\_——T\(X\)=q\) = q\[ρ\] = ∑\_x ∈Xh\_——T\(x\)=qP\(Π\_——B=ρ— X=x\) P\(X=x — h\_——T\(X\)=q\)\.Particularly ask<mk<mit holdsm\!k\!=\|𝒫​\(ρ\)\|\>\|𝒯​\(ρ\)\|=\(m−k\)\!\\frac\{m\!\}\{k\!\}=\|\\mathcal\{P\}\(\\rho\)\|\>\|\\mathcal\{T\}\(\\rho\)\|=\(m\-k\)\!\. We now construct a new feature subset𝒵=\{z1,…,zn\}\\mathcal\{Z\}=\\\{z\_\{1\},\\dots,z\_\{n\}\\\}such that

P​\(X=zi\|h\|\|𝒯​\(X\)=q\)=P​\(X=xi\|h\|\|𝒯​\(X\)=q\)\>0\\displaystyle P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)=P\(X=x\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)\>0\(39\)and

P​\(Π=π\|X=zi\)=\{P​\(Π=π\|X=xi\)\+δif​π=τ1​or​π=τ2P​\(Π=π\|X=xi\)−δ\|𝒫​\(ρ\)\|−1if​π∈𝒫​\(ρ\)∖\{τ1,τ2\}P​\(Π=π\|X=xi\)−δm\!−\|𝒫​\(ρ\)\|−1otherwise\\displaystyle P\(\\Pi=\\pi\\ \|\\ X=z\_\{i\}\)=\\begin\{cases\}P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\+\\delta&\\text\{if \}\\pi=\\tau\_\{1\}\\text\{ or \}\\pi=\\tau\_\{2\}\\\\ P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{if \}\\pi\\in\\mathcal\{P\}\(\\rho\)\\setminus\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\\\ P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\-\\frac\{\\delta\}\{m\!\-\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{otherwise\}\\end\{cases\}\(40\)for all1≤i≤n1\\leq i\\leq n\. Hereδ\\deltais chosen such that

0<δ≤mini≤n\(\\displaystyle 0<\\delta\\leq\\min\_\{i\\leq n\}\\Big\(minπ∈\{τ1,τ2\}⁡1−P​\(Π=π\|X=xi\),minπ∈𝒫​\(ρ\)τ∉\{τ1,τ2\}⁡\(\|𝒫​\(ρ\)\|−1\)​P​\(Π=π\|X=xi\)\\displaystyle\\min\_\{\\pi\\in\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\}\\,1\-P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\),\\,\\min\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{P\}\(\\rho\)\\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}\(\|\\mathcal\{P\}\(\\rho\)\|\-1\)P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)minπ∈𝒮ℐ∖𝒫​\(ρ\)τ∉\{τ1,τ2\}\(m\!−\|𝒫\(ρ\)\|−1\)P\(Π=π\|X=xi\)\)\.\\displaystyle\\quad\\min\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\setminus\\mathcal\{P\}\(\\rho\)\\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}\(m\!\-\|\\mathcal\{P\}\(\\rho\)\|\-1\)P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\\Big\)\\,\\text\{\.\}This ensures that the probability on the left hand side of \([40](https://arxiv.org/html/2605.30447#A1.E40)\) takes only values between 0 and 1\. The redistribution of the probability mass is designed in such a way that the left hand side of \([40](https://arxiv.org/html/2605.30447#A1.E40)\) is still a valid distribution summing to 1\. Indeed, note that eitherτ1\\tau\_\{1\}orτ2\\tau\_\{2\}can be an element of𝒫​\(ρ\)\\mathcal\{P\}\(\\rho\)\. Indeed, for eachzi∈𝒵z\_\{i\}\\in\\mathcal\{Z\}we have

∑π∈𝒮ℐ\\displaystyle\\sum\_\{\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\}P​\(Π=π\|X=zi\)\\displaystyle P\(\\Pi=\\pi\\ \|\\ X=z\_\{i\}\)=P​\(Π=τ1\|X=xi\)\+P​\(Π=τ2\|X=xi\)\+2​δ\+∑π∈𝒫​\(ρ\)τ∉\{τ1,τ2\}P​\(Π=π\|X=zi\)\+∑π∈𝒮ℐ∖𝒫​\(ρ\)τ∉\{τ1,τ2\}P​\(Π=π\|X=zi\)\\displaystyle=P\(\\Pi=\\tau\_\{1\}\\ \|\\ X=x\_\{i\}\)\+P\(\\Pi=\\tau\_\{2\}\\ \|\\ X=x\_\{i\}\)\+2\\delta\+\\sum\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{P\}\(\\rho\)\\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=z\_\{i\}\)\+\\sum\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\setminus\\mathcal\{P\}\(\\rho\)\\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}P\(\\Pi=\\pi\\ \|\\ X=z\_\{i\}\)=∑π∈𝒮ℐP​\(Π=π\|X=xi\)\+2​δ−∑π∈𝒫​\(ρ\)τ∉\{τ1,τ2\}δ\|𝒫​\(ρ\)\|−1−∑π∈𝒮ℐ∖𝒫​\(ρ\)τ∉\{τ1,τ2\}δm\!−\|𝒫​\(ρ\)\|−1\\displaystyle=\\sum\_\{\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\}P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\+2\\delta\-\\sum\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{P\}\(\\rho\)\\ \\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}\-\\sum\_\{\\begin\{subarray\}\{c\}\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\\setminus\\mathcal\{P\}\(\\rho\)\\\\ \\tau\\notin\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{subarray\}\}\\frac\{\\delta\}\{m\!\-\|\\mathcal\{P\}\(\\rho\)\|\-1\}=∑π∈𝒮ℐP​\(Π=π\|X=xi\)⏟=1\+2​δ−δ−δ\\displaystyle=\\underbrace\{\\sum\_\{\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}\}P\(\\Pi=\\pi\\ \|\\ X=x\_\{i\}\)\}\_\{=1\}\+2\\delta\-\\delta\-\\delta=1,\\displaystyle=1,where we used that the second sum has\|𝒫​\(ρ\)\|−1\|\\mathcal\{P\}\(\\rho\)\|\-1many summands, since only eitherτ1\\tau\_\{1\}orτ2\\tau\_\{2\}are excluded, while the third sum hasm\!−\|𝒫​\(ρ\)\|−1m\!\-\|\\mathcal\{P\}\(\\rho\)\|\-1many summands, since𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}hasm\!m\!many elements of which\|𝒫​\(ρ\)\|\|\\mathcal\{P\}\(\\rho\)\|many are removed and in addition eitherτ1\\tau\_\{1\}orτ2\\tau\_\{2\}are excluded\. Now it holds thathhis sub\-k calibrated on𝒳∖𝒳0∪𝒵\\mathcal\{X\}\\setminus\\mathcal\{X\}\_\{0\}\\cup\\mathcal\{Z\}:

P\(Π\|\|ℬ=ρ\|\\displaystyle P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\h\|\|𝒫\(X\)=q\)\\displaystyle h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)=∑x∈𝒳∖𝒳0∪𝒵h\|\|𝒫​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=x\)​P​\(X=x\|h\|\|𝒫​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\setminus\\mathcal\{X\}\_\{0\}\\cup\\mathcal\{Z\}\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=x\)\\,P\(X=x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\(41\)=∑x∈𝒳∖𝒳0h\|\|𝒫​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=x\)​P​\(X=x\|h\|\|𝒫​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\setminus\\mathcal\{X\}\_\{0\}\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=x\)\\,P\(X=x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\+∑i∈\[n\]h\|\|𝒫​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=zi\)​P​\(X=zi\|h\|\|𝒫​\(X\)=q\)\\displaystyle\\quad\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=z\_\{i\}\)\\,P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\(42\)=∑x∈𝒳∖𝒳0h\|\|𝒫​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=x\)​P​\(X=x\|h\|\|𝒫​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\setminus\\mathcal\{X\}\_\{0\}\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=x\)\\,P\(X=x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\+∑i∈\[n\]h\|\|𝒫​\(x\)=qP​\(X=zi\|h\|\|𝒫​\(X\)=q\)⋅P​\(Π\|\|ℬ=ρ\|X=zi\)\\displaystyle\\quad\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\\cdot P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=z\_\{i\}\)\(43\)=∑x∈𝒳h\|\|𝒫​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=x\)​P​\(X=x\|h\|\|𝒫​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=x\)\\,P\(X=x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\+∑i∈\[n\]h\|\|𝒫​\(x\)=qP​\(X=zi\|h\|\|𝒫​\(X\)=q\)⋅∑π∈𝒫​\(ρ\)\{δif​π∈\{τ1,τ2\}−δ\|𝒫​\(ρ\)\|−1if​π∈𝒫​\(ρ\)∖\{τ1,τ2\}\\displaystyle\\quad\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\\cdot\\sum\_\{\\pi\\in\\mathcal\{P\}\(\\rho\)\}\\begin\{cases\}\\delta&\\text\{if \}\\pi\\in\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\\\ \-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{if \}\\pi\\in\\mathcal\{P\}\(\\rho\)\\setminus\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{cases\}\(44\)=q​\[ρ\]\+∑i∈\[n\]h\|\|𝒫​\(x\)=qP​\(X=zi\|h\|\|𝒫​\(X\)=q\)⋅∑π∈𝒫​\(ρ\)\{δif​π∈\{τ1,τ2\}−δ\|𝒫​\(ρ\)\|−1if​π∈𝒫​\(ρ\)∖\{τ1,τ2\}\\displaystyle=q\[\\rho\]\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\\cdot\\sum\_\{\\pi\\in\\mathcal\{P\}\(\\rho\)\}\\begin\{cases\}\\delta&\\text\{if \}\\pi\\in\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\\\ \-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{if \}\\pi\\in\\mathcal\{P\}\(\\rho\)\\setminus\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{cases\}\(45\)=q​\[ρ\]\+∑i∈\[n\]h\|\|𝒫​\(x\)=qP​\(X=zi\|h\|\|𝒫​\(X\)=q\)⋅\(δ−δ\|𝒫​\(ρ\)\|−1​\(\|𝒫​\(ρ\)\|−1\)\)\\displaystyle=q\[\\rho\]\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(X\)=q\)\\cdot\\Big\(\\delta\-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}\(\|\\mathcal\{P\}\(\\rho\)\|\-1\)\\Big\)\(46\)=q​\[ρ\]\.\\displaystyle=q\[\\rho\]\.\(47\)Here, \([41](https://arxiv.org/html/2605.30447#A1.E41)\) is by the law of total probability\. \([42](https://arxiv.org/html/2605.30447#A1.E42)\) is by splitting the sum into the partition of the summands, while \([43](https://arxiv.org/html/2605.30447#A1.E43)\) uses \([40](https://arxiv.org/html/2605.30447#A1.E40)\)\. Using \([39](https://arxiv.org/html/2605.30447#A1.E39)\) parts of the sums can be merged as in \([44](https://arxiv.org/html/2605.30447#A1.E44)\)\. Sincehhis full\-rank calibrated onXXand therefore sub\-k calibrated onXXwe obtain \([45](https://arxiv.org/html/2605.30447#A1.E45)\), evaluating the sum leads to \([46](https://arxiv.org/html/2605.30447#A1.E46)\), while \([47](https://arxiv.org/html/2605.30447#A1.E47)\) holds as the last term in brackets is zero\.

Now it holds thathhis not top\-k calibrated on𝒳∖𝒳0∪𝒵\\mathcal\{X\}\\setminus\\mathcal\{X\}\_\{0\}\\cup\\mathcal\{Z\}\. Indeed, following the same steps as before and exchanging𝒫\\mathcal\{P\}by𝒯\\mathcal\{T\}we obtain:

P\(Π\|\|ℬ=ρ\|\\displaystyle P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\h\|\|𝒯\(X\)=q\)\\displaystyle h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)=∑x∈𝒳h\|\|𝒯​\(x\)=qP​\(Π\|\|ℬ=ρ\|X=x\)​P​\(X=x\|h\|\|𝒯​\(X\)=q\)\\displaystyle=\\sum\_\{\\begin\{subarray\}\{c\}x\\in\\mathcal\{X\}\\\\ h\_\{\|\|\\mathcal\{T\}\}\(x\)=q\\end\{subarray\}\}P\(\\Pi\_\{\|\|\\mathcal\{B\}\}=\\rho\\ \|\\ X=x\)\\,P\(X=x\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)\+∑i∈\[n\]h\|\|𝒯​\(x\)=qP​\(X=zi\|h\|\|𝒯​\(X\)=q\)⋅∑π∈𝒯​\(ρ\)\{δif​π∈\{τ1,τ2\}−δ\|𝒫​\(ρ\)\|−1if​π∈𝒫​\(ρ\)∖\{τ1,τ2\}\\displaystyle\\quad\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{T\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)\\cdot\\sum\_\{\\pi\\in\\mathcal\{T\}\(\\rho\)\}\\begin\{cases\}\\delta&\\text\{if \}\\pi\\in\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\\\ \-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{if \}\\pi\\in\\mathcal\{P\}\(\\rho\)\\setminus\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{cases\}\(48\)=q​\[ρ\]\+∑i∈\[n\]h\|\|𝒯​\(x\)=qP​\(X=zi\|h\|\|𝒯​\(X\)=q\)⋅∑π∈𝒯​\(ρ\)\{δif​π∈\{τ1,τ2\}−δ\|𝒫​\(ρ\)\|−1if​π∈𝒫​\(ρ\)∖\{τ1,τ2\}\\displaystyle=q\[\\rho\]\+\\sum\_\{\\begin\{subarray\}\{c\}i\\in\[n\]\\\\ h\_\{\|\|\\mathcal\{T\}\}\(x\)=q\\end\{subarray\}\}P\(X=z\_\{i\}\\ \|\\ h\_\{\|\|\\mathcal\{T\}\}\(X\)=q\)\\cdot\\sum\_\{\\pi\\in\\mathcal\{T\}\(\\rho\)\}\\begin\{cases\}\\delta&\\text\{if \}\\pi\\in\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\\\ \-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}&\\text\{if \}\\pi\\in\\mathcal\{P\}\(\\rho\)\\setminus\\\{\\tau\_\{1\},\\tau\_\{2\}\\\}\\end\{cases\}\(49\)=q​\[ρ\]\+\(δ−δ\|𝒫​\(ρ\)\|−1​\(\|𝒯​\(ρ\)\|−1\)\)\\displaystyle=q\[\\rho\]\+\\Big\(\\delta\-\\frac\{\\delta\}\{\|\\mathcal\{P\}\(\\rho\)\|\-1\}\(\|\\mathcal\{T\}\(\\rho\)\|\-1\)\\Big\)\(50\)≠q​\[ρ\]\.\\displaystyle\\neq q\[\\rho\]\.\(51\)Here, \([48](https://arxiv.org/html/2605.30447#A1.E48)\) is by the same arguments as for obtaining \([44](https://arxiv.org/html/2605.30447#A1.E44)\)\. Note that𝒯​\(ρ\)⊂𝒫​\(ρ\),\\mathcal\{T\}\(\\rho\)\\subset\\mathcal\{P\}\(\\rho\),so the otherwise case in \([40](https://arxiv.org/html/2605.30447#A1.E40)\) does not appear\. \([49](https://arxiv.org/html/2605.30447#A1.E49)\) is sincehhis full\-rank calibrated on𝒳\\mathcal\{X\}, while \([50](https://arxiv.org/html/2605.30447#A1.E50)\) evaluates the sum\. Finally, \([50](https://arxiv.org/html/2605.30447#A1.E50)\) holds as\|𝒯​\(ρ\)\|<\|𝒫​\(ρ\)\|\|\\mathcal\{T\}\(\\rho\)\|<\|\\mathcal\{P\}\(\\rho\)\|\.

∎

###### Theorem A\.3\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k<m−1:∃h∈ℙ​\(𝒮ℐ\):h​top\-k calibrated∧h​notsub\-k calibrated​\.\\forall k<m\-1:\\ \\exists h\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{top\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} sub\-k calibrated\}\\,\\text\{\.\}Fork=m−1k=m\-1it holds that top calibration induces sub\-k calibration\.

###### Proof of Theorem[A\.3](https://arxiv.org/html/2605.30447#A1.Thmtheorem3)\.

The proof is analogous to[Theorem A\.2](https://arxiv.org/html/2605.30447#A1.Thmtheorem2), exchanging the roles of𝒫\\mathcal\{P\}and𝒯\\mathcal\{T\}\. For the case ofk=m−1k=m\-1, it holds obviously using Corollary[A\.6](https://arxiv.org/html/2605.30447#A1.Thmtheorem6)and then[Theorem4\.2](https://arxiv.org/html/2605.30447#S4.Thmtheorem2)\. ∎

###### Corollary A\.4\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k∈\{2,…,m−1\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​sub\-k calibrated∧h​notfull\-rank calibrated​\.\\forall k\\in\\\{2,\\dots,m\-1\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{sub\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} full\-rank calibrated\}\\,\\text\{\.\}

###### Proof of Corollary[A\.4](https://arxiv.org/html/2605.30447#A1.Thmtheorem4)\.

This is a direct consequence of[Theorem A\.2](https://arxiv.org/html/2605.30447#A1.Thmtheorem2)and[Theorem 4\.2](https://arxiv.org/html/2605.30447#S4.Thmtheorem2)\(ii\): We can find a model that is sub\-k calibrated, but not top\-k calibrated and consequently not full\-rank calibrated\. ∎

Notice that for Plackett–Luce and Mallows Models the conditions onhhhold, such that the proof of[A\.4](https://arxiv.org/html/2605.30447#A1.Thmtheorem4)is also valid for those\.

###### Corollary A\.5\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:∀k∈\{1,…,m−2\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​top\-k calibrated∧h​notfull\-rank calibrated\.\\forall k\\in\\\{1,\\dots,m\-2\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{top\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} full\-rank calibrated\.\}

###### Proof of Corollary[A\.5](https://arxiv.org/html/2605.30447#A1.Thmtheorem5)\.

This is a direct consequence of[Theorem A\.3](https://arxiv.org/html/2605.30447#A1.Thmtheorem3)and[Theorem 4\.2](https://arxiv.org/html/2605.30447#S4.Thmtheorem2)\(i\): We can find a model that is top\-k calibrated, but not sub\-k calibrated and consequently not full\-rank calibrated\. ∎
###### Corollary A\.6\.

If h is top\-\(m\-1\) calibrated, then h is full\-rank calibrated

###### Proof of Corollary[A\.6](https://arxiv.org/html/2605.30447#A1.Thmtheorem6)\.

Recall that the marginalization of top\-k marginalization \([Definition 3](https://arxiv.org/html/2605.30447#Thmdefinition3)\) uses exactlyc=\(m−k\)\!c=\(m\-k\)\!many rankingsπ∈𝒮ℐ\\pi\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\}wherek∈\{1,…,m\}k\\in\\\{1,\\dots,m\\\}\. In particular fork=\(m−1\)k=\(m\-1\)we have exactly one rankingπ\\piin the marginalisation, asc=\(m−\(m−1\)\)\!=1c=\(m\-\(m\-1\)\)\!=1\. Obviously, it must hold thath∈ℋh\\in\\mathcal\{H\}being top\-\(m−1\)\(m\-1\)calibrated also induceshhfull\-rank calibrated\. ∎

###### Theorem A\.7\.

Under the same assumptions asLABEL:th:rankwise\_not\_full\_rank\_calibit holds:

∀k∈\{2,…,m−1\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise calibrated∧h​notsub\-k calibrated​\.\\forall k\\in\\\{2,\\dots,m\-1\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise calibrated \}\\land h\\ \\text\{ \\emph\{not\} sub\-k calibrated\}\\,\\text\{\.\}

###### Proof of Theorem[A\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7)\.

We proceed similarly to the proof ofLABEL:th:rankwise\_not\_full\_rank\_calib: Let𝒳\\mathcal\{X\}consist without loss of generality ofn<∞n<\\inftydata points for whichh∈ℋh\\in\\mathcal\{H\}be rankwise and sub\-k calibrated for somek∈\{2,…,m\}k\\in\\\{2,\\dots,m\\\}arbitrary but fixed\. Letℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}be a subset of items with\|ℬ\|=k\|\\mathcal\{B\}\|=k,π,τ∈𝒮ℐ,q∈ℙ​\(𝒮ℬ\)\\pi,\\tau\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\},q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\)andρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}arbitrary but fixed such thatπ∈𝒫​\(ρ\)\\pi\\in\\mathcal\{P\}\(\\rho\)\. Additionally, we chooseqqsuch thatq​\[ρ\]≠αq\[\\rho\]\\neq\\alpha\. We choose the data points in𝒳\\mathcal\{X\}such that there exist twoxi,xj∈𝒳x\_\{i\},x\_\{j\}\\in\\mathcal\{X\}for whichxi,xj∈𝒢=\{x\|h​\(x\)​\[π\]=α\}x\_\{i\},x\_\{j\}\\in\\mathcal\{G\}=\\\{x\\ \|\\ h\(x\)\[\\pi\]=\\alpha\\\},xi∈ℱ=\{x\|h\|\|𝒫​\(x\)=q\}x\_\{i\}\\in\\mathcal\{F\}=\\\{x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\\},xj∉ℱx\_\{j\}\\notin\\mathcal\{F\}andP​\(X=xi\|h​\(xi\)​\[π\]=α\)=P​\(X=xj\|h​\(xj\)​\[π\]=α\)P\(X=x\_\{i\}\\ \|\\ h\(x\_\{i\}\)\[\\pi\]=\\alpha\)=P\(X=x\_\{j\}\\ \|\\ h\(x\_\{j\}\)\[\\pi\]=\\alpha\)\. Intuitively,𝒢\\mathcal\{G\}are the data points for whichhhassigns the probabilityα\\alphato the rankingπ\\piandℱ\\mathcal\{F\}are the points for which the marginalsh\|\|𝒫h\_\{\|\|\\mathcal\{P\}\}are identical to the probability vectorqq\. Importantly, we have chosenxi,xjx\_\{i\},x\_\{j\}in such a way, that both are assigned the same probability forπ\\pi, i\.e\.,h​\(xi\)​\[π\]=h​\(xj\)​\[π\]=αh\(x\_\{i\}\)\[\\pi\]=h\(x\_\{j\}\)\[\\pi\]=\\alphabut their marginals are not identical, i\.e\.,h\|\|𝒫​\(xi\)≠h\|\|𝒫​\(xj\)h\_\{\|\|\\mathcal\{P\}\}\(x\_\{i\}\)\\neq h\_\{\|\|\\mathcal\{P\}\}\(x\_\{j\}\)\.

From here we can proceed similarly to the proof ofLABEL:th:rankwise\_not\_full\_rank\_caliband construct the samez1,…,znz\_\{1\},\\dots,z\_\{n\}based onx1,…,xnx\_\{1\},\\dots,x\_\{n\}such thathhis rankwise calibrated on these yet not sub\-k calibrated\. The same reason applies here: Sincexi,xjx\_\{i\},x\_\{j\}are elements of𝒢\\mathcal\{G\}and we have thatP​\(X=xi\|h​\(xi\)​\[π\]=α\)=P​\(X=xj\|h​\(xj\)​\[π\]=α\)P\(X=x\_\{i\}\\ \|\\ h\(x\_\{i\}\)\[\\pi\]=\\alpha\)=P\(X=x\_\{j\}\\ \|\\ h\(x\_\{j\}\)\[\\pi\]=\\alpha\)we can repeat the same steps \([28](https://arxiv.org/html/2605.30447#A1.E28)\) – \([33](https://arxiv.org/html/2605.30447#A1.E33)\) to show thathhis rankwise calibrated onz1,…,znz\_\{1\},\\dots,z\_\{n\}\. Sincexix\_\{i\}is an element ofℱ,\\mathcal\{F\},butxjx\_\{j\}is not,hhis not sub\-k calibrated onz1,…,znz\_\{1\},\\dots,z\_\{n\}\.

∎

###### Theorem A\.8\.

Assume that the number of itemsm≥3m\\geq 3and let𝒳=\{X1,…,Xn\}\\mathcal\{X\}=\\\{X\_\{1\},\\dots,X\_\{n\}\\\}be the set of \(random\) data points for whichn<∞n<\\infty\. Then it holds:

∀k∈\{1,…,m−2\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise calibrated∧h​nottop\-k calibrated​\.\\forall k\\in\\\{1,\\dots,m\-2\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise calibrated \}\\land h\\ \\text\{\\emph\{not\} top\-k calibrated\}\\,\\text\{\.\}

###### Proof of Theorem[A\.8](https://arxiv.org/html/2605.30447#A1.Thmtheorem8)\.

The proof is analogous to[Theorem A\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7), by replacing𝒫\\mathcal\{P\}for𝒯\\mathcal\{T\}\. ∎

###### Theorem A\.9\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k<m:∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise sub\-k calibrated∧h​notrankwise top\-k calibrated​\.\\forall k<m:\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise sub\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} rankwise top\-k calibrated\}\\,\\text\{\.\}

###### Proof of Theorem[A\.9](https://arxiv.org/html/2605.30447#A1.Thmtheorem9)\.

The proof is quite similar to the proof of[Theorem A\.2](https://arxiv.org/html/2605.30447#A1.Thmtheorem2), where instead of

1. 1\.choosing someq∈ℙ​\(𝒮ℬ\)q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\)arbitrary but fixed, take someα∈\[0,1\]\\alpha\\in\[0,1\]arbitrary but fixed,
2. 2\.choosinghhto be full\-rank calibrated, use anhhthat is rankwise calibrated,
3. 3\.conditioning onh\|\|𝒫​\(X\)=qh\_\{\|\|\\mathcal\{P\}\}\(X\)=q, condition onh\|\|𝒫​\(X\)​\[ρ\]=α,h\_\{\|\|\\mathcal\{P\}\}\(X\)\[\\rho\]=\\alpha,
4. 4\.conditioning onh\|\|𝒯​\(X\)=qh\_\{\|\|\\mathcal\{T\}\}\(X\)=q, condition onh\|\|𝒯​\(X\)​\[ρ\]=α,h\_\{\|\|\\mathcal\{T\}\}\(X\)\[\\rho\]=\\alpha,
5. 5\.replace each appearance ofq​\[ρ\]q\[\\rho\]byα\\alpha\.

∎

###### Theorem A\.10\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k<m:∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise top\-k calibrated∧h​notrankwise sub\-k calibrated​\.\\forall k<m:\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise top\-k calibrated \}\\land h\\ \\text\{ \\emph\{not\} rankwise sub\-k calibrated\}\\,\\text\{\.\}

###### Proof of Theorem[A\.10](https://arxiv.org/html/2605.30447#A1.Thmtheorem10)\.

This proof is analogous to[Theorem A\.9](https://arxiv.org/html/2605.30447#A1.Thmtheorem9), replacing𝒫\\mathcal\{P\}for𝒯\\mathcal\{T\}\. ∎

###### Corollary A\.11\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k∈\{2,…,m−1\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise sub\-k calibrated∧h​not rankwise calibrated\.\\forall k\\in\\\{2,\\dots,m\-1\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise sub\-k calibrated\}\\land h\\ \\text\{ not rankwise calibrated \}\\text\{\.\}

With a very similar proof technique one derives the following corollaries\.

###### Corollary A\.12\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:∀k∈\{1,…,m−2\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise top\-k calibrated∧h​not rankwise calibrated\.\\forall k\\in\\\{1,\\dots,m\-2\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise top\-k calibrated\}\\land h\\ \\text\{ not rankwise calibrated\.\}

Notice that Theorems[A\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7),[A\.8](https://arxiv.org/html/2605.30447#A1.Thmtheorem8),[A\.13](https://arxiv.org/html/2605.30447#A1.Thmtheorem13)and[A\.14](https://arxiv.org/html/2605.30447#A1.Thmtheorem14)also apply to Mallows Models and Plackett–Luce models, as there is no condition imposed onhhother than being calibrated\.

###### Theorem A\.13\.

Let𝒳\\mathcal\{X\}consist of at least two elements\. If there exist two data pointsxi,xj∈𝒳x\_\{i\},x\_\{j\}\\in\\mathcal\{X\}withi≠ji\\neq jsuch thatP​\(X=xi\|h​\(xi\)​\[α\]=α\)≠P​\(X=xj\|h​\(xj\)​\[α\]=α\)P\(X=x\_\{i\}\\ \|\\ h\(x\_\{i\}\)\[\\alpha\]=\\alpha\)\\neq P\(X=x\_\{j\}\\ \|\\ h\(x\_\{j\}\)\[\\alpha\]=\\alpha\)for someα∈\[0,1\]\\alpha\\in\[0,1\], then it holds:

∀k∈\{2,…,m−1\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​sub\-k calibrated∧h​not rankwise calibrated\.\\forall k\\in\\\{2,\\dots,m\-1\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{sub\-k calibrated\}\\land h\\ \\text\{ not rankwise calibrated\.\}

###### Proof of Theorem[A\.13](https://arxiv.org/html/2605.30447#A1.Thmtheorem13)\.

We proceed similarly to the proofs ofLABEL:th:rankwise\_not\_full\_rank\_caliband[Theorem A\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7): Let𝒳\\mathcal\{X\}consist without loss of generality ofn<∞n<\\inftydata points for whichh∈ℋh\\in\\mathcal\{H\}be rankwise and sub\-k calibrated for somek∈\{2,…,m\}k\\in\\\{2,\\dots,m\\\}arbitrary but fixed\. Letℬ⊆ℐ\\mathcal\{B\}\\subseteq\\mathcal\{I\}be a subset of items with\|ℬ\|=k\|\\mathcal\{B\}\|=k,π,τ∈𝒮ℐ,q∈ℙ​\(𝒮ℬ\)\\pi,\\tau\\in\\mathcal\{S\}\_\{\\mathcal\{I\}\},q\\in\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{B\}\}\)andρ∈𝒮ℬ\\rho\\in\\mathcal\{S\}\_\{\\mathcal\{B\}\}arbitrary but fixed such thatπ∈𝒫​\(ρ\)\\pi\\in\\mathcal\{P\}\(\\rho\)\. Additionally, we chooseqqsuch thatq​\[ρ\]≠αq\[\\rho\]\\neq\\alpha\. We choose the data points in𝒳\\mathcal\{X\}such that there exist twoxi,xj∈𝒳x\_\{i\},x\_\{j\}\\in\\mathcal\{X\}for whichxi,xj∈ℱ=\{x\|h\|\|𝒫​\(x\)=q\}x\_\{i\},x\_\{j\}\\in\\mathcal\{F\}=\\\{x\\ \|\\ h\_\{\|\|\\mathcal\{P\}\}\(x\)=q\\\},xi∈𝒢=\{x\|h​\(x\)​\[π\]=α\}x\_\{i\}\\in\\mathcal\{G\}=\\\{x\\ \|\\ h\(x\)\[\\pi\]=\\alpha\\\}andxj∉𝒢x\_\{j\}\\notin\\mathcal\{G\}\. Intuitively,𝒢\\mathcal\{G\}is the set of data points for whichhhassigns the probabilityα\\alphato the rankingπ\\pi, andℱ\\mathcal\{F\}is the set of points for which the marginalsh\|\|𝒫h\_\{\|\|\\mathcal\{P\}\}are identical to the probability vectorqq\. Importantly, we have chosenxi,xjx\_\{i\},x\_\{j\}in such a way that they have different probability forπ\\pi, i\.e\.,h​\(xi\)​\[π\]≠h​\(xj\)​\[π\]=αh\(x\_\{i\}\)\[\\pi\]\\neq h\(x\_\{j\}\)\[\\pi\]=\\alphabut their marginals are identical, i\.e\.,h\|\|𝒫​\(xi\)=h\|\|𝒫​\(xj\)h\_\{\|\|\\mathcal\{P\}\}\(x\_\{i\}\)=h\_\{\|\|\\mathcal\{P\}\}\(x\_\{j\}\)\.

From here we can proceed similarly to the proofs ofLABEL:th:rankwise\_not\_full\_rank\_caliband[Theorem A\.7](https://arxiv.org/html/2605.30447#A1.Thmtheorem7)\. That is, we construct the samez1,…,znz\_\{1\},\\dots,z\_\{n\}based onx1,…,xnx\_\{1\},\\dots,x\_\{n\}such thathhis sub\-k calibrated on these yet not rankwise calibrated\. The same reason applies here: Sincexi,xjx\_\{i\},x\_\{j\}are elements ofℱ\\mathcal\{F\}and we have that we can repeat the same steps \([28](https://arxiv.org/html/2605.30447#A1.E28)\) – \([33](https://arxiv.org/html/2605.30447#A1.E33)\) by using the appropriate marginalizations and projections to show thathhis sub\-k calibrated onz1,…,znz\_\{1\},\\dots,z\_\{n\}\. Sincexix\_\{i\}is an element of𝒢,\\mathcal\{G\},butxjx\_\{j\}is not, andP​\(X=xi\|h​\(xi\)​\[π\]=α\)≠P​\(X=xj\|h​\(xj\)​\[π\]=α\)P\(X=x\_\{i\}\\ \|\\ h\(x\_\{i\}\)\[\\pi\]=\\alpha\)\\neq P\(X=x\_\{j\}\\ \|\\ h\(x\_\{j\}\)\[\\pi\]=\\alpha\), we see thathhis not rankwise calibrated onz1,…,znz\_\{1\},\\dots,z\_\{n\}\.

∎

###### Theorem A\.14\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k∈\{1,…,m−2\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​top\-k calibrated∧h​not rankwise calibrated\.\\forall k\\in\\\{1,\\dots,m\-2\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{top\-k calibrated\}\\land h\\ \\text\{ not rankwise calibrated\.\}

###### Proof of Theorem[A\.14](https://arxiv.org/html/2605.30447#A1.Thmtheorem14)\.

The proof is analogous to[Theorem A\.13](https://arxiv.org/html/2605.30447#A1.Thmtheorem13), replacing𝒫\\mathcal\{P\}for𝒯\\mathcal\{T\}\. ∎

With a very similar proof technique one derives the following corollaries\.

###### Corollary A\.15\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:∀k∈\{2,…,m−1\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise sub\-k calibrated∧h​not sub\-k calibrated\.\\forall k\\in\\\{2,\\dots,m\-1\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise sub\-k calibrated\}\\land h\\ \\text\{ not sub\-k calibrated \}\\text\{\.\}

###### Corollary A\.16\.

Assume that the number of itemsm≥3m\\geq 3\. Then it holds:

∀k∈\{1,…,m−2\}​∃h∈𝒳→ℙ​\(𝒮ℐ\):h​rankwise top\-k calibrated∧h​not top\-k calibrated\.\\forall k\\in\\\{1,\\dots,m\-2\\\}\\ \\exists h\\in\\mathcal\{X\}\\rightarrow\\mathbb\{P\}\(\\mathcal\{S\}\_\{\\mathcal\{I\}\}\):h\\ \\text\{rankwise top\-k calibrated\}\\land h\\ \\text\{ not top\-k calibrated\.\}

## Appendix BDatasets Details

Table 4:Datasets used for the experiments\. The movies and political datasets correspond to real\-world problems\.All datasets used in our experiments are publicly available on[OpenML](https://www.openml.org/search?type=data&sort=runs&status=active&uploader_id=%3D_25829)\. An overview of the datasets employed for benchmarking is provided in[Table 4](https://arxiv.org/html/2605.30447#A2.T4), where*political*and*movies*constitute real\-world datasets\. The other datasets are adapted versions of classical classification tasks to the problem of label rankingHullermeier\.2008\. Specifically, for each dataset, we first train a Naive Bayes classifier\. The predicted class probabilities are then used to induce a ranking over all labels by sorting them in descending order\. In the event of ties, labels with smaller indices are ranked first\.

The*political*dataset\(Thies\.2024\)is derived from Likert\-scale survey questions, such as “How conservative would you rate CDU/CSU?”, which serve as features\. The learning task is to predict an overall ranking of the political parties*LINKE*,*GRUENE*,*SPD*,*CDU/CSU*,*FDP*, and*AfD*\. The*movies*dataset\(Harper\.2016\)consists of a subset of602602instances and contains ratings for the1515most frequently rated movies\. These movies are ranked according to their assigned star ratings, following the construction described byAlfaro\.2023\. All data files used in our experiments are available in our public repository:[https://github\.com/Advueu963/Calibrated\_Preference\_Learning](https://github.com/Advueu963/Calibrated_Preference_Learning)\.

## Appendix CImplementation Details

The ranking models are implemented in PyTorch\(Paszke\.2019\), using neural networks with two hidden layers of size 100 and ReLU activations\. For the Plackett–Luce model and RankClassifier, we use the Adam optimizer\(Kingma\.2014\)with learning rate10−310^\{\-3\}and no weight decay\. On each dataset, we train for 50 epochs using a batch size of 64\.

#### RankClassifier

RankClassifier is a multi\-class classification model that outputs a probability distribution over all possible rankings𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}\. Naively training such a model is infeasible for largemm, since the number of rankings grows factorially asm\!m\!\. FollowingResin\.2023, we therefore learn a distribution only over theNNunique rankings observed in the training data and assign uniform probability mass to all remaining rankings\. Concretely, letxxdenote the total probability mass assigned to theNNobserved rankings\. All other rankings are assigned probability1−xm\!−N\\frac\{1\-x\}\{m\!\-N\}\. We implement this by modifying the softmax normalization to implicitly account form\!−Nm\!\-Nadditional logits that share a constant valuecc\(instead of explicitly representingm\!−Nm\!\-Noutputs\)\. Letzrz\_\{r\}denote the logit for an observed rankingrr\. Then the probability of an observed ranking is

P​\(r\)=ezr∑r′∈ℛobsezr′\+\(m\!−N\)​ec,P\(r\)\\;=\\;\\frac\{e^\{z\_\{r\}\}\}\{\\sum\_\{r^\{\\prime\}\\in\\mathcal\{R\}\_\{\\text\{obs\}\}\}e^\{z\_\{r^\{\\prime\}\}\}\+\(m\!\-N\)e^\{c\}\},and each unobserved ranking receives probability

P​\(runobs\)=ec∑r′∈ℛobsezr′\+\(m\!−N\)​ec\.P\(r\_\{\\text\{unobs\}\}\)\\;=\\;\\frac\{e^\{c\}\}\{\\sum\_\{r^\{\\prime\}\\in\\mathcal\{R\}\_\{\\text\{obs\}\}\}e^\{z\_\{r^\{\\prime\}\}\}\+\(m\!\-N\)e^\{c\}\}\.This is memory efficient and allows us to represent full ranking distributions even for moderately large item sets \(e\.g\.,m=15m=15\)\. Assigning uniform probability mass to all unobserved rankings can be interpreted as a prior assumption; in contrast, assigning very negative logits to these rankings corresponds to assuming that such rankings cannot occur\. Model parameters are learned by minimizing the cross\-entropy loss between the predicted distribution and the observed ranking\.

#### Plackett–Luce model

The Plackett–Luce model is learned by minimizing the negative log\-likelihood of the observed rankings as described in\(Cheng\.2010\)\. The learned logits are exponentiated and normalized to obtain the corresponding choice probabilities\. Since the Plackett–Luce model is not uniquely identified without a normalization constraint\(Vitelli\.2017\), we normalize the parameters accordingly\. To avoid numerical issues, we subtract the maximum logit prior to exponentiation\.

#### Mallows model

The Mallows model is learned in interleaved steps, since both the dispersion parameterλ\\lambdaand the reference rankingτ\\taumust be estimated\. FollowingVitelli\.2017, we initializeτ\\tauas an “average” ranking in the training data by computing each item’s average position and sorting items by these averages \(ties are broken randomly\)\. We then alternate between updatingλ\\lambdausing the closed\-form solution inVitelli\.2017and updatingτ\\tauvia the local search procedure ofBusse\.2007\.

#### Ranking by pairwise comparisons \(RPC\)

Ranking by pairwise comparisons is implemented using the approach ofAlfaro\.2023\. We first construct a pairwise comparison dataset from the rankings by considering all implied pairwise preferences\. Then, for each item pair, we train a binary classifier to predict which item is preferred\. We use scikit\-learn’sDecisionTreeClassifier, as it has been reported to perform well in this setting\(Hullermeier\.2004\)\. We apply Platt scaling\(Platt\.1999\)as a post\-hoc calibration method to ensure that the underlying binary models output calibrated probabilities\. Finally, we aggregate the pairwise predictions using the Borda method\(McLean\.1995\)to obtain a ranking over all items\. We base our implementation onalfaro\_learning\_2021and adapt it to the label ranking setting\. Importantly the calibration of RPC can not be investigated for anything else than sub\-2, as it inherently only learns sub\-2 pairwise probabilities\.

#### Plackett–Luce\-RPC

For RPC one cannot directly obtain a probability distribution over all rankings, since the model only yields pairwise preference probabilities\. Interpreting these probabilities as Bradley–Terry probabilities\(Bradley\.1952\), we use the \(improved\) Zermelo algorithm\(Newman\.2023\)to obtain Plackett–Luce parameters, as described in[Section 3](https://arxiv.org/html/2605.30447#S3)\. Givenpi,j=P​\(i≻j\)p\_\{i,j\}=P\(i\\succ j\)for alli,j∈ℐi,j\\in\\mathcal\{I\}, we iteratively update parametersθi\\theta\_\{i\}fort=1,…,Tt=1,\\dots,Tvia

θi\(t\+1\)=∑j≠ipi,j​θi\(t\)θi\(t\)\+θj\(t\)∑j≠ipj,iθi\(t\)\+θj\(t\)\.\\theta\_\{i\}^\{\(t\+1\)\}\\;=\\;\\frac\{\\sum\_\{j\\neq i\}\\frac\{p\_\{i,j\}\\,\\theta\_\{i\}^\{\(t\)\}\}\{\\theta\_\{i\}^\{\(t\)\}\+\\theta\_\{j\}^\{\(t\)\}\}\}\{\\sum\_\{j\\neq i\}\\frac\{p\_\{j,i\}\}\{\\theta\_\{i\}^\{\(t\)\}\+\\theta\_\{j\}^\{\(t\)\}\}\}\\,\.As shown empirically byNewman\.2023, this update converges faster than the original Zermelo algorithm\(Zermelo\.1929\)while yielding the same estimates\. We setT=100T=100, which is safely above typical convergence thresholds \(e\.g\.,Newman\.2023report that fewer than 20 iterations can already achieve a squared difference of10−610^\{\-6\}\)\. This procedure relates to rank\-breaking, where pairwise preferences are used to infer a distribution over full rankings\(Soufiani\.2013\)\.

## Appendix DReward Model Calibration

This appendix details the reward model calibration experiments presented inLABEL:section:calibration\_of\_rlhf\_reward\_models\.

### D\.1Setup

Dataset\.RewardBench2\(Malik\.2025\)evaluates reward models across six categories: Factuality, Precise Instruction Following, Math, Safety, Focus, and Ties\. Each datapoint is a six\-tuple\(Q,R1,R2,R3,R4\)\(Q,R\_\{1\},R\_\{2\},R\_\{3\},R\_\{4\}\), whereQQis the prompt,R1,R2,R3,R4R\_\{1\},R\_\{2\},R\_\{3\},R\_\{4\}are candidate responses, andT∈\{1,2,3,4\}T\\in\\\{1,2,3,4\\\}indicates the correct answer\. We exclude the “Ties” category, which allows for more than four responses with multiple correct answers and thus does not fit our top\-1 formulation\.

Model selection\.We evaluate all Bradley–Terry\-trained reward models from the RewardBench2 leaderboard and report the results for the ten best\-calibrated ones inLABEL:fig:reward\_bench\_ece\. We exclude two models \(nicolinho/QRM\-Gemma\-2\-27Bandnicolinho/QRM\-Llama3\.1\-8B\-v2byDorka\.2024\) trained with quantile regression, as their outputs cannot be interpreted as Plackett–Luce utilities\.

Label ranking formulation\.Although ranking free\-form responses is naturally an object ranking problem, with the responses themselves having attributes beyond their identity\(Kamishima\.2011\), the fixed four\-response structure of RewardBench2 allows for a label ranking formulation\. We define the feature space𝒳\\mathcal\{X\}as the set of prompt\-response tuples\(Q,R1,R2,R3,R4\)\(Q,R\_\{1\},R\_\{2\},R\_\{3\},R\_\{4\}\)and take labelsℐ=\{1,2,3,4\}\\mathcal\{I\}=\\\{1,2,3,4\\\}to represent response positions in this context\. The prediction task is then to identify which position contains the best response\.

### D\.2Calibration Measurement

Given a reward modelrr, we define the induced top\-1 prediction model by

hr​\(\(Q,R1,R2,R3,R4\)\)​\[i\]=r​\(Q,Ri\)∑j=14r​\(Q,Rj\)​\.h\_\{r\}\\big\(\(Q,R\_\{1\},R\_\{2\},R\_\{3\},R\_\{4\}\)\\big\)\[i\]=\\frac\{r\(Q,R\_\{i\}\)\}\{\\sum\_\{j=1\}^\{4\}r\(Q,R\_\{j\}\)\}\\,\\text\{\.\}This is the top\-1 probability under the Plackett–Luce model with utilities given by the reward scores\.

ECE computation\.For each context, the model predicts a top\-1 probability for each of the four positions\. We compute ECE by pooling all \(context, position\) predictions, binning by predicted probability, and comparing mean predicted probability to empirical accuracy within each bin\. We use1010equal\-width bins\.

## Appendix ECalibration in Label Ranking

### E\.1Evaluation Metrics: Critical View and Discussion

![Refer to caption](https://arxiv.org/html/2605.30447v1/x1.png)
![Refer to caption](https://arxiv.org/html/2605.30447v1/x2.png)
![Refer to caption](https://arxiv.org/html/2605.30447v1/x3.png)

Figure 5:Comparison of using\|⋅\|\|\\cdot\|in contrast to Jeffrey Divergence for calculating sub\-k calibration via ECE\. Rank correlation of the methods is reported only if the values using\|⋅\|\|\\cdot\|are significantly different\.Expected Calibration Error \(ECE\) is a widely adopted measure for assessing the miscalibration of probabilistic classifiers\. However, beyond its well\-known sensitivity to binning hyperparameters, ECE exhibits limited expressiveness as the number of classes increases\. As a consequence, several alternative calibration measures have been proposed, most of which aim to reduce the effective number of observed classes\(Nixon\.2019\)\. This limitation becomes particularly severe in the context of label ranking, where the number of possible classes grows factorial and already exceeds100100for as few as55items\.

As the number of items increases, probabilistic ranking models inevitably predict increasingly flat distributions over rankings\. In this regime, the absolute differences used in ECE shrink in magnitude, leading to ECE values that are close to zero, almost independent of the actual calibration quality\. We mitigate this issue by restricting the computation of ECE to the95%95\\%most frequent rankings observed in the dataset\. For instance, in the*movies*dataset, only260260distinct rankings are observed in the data, despite there being15\!15\!possible rankings in total\. Among these observed rankings, we retain the95%95\\%most frequent ones based on empirical frequency\. Nevertheless, even under this restriction, ECE remains close to zero, as shown in[Figure 6](https://arxiv.org/html/2605.30447#A5.F6)\.

These observations indicate that using absolute differences in ECE becomes increasingly intractable for large\-scale label\-ranking problems\. They also raise the more fundamental question of whether predicting complete distributions over rankings is meaningful for large item sets such asm=15m=15, which we do not address in this work\. One possible direction to alleviate this limitation is to replace absolute error measures with relative divergence\-based metrics\. For example, the Kullback–Leibler divergenceDK​LD\_\{KL\}, or the symmetric Jeffreys divergence

Djeff​\(p∥q\)=DK​L​\(p∥q\)\+DK​L​\(q∥p\),D\_\{\\text\{jeff\}\}\(p\\\|q\)=D\_\{KL\}\(p\\\|q\)\+D\_\{KL\}\(q\\\|p\),whereppdenotes the empirical frequency andqqthe predicted distribution, could be employed\. Compared toDK​LD\_\{KL\},DjeffD\_\{\\text\{jeff\}\}has the appealing property of detecting miscalibration even when the model underestimates the true probability mass\.[Figure5](https://arxiv.org/html/2605.30447#A5.F5)showcases the potential differences when using a relative measure, such as the Jeffreys divergence, in contrast to using the standard absolute difference\.

Such alternatives require systematic investigation, which we leave to future work and instead propose a stagewise evaluation strategy\. Despite low absolute ECE values, meaningful relative comparisons between models remain possible \(e\.g\., determining whether one model is better calibrated than another\)\. Moreover, to assess calibration for rankings of lengthkk, we recommend starting with calibration atk′=2k^\{\\prime\}=2\. If ECE is high \(e\.g\., above0\.10\.1\), the model is clearly uncalibrated for larger rankings such ask=5k=5\. If ECE is acceptable, calibration can then be examined progressively fork′=3k^\{\\prime\}=3and higher values\. This strategy exploits the inherent calibration structure of label ranking models \(see[Figure 1](https://arxiv.org/html/2605.30447#S1.F1)\), since calibration at higherkknecessarily implies calibration for all lower\-order interactions\.

To assess full\-rank calibration, we employ the canonical multiclass calibration error estimator based on Dirichlet kernel density estimation proposed byPopordanoska\.2022\. This estimator targets the strongest notion of multiclass calibration, often referred to as canonical or distribution calibration\(Alvo\.2014\), which requires the entire predicted probability vector over rankings to be calibrated\. Intuitively, the method estimates the conditional empirical distribution of true rankings in a local neighborhood of each predicted probability vector, where locality is defined via a Dirichlet kernel\. The calibration error is then computed as the averageℓ2\\ell\_\{2\}distance between the predicted probability vector and this locally estimated empirical distribution:Full\-Rank\-ECE= 1n ∑\_j=1^n ∥∑i ≠jkDir\(h\(xj\);h\(xi\)\)yi∑i≠jkDir\(h\(xj\);h\(xi\)\) \- h\(x\_j\) ∥^2\_2 where k\_Dir\(h\(x\_i\);h\(x\_j\)\) = Γ\(∑k=1Kαi,k\)∏k=1KΓ\(αi,k\) ∏\_k=1^K h\(x\_j\)^α\_i,k \- 1\_k whereαi,k=h​\(xi\)kSi\+1\\alpha\_\{i,k\}=\\frac\{h\(x\_\{i\}\)\_\{k\}\}\{S\_\{i\}\}\+1,Si=∑k=1Kh​\(xi\)kS\_\{i\}=\\sum\_\{k=1\}^\{K\}h\(x\_\{i\}\)\_\{k\}andyiy\_\{i\}is the number ofπi\\pi\_\{i\}when enumerating the rankings𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}\. To apply this method we first encode the given \(sub\-\) rankings into a classification problem, via enumeration of the rankings\. For full rank sub\-k or full rank top\-k calibration, we use enumerate only the rankings present in the subset and not the entire set𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}\. For example2≻1≻32\\succ 1\\succ 3, might obtain the number33when enumerating over𝒮ℐ\\mathcal\{S\}\_\{\\mathcal\{I\}\}, yet if the consider the sub\-ranking1≻31\\succ 3it obtains the number22\. Unlike binning\-based approaches, the Dirichlet kernel estimator provides a smooth and consistent estimate of full\-distribution miscalibration in high\-dimensional probability simplices, making it particularly well suited for evaluating calibration of probabilistic label ranking models over the full ranking space\.

### E\.2Further Results

We report complete results for both full\-rank / rankwise sub\-kkcalibration and top\-kkcalibration of the considered label ranking models\. Full\-rank calibration is computed using the Dirichlet kernel proposed byPopordanoska\.2022\. Overall, models tend to be well calibrated on smaller datasets such as*iris*\([Figure 7](https://arxiv.org/html/2605.30447#A5.F7)\) and*authorship*\([Figure 8](https://arxiv.org/html/2605.30447#A5.F8)\), but become increasingly uncalibrated on datasets with larger ranking spaces, such as*movies*\([Figure 6](https://arxiv.org/html/2605.30447#A5.F6)\) and*political*\([Figure 9](https://arxiv.org/html/2605.30447#A5.F9)\)\. Across nearly all datasets, RPC and RankClassifier are among the best calibrated models, as both can leverage standard calibration methods for classification\(Aly\.2005\)\. RPC is most often the best calibrated model, while RankClassifier performs best on*iris*\([Figure 7](https://arxiv.org/html/2605.30447#A5.F7)\),*authorship*\([Figure 8](https://arxiv.org/html/2605.30447#A5.F8)\),*vehicle*\([Figure 10](https://arxiv.org/html/2605.30447#A5.F10)\), and*wine*\([Figure 13](https://arxiv.org/html/2605.30447#A5.F13)\)\. In contrast, the Mallows model consistently exhibits the worst calibration, since it learns only global parameters, which may fail to capture instance\-specific ranking uncertainty\. Among native label ranking models, Plackett–Luce achieves the strongest calibration performance, as its weights are learned individually for each data point rather than globally\. Nevertheless, as illustrated on*vowel*\([Figure 12](https://arxiv.org/html/2605.30447#A5.F12)\),*yeast*\([Figure 14](https://arxiv.org/html/2605.30447#A5.F14)\), and*glass*\([Figure 15](https://arxiv.org/html/2605.30447#A5.F15)\), Plackett–Luce can still be substantially uncalibrated\. Consistent with rank breaking effects, the Plackett–Luce–RPC variant shows inferior calibration, as it is trained only on pairwise comparisons but predicts a distribution over full rankings\. We further observe that, despite low ECE values for larger item sets \(k\>4k\>4\), almost all models exhibit high ECE fork=2k=2andk=3k=3\. This behavior is explained by the rapidly increasing number of possible rankings fork≥4k\\geq 4, which leads to extremely small probability masses and consequently small values of\|acc​\(Ib\)−conf​\(Ib\)\|\\lvert\\text\{acc\}\(I\_\{b\}\)\-\\text\{conf\}\(I\_\{b\}\)\\rvert, artificially reducing ECE\. This suggests that more sensitive calibration measures should be explored, which we leave for future work\. Overall, these results highlight the need for calibration methods tailored to label ranking, as \(i\) RankClassifier incurs high computational cost and \(ii\) RPC only produces distributions over pairwise comparisons, which do not reliably generalize to full ranking distributions\.

![Refer to caption](https://arxiv.org/html/2605.30447v1/x4.png)Figure 6:Calibration of label ranking models for the “movies” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x5.png)Figure 7:Calibration of label ranking models for the “iris” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x6.png)Figure 8:Calibration of label ranking models for the “authorship” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x7.png)Figure 9:Calibration of label ranking models for the “political” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x8.png)Figure 10:Calibration of label ranking models for the “vehicle” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x9.png)Figure 11:Calibration of label ranking models for the “segment” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x10.png)Figure 12:Calibration of label ranking models for the “vowel” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x11.png)Figure 13:Calibration of label ranking models for the “wine” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x12.png)Figure 14:Calibration of label ranking models for the “yeast” dataset, considering only the rankings covering95%95\\%of the probability mass\.![Refer to caption](https://arxiv.org/html/2605.30447v1/x13.png)Figure 15:Calibration of label ranking models for the “glass” dataset, considering only the rankings covering95%95\\%of the probability mass\.