Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion

# Calibrated Trust, Not Sharper Prediction
Source: [https://arxiv.org/html/2608.14617](https://arxiv.org/html/2608.14617)
An Empirical Study of Evidence\-Fusion and Conformal Triage for Legal Outcome Classification

###### Abstract\.

A recurring proposal in legal artificial intelligence is to improve case\-outcome prediction by fusing a battery of uncertainty\-quantification tools — an evidence graph with belief propagation, sequential Bayesian odds updating, Dempster–Shafer mass combination, and conformal prediction — into a single “groundbreaking” pipeline\. We subject this proposal to a direct empirical test on 1,000 real European Court of Human Rights \(ECtHR\) cases drawn from the LexGLUE and FairLex benchmarks, predicting whether the Court found at least one Convention violation from the case’s fact paragraphs\. We compare three families of systems across two frontier large language models \(LLMs\) — Anthropic Claude Opus 4\.8 and OpenAI GPT\-5\.5 — used as per\-fact evidence estimators: \(A\) the raw LLM, \(B\) the LLM routed through the fusion pipeline, and \(C\) a no\-LLM term\-frequency baseline routed through the same pipeline\. Across roughly 4,750 controlled tests we report four findings\. First, on discrimination \(AUROC≈0\.83\\approx 0\.83across all systems\) the fusion pipeline yields no improvement over either the raw LLM or the term\-frequency baseline; the strongest single discriminator is a frontier LLM used directly\. Second, naively composing an LLM with Bayesian\-odds and Dempster–Shafer fusion more than doubles calibration error \(expected calibration error rising from≈0\.16\\approx 0\.16to≈0\.46\\approx 0\.46\) through a prior\-mismatch mechanism that replicates exactly across both models\. Third, Dempster–Shafer fusion is actively unsafe on long evidence chains, committing confidently to incorrect labels at below\-chance accuracy; we recommend its removal\. Fourth — and most consequentially — the pipeline’s genuine value is not predictive but*operational*: routed through a conformal selective\-prediction layer, the system becomes substantially better at deciding which cases to automate and which to escalate to a human\. After removing Dempster–Shafer, recalibrating, and applying class\-conditional risk control validated on the full 1,000\-case set, the tuned engine auto\-clears cases at 96\.8% accuracy with 0\.5% of errors escaping unreviewed and 96\.3% of would\-be errors caught for human review, versus 85\.9% / 3\.8% / 72\.1% for an untuned baseline\. We argue that the contribution of such pipelines in law is*calibrated trust*— auditable, guarantee\-backed selective automation — rather than sharper prediction, and we situate this conclusion within the conformal\-risk\-control and learning\-to\-defer literatures\.

legal artificial intelligence; conformal prediction; selective prediction; calibration; learning to defer; case outcome prediction; human\-in\-the\-loop; large language models

††copyright:none††conference:arXiv preprint; 2026;††journalyear:2026## 1\.Introduction

Legal information retrieval and outcome prediction sit inside billable work product: a missed clause or a hallucinated citation is not an inconvenience but a professional liability\. A pattern has emerged in practitioner and product literature that we will call the*fusion proposal*: the claim that combining several distinct mathematical instruments for handling uncertainty — an evidence graph propagated by message passing, a sequential Bayesian odds update over items of evidence, Dempster–Shafer fusion of uncertain mass assignments, and conformal prediction as a calibrated wrapper — produces a qualitatively superior outcome\-prediction engine\. The instruments are individually well\-founded, and their juxtaposition is rhetorically attractive: each addresses a recognisable facet of legal reasoning \(dependency among facts, incremental belief revision, irreducible uncertainty, and finite\-sample guarantees, respectively\)\.

The proposal is rarely tested end to end on real case data, and almost never with an explicit accounting of*what*is supposed to improve and by how much\. This paper supplies that test\. We ask three operational questions that a general counsel, a court technology office, or a regulator would actually pose:

1. \(1\)Does the fusion pipeline predict outcomes more accurately than simpler alternatives — including doing nothing more elaborate than asking a single LLM?
2. \(2\)Does adding a frontier LLM to the pipeline produce a measurable improvement over a cheap, no\-LLM baseline?
3. \(3\)Can the pipeline be tuned so that an AI system safely automates the genuinely easy cases — reducing, not adding to, the human review burden — while escalating only cases that truly require a lawyer?

The third question is the one that matters for deployment\. An automation tool that handles every case but is silently wrong on a meaningful fraction is a liability in a setting where a confidently mistaken disposition can prejudice a client, breach a duty, or invite sanction\. Conversely, a tool so cautious that it escalates everything is worse than useless: it consumes the lawyer’s time*and*the cost of the model\. The deployable sweet spot is a system that knows the boundary of its own competence and can document that boundary\.

#### Contributions\.

We make the following contributions\.

- •C1 — Adversarial empirical evaluationof the fusion proposal on 1,000 real ECtHR cases, with two frontier LLMs as per\-fact evidence estimators and a term\-frequency baseline as the no\-LLM control, comprising approximately 4,750 controlled tests with bootstrap confidence intervals and multi\-seed stability checks\.
- •C2 — Clean separation of three effectsthat the fusion proposal conflates: the value of the LLM \(over a cheap baseline\), the value of the fusion mathematics \(over simple aggregation\), and the value of the conformal selective\-prediction layer \(over forced prediction\)\.
- •C3 — Documented negative result: Dempster–Shafer fusion, applied to long legal evidence chains, is not merely unhelpful but actively harmful, producing below\-chance accuracy on the cases to which it commits most confidently\.
- •C4 — Diagnosed and corrected failure mode: naive composition of an LLM with Bayesian\-odds fusion destroys calibration via prior mismatch, and a single\-parameter recalibration restores it\.
- •C5 — Tuned, validated triage enginethat, on the full 1,000\-case set, demonstrates a risk\-controlled selective\-automation policy with an auditable accuracy floor, presented in the risk\-control format established by the selective\-prediction literature\.
- •C6 — Reframingof the contribution of evidence\-fusion pipelines in law: from “sharper prediction” \(which we do not observe\) to “calibrated trust” \(which we do\), with the supporting design recommendations\.

#### On what “understanding” means here\.

None of the four instruments — belief propagation, Bayesian odds, Dempster–Shafer, conformal prediction — reads or understands legal text\. They operate exclusively on numbers\. The act of understanding a fact paragraph and converting it into a number is a separate stage, performed either by a language model \(which brings genuine, if imperfect, comprehension\) or by a term\-frequency statistic \(which brings none, only word\-occurrence correlations learned from training labels\)\. The pipeline is downstream of this translation and is blind to its meaning; it would behave identically if the numbers described soup recipes\. This two\-stage structure —*reader*then*arithmetic*— organises our entire experimental design, and keeping it explicit is what allows us to attribute observed effects correctly\.

## 2\.Related work

#### Legal judgment prediction\.

The modern English\-language legal\-judgment\-prediction task was established byChalkidis et al\.\([2019](https://arxiv.org/html/2608.14617#bib.bib8)\), who introduced an ECtHR dataset and neural baselines for binary violation, multi\-label article, and case\-importance prediction, and who explicitly examined demographic confounding through anonymisation\. That work, and the subsequent LexGLUE benchmark suite\(Chalkidis et al\.,[2022a](https://arxiv.org/html/2608.14617#bib.bib10)\)and the FairLex fairness benchmark\(Chalkidis et al\.,[2022b](https://arxiv.org/html/2608.14617#bib.bib11)\), provide the data and the task framing we adopt\. A central, sometimes uncomfortable, finding of this literature is that strong predictive accuracy on ECtHR facts is achievable from surface features, which simultaneously motivates a deferral mechanism: if a model can be right for the wrong reasons, a safety valve that escalates uncertain or atypical cases to a human is not optional but necessary\.

#### The four instruments\.

*Bayesian odds updating*formalises incremental belief revision: beginning from a base rate and multiplying in a likelihood ratio for each item of evidence, it mirrors the intuition of a fact\-finder whose view shifts as testimony accumulates\. Its Achilles heel, well known since the earliest naive\-Bayes text classifiers, is the conditional\-independence assumption\.*Belief propagation*on an evidence graph is the standard remedy: by modelling the dependency structure explicitly and damping messages, it discounts redundant evidence\.*Dempster–Shafer theory*was conceived for settings of irreducible ignorance, allowing mass to be placed on the whole frame rather than forced onto a single hypothesis; its combination rule is notoriously sensitive to conflict and, as we show, pathological when many confident sources are combined sequentially\.*Conformal prediction*is different in kind: it makes no modelling claim about the evidence at all, wrapping any score with a finite\-sample coverage guarantee\.

#### Conformal prediction and risk control\.

Conformal prediction provides distribution\-free, finite\-sample coverage guarantees under exchangeability\(Vovk et al\.,[2022](https://arxiv.org/html/2608.14617#bib.bib26); Angelopoulos and Bates,[2021](https://arxiv.org/html/2608.14617#bib.bib4)\)\. Its extension to risk control is given by Conformal Risk Control\(Angelopoulos et al\.,[2022](https://arxiv.org/html/2608.14617#bib.bib6)\), and its high\-probability form by Risk\-Controlling Prediction Sets and the Learn\-then\-Test calibration framework\(Bates et al\.,[2021](https://arxiv.org/html/2608.14617#bib.bib7); Angelopoulos et al\.,[2021](https://arxiv.org/html/2608.14617#bib.bib5)\)\. These tools are precisely what an auto\-clear gate requires: a way to pin the error rate on the automated subset to a chosen level, with a certificate\.

#### Selective prediction and learning to defer\.

Selective classification — equipping a predictor with a reject option that abstains on low\-confidence inputs — was placed on formal footing byEl\-Yaniv and Wiener \([2010](https://arxiv.org/html/2608.14617#bib.bib14)\)and operationalised for deep models with guaranteed risk byGeifman and El\-Yaniv \([2017](https://arxiv.org/html/2608.14617#bib.bib16),[2019](https://arxiv.org/html/2608.14617#bib.bib17)\)\. The standard performance object is the risk–coverage curve and its summary AURC; a complementary operating\-point metric is the Selective Accuracy Constraint \(SAC\)\(Galil et al\.,[2023](https://arxiv.org/html/2608.14617#bib.bib15)\)\. The question of*to whom*a rejected case should be routed is studied under learning to defer\(Madras et al\.,[2018](https://arxiv.org/html/2608.14617#bib.bib21); Mozannar and Sontag,[2020](https://arxiv.org/html/2608.14617#bib.bib22); Verma et al\.,[2023](https://arxiv.org/html/2608.14617#bib.bib25)\)and its cost\- and capacity\-aware extensions\(Leitão et al\.,[2022](https://arxiv.org/html/2608.14617#bib.bib20); Alves et al\.,[2024](https://arxiv.org/html/2608.14617#bib.bib3)\)\. We adopt the risk–coverage and SAC vocabulary throughout\.

#### Calibration\.

A threshold on a model’s confidence means what it claims only if that confidence is calibrated\.Guo et al\.\([2017](https://arxiv.org/html/2608.14617#bib.bib18)\)document systematic over\-confidence in modern neural networks and show that single\-parameter temperature scaling restores calibration cheaply without altering accuracy;Vovk and Petej \([2014](https://arxiv.org/html/2608.14617#bib.bib27)\)give distribution\-free calibrated binary probabilities\. Calibration is the first stage of our tuned pipeline and the lens for our central negative result about naive fusion\.

#### Selective prediction in law and for LLMs\.

Closest to our setting,Wang et al\.\([2024](https://arxiv.org/html/2608.14617#bib.bib28)\)present the first systematic study of selective prediction for legal case\-outcome classification\. For LLMs specifically, conformal abstention bounds the error rate on answered queries while keeping abstention low\(Abbasi\-Yadkori et al\.,[2024](https://arxiv.org/html/2608.14617#bib.bib2)\)\. The human side is not automatic:Hullman et al\.\([2025](https://arxiv.org/html/2608.14617#bib.bib19)\)caution that handing prediction sets to reviewers does not mechanically improve decisions, whileCresswell et al\.\([2024](https://arxiv.org/html/2608.14617#bib.bib13)\)find that conformal sets do improve human accuracy relative to fixed\-size sets at equal coverage\.

## 3\.Data and task

### 3\.1\.Source and provenance

We use the ECtHR Task A subset of LexGLUE\(Chalkidis et al\.,[2022a](https://arxiv.org/html/2608.14617#bib.bib10)\), comprising 9,000 training, 1,000 validation, and 1,000 test cases, and the ECtHR configuration of the FairLex benchmark\(Chalkidis et al\.,[2022b](https://arxiv.org/html/2608.14617#bib.bib11)\), which adds applicant gender, applicant age, and respondent\-state attributes\. All cases originate in real judgments of the European Court of Human Rights as released through the Court’s HUDOC database\. Each case is represented as an ordered list of fact paragraphs together with the set of Convention articles the Court found to have been violated\.

### 3\.2\.Task definition

We study the binary outcome

y=1​\{the Court found at least one Convention violation\},y\\;=\\;\\mathbf\{1\}\\\{\\text\{the Court found at least one Convention violation\}\\\},predicted from the case’s fact paragraphs alone\. Each fact paragraph is treated as one item of evidence, in keeping with the fusion proposal’s own framing\. A case contains on average roughly 20–25 fact paragraphs\.

### 3\.3\.Class balance and sampling

In the full 1,000\-case test set the positive \(violation\) rate is 89\.8%, leaving only about 10% no\-violation cases; the entire discriminative challenge therefore lies in separating the minority\. Because severe imbalance renders accuracy and calibration metrics hard to interpret, for the LLM\-scored experiments we draw a*balanced*stratified sample of 200 test cases \(100 violation, 100 no\-violation\)\. All per\-arm comparisons that involve LLM scores are computed on this balanced sample; the full 1,000\-case set is used for the term\-frequency pipeline internals \(Section[5\.4](https://arxiv.org/html/2608.14617#S5.SS4)\) and for the risk\-control validation of the tuned engine \(Section[7](https://arxiv.org/html/2608.14617#S7)\), where calibration\-set size is itself the object of study\.

## 4\.Methods

### 4\.1\.Two\-stage architecture

Every system we evaluate has the two\-stage structure described in the Introduction: a*reader*that maps each fact paragraph to a violation\-leaning score in\[0,1\]\[0,1\], and an*aggregator*that combines the per\-fact scores into a case\-level decision, optionally wrapped by a conformal layer\. We instantiate the reader in three ways and the aggregator in several\.

### 4\.2\.Readers

#### Term\-frequency reader \(no LLM\)\.

We fit a term\-frequency–inverse\-document\-frequency \(TF–IDF\) vectoriser \(1–2 grams, 20,000\-term vocabulary, sublinear term frequency\) followed by L2\-regularised logistic regression, trained on all fact paragraphs of the 9,000 training cases, each paragraph inheriting its case’s binary label\. The reader outputsPr⁡\(violation∣paragraph\)\\Pr\(\\text\{violation\}\\mid\\text\{paragraph\}\)for any paragraph\. This reader has no comprehension; it exploits learned word\-occurrence correlations\.

#### LLM readers\.

We use two frontier models, Anthropic Claude Opus 4\.8 and OpenAI GPT\-5\.5\. Each model reads the fact paragraphs of a case and returns \(i\) a holistic case\-level violation probability and \(ii\) a per\-paragraph violation\-leaning score in\[0,1\]\[0,1\]\. Models were instructed to reason from the facts alone, to use the full\[0,1\]\[0,1\]range, and to avoid clustering scores at 0\.5; they were not shown ground\-truth labels\. All 200 cases were scored by each model, and every returned object was validated for format and per\-paragraph count \(400/400 cases valid, zero format failures\)\.

### 4\.3\.Aggregators

Given per\-fact scoresp1,…,pKp\_\{1\},\\dots,p\_\{K\}and a base ratebb\(the training positive rate, 0\.898\), we consider the following case\-level aggregators\.

- •Mean and Max\(baselines\): the arithmetic mean and the maximum of the per\-fact scores\.
- •Bayesian odds\.Treating facts as conditionally independent evidence, the posterior odds equal the prior odds times the product of per\-fact likelihood ratios: O​\(H∣E\)=b1−b​∏ipi/b\(1−pi\)/\(1−b\)\.O\(H\\mid E\)\\;=\\;\\tfrac\{b\}\{1\-b\}\\prod\_\{i\}\\tfrac\{p\_\{i\}/b\}\{\(1\-p\_\{i\}\)/\(1\-b\)\}\.
- •Belief propagation\.An evidence graph with a hidden outcome node connected to each fact node, propagated with a damping factor that discounts correlated evidence so thatKKoverlapping paragraphs are not counted asKKindependent proofs\. We also study a chain\-aware variant with dampingλ=1/K\\lambda=1/\\sqrt\{K\}, which strengthens with the number of paragraphs\.
- •Dempster–Shafer\.On the frame\{violation,no\-violation\}\\\{\\text\{violation\},\\text\{no\-violation\}\\\}, each fact contributes a mass function with an explicit uncertainty mass; masses are combined sequentially by Dempster’s rule with conflict normalisation, and the decision is taken from the pignistic transform\.

### 4\.4\.Conformal layer

We apply split conformal prediction with a class\-conditional nonconformity score, producing for each case a prediction*set*over\{violation,no\-violation\}\\\{\\text\{violation\},\\text\{no\-violation\}\\\}with target coverage1−α1\-\\alpha\. A singleton set is read as a confident, committed prediction \(Proceed\); a two\-element set signals genuine uncertainty \(Review\)\. Concretely, lets​\(x\)s\(x\)be a case\-level score interpreted as the estimated probability of violation givenxx\. On a calibration set we compute class\-conditional nonconformity scores —1−s1\-sfor true\-violation cases andssfor true\-no\-violation cases — and form the per\-class quantilesq1q\_\{1\}andq0q\_\{0\}at level⌈\(n\+1\)​\(1−α\)⌉/n\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil/n\. For a test case the prediction set includes label 1 if1−s​\(x\)≤q11\-s\(x\)\\leq q\_\{1\}and label 0 ifs​\(x\)≤q0s\(x\)\\leq q\_\{0\}; if both inclusion tests fail we add the arg\-max label so the set is never empty\. Class\-conditional quantiles, rather than a single marginal threshold, are what later permit the asymmetric, minority\-aware operating points of Section[7](https://arxiv.org/html/2608.14617#S7)\.

### 4\.5\.Selective\-prediction and triage metrics

We use the vocabulary of selective classification\. Writingg​\(x\)∈\{0,1\}g\(x\)\\in\\\{0,1\\\}for the decision to accept \(auto\-clear\) a case andy^​\(x\)\\hat\{y\}\(x\)for the committed label:

coverage:ϕ\\displaystyle\\text\{coverage:\}\\quad\\phi=𝔼​\[g​\(X\)\],\\displaystyle=\\mathbb\{E\}\[g\(X\)\],selective risk:R\\displaystyle\\text\{selective risk:\}\\quad R=𝔼​\[𝟏​\{y^​\(X\)≠Y\}⋅g​\(X\)\]/ϕ\.\\displaystyle=\\mathbb\{E\}\\bigl\[\\mathbf\{1\}\\\{\\hat\{y\}\(X\)\\neq Y\\\}\\cdot g\(X\)\\bigr\]/\\phi\.The*risk–coverage curve*tracesRRagainstϕ\\phias the acceptance threshold sweeps; its summary is AURC \(lower is better\)\. The*Selective Accuracy Constraint*SAC​\(a\)\\text\{SAC\}\(a\)is the maximum coverage achievable while holding selective accuracy at or aboveaa\. For the triage analysis we additionally report the*caught\-error rate*\(share of would\-be errors escalated to review\), the*escaped\-error rate*\(share of all cases both auto\-cleared and wrong\), the*wasted\-review rate*, and the auto\-clear rate\.

### 4\.6\.Experimental arms

The three principal arms isolate the three effects the fusion proposal conflates:

- •Arm A — Raw LLM\.The LLM’s holistic case probability, with no fusion mathematics\.
- •Arm B — LLM \+ Combination\.The LLM’s per\-fact scores routed through the fusion aggregators and the conformal layer\.
- •Arm C — TF–IDF \+ Combination\.The term\-frequency reader’s per\-fact scores through the identical pipeline \(no LLM\)\.

Two controls sharpen the attribution:B2, the LLM per\-fact scores under simple mean aggregation \(isolating the value of the fusion mathematics on identical LLM evidence\), andC2, the analogous term\-frequency control\.

### 4\.7\.Evaluation protocol

Discrimination is measured by AUROC with 200\-resample bootstrap 95% confidence intervals; calibration by the Brier score and expected calibration error \(ECE\) over ten equal\-width bins; selective performance by risk–coverage curves, AURC, and SAC; and triage by the auto\-clear rate, the accuracy on the proceeded subset, the fraction of would\-be errors caught for review, and the fraction of wrong decisions that escape to auto\-decision\. The tuned engine is validated with class\-conditional binomial upper confidence bounds \(Clopper–Pearson\) on the error of the auto\-cleared subset, over many random calibration/test splits, reporting the realised floor\-violation frequency in the style ofGeifman and El\-Yaniv \([2017](https://arxiv.org/html/2608.14617#bib.bib16)\)\.

## 5\.Results: discrimination, calibration, coverage

### 5\.1\.The fusion pipeline does not improve discrimination

Table[1](https://arxiv.org/html/2608.14617#S5.T1)reports AUROC on the balanced 200\-case sample for the three arms and two LLM readers, with 95% bootstrap confidence intervals\. The headline is the absence of any fusion benefit\. The strongest single discriminator in the entire study is a frontier LLM used directly: GPT\-5\.5 raw attains AUROC 0\.851 \(95% CI\[0\.80,0\.90\]\[0\.80,0\.90\]\)\. Routing that same model through the fusion pipeline*reduces*discrimination to 0\.783, and the term\-frequency baseline through the identical pipeline matches the LLM arms at 0\.831\. Every confidence interval overlaps the term\-frequency control\. On the question “rank cases by their probability of a violation finding,” neither the fusion mathematics nor the choice between a frontier LLM and a regularised logistic regression on word features is decisive\.

Table 1\.Discrimination \(AUROC\) on the balanced 200\-case ECtHR sample\. Brackets give 95% bootstrap CIs\. Higher is better\.Figure[1](https://arxiv.org/html/2608.14617#S5.F1)displays the same comparison graphically; the overlap of intervals with the no\-LLM control is the salient feature\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x1.png)Figure 1\.Discrimination \(AUROC\) across arms and models, with 95% bootstrap intervals\. The dashed line marks chance\. No arm dominates\.
### 5\.2\.Naive fusion destroys calibration

Although discrimination is flat, calibration is not\. Table[2](https://arxiv.org/html/2608.14617#S5.T2)shows that composing an LLM with the Bayesian\-odds and Dempster–Shafer aggregators more than doubles both Brier score and ECE relative to the raw LLM, and that this degradation is essentially identical across the two models \(ECE rising to 0\.457 for Opus and 0\.463 for GPT\-5\.5\)\. The mechanism is a prior mismatch: the Bayesian\-odds update re\-injects the 89\.8% training prior into scores produced on a balanced sample, dragging posterior probabilities toward the majority class and breaking the correspondence between confidence and accuracy\. The control arm B2 — the same LLM evidence under simple mean aggregation — retains the raw LLM’s calibration, confirming that the damage is caused by the fusion mathematics, not by the LLM evidence\.

Table 2\.Calibration on the balanced 200\-case sample\. Lower is better\.![Refer to caption](https://arxiv.org/html/2608.14617v1/x2.png)Figure 2\.Calibration \(Brier, left; ECE, right\) across arms and models\. Composing an LLM with the fusion mathematics more than doubles both metrics; the damage is caused by prior mismatch, not by the LLM\.Figure[3](https://arxiv.org/html/2608.14617#S5.F3)presents reliability diagrams for the Opus reader in the style ofGuo et al\.\([2017](https://arxiv.org/html/2608.14617#bib.bib18)\): a confidence histogram above a reliability diagram, with the shaded region marking the gap between confidence and accuracy\. The raw LLM is mildly over\-confident \(ECE 0\.158\); a single\-parameter recalibration of the fused score — which we adopt in the tuned engine of Section[7](https://arxiv.org/html/2608.14617#S7)— removes essentially all of the gap\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x3.png)Figure 3\.Reliability diagrams \(Opus 4\.8\): raw LLM versus the calibrated fused score\. Perfect calibration lies on the diagonal; shaded bars show the confidence–accuracy gap; ECE is annotated\.
### 5\.3\.Conformal coverage holds universally

The one property that holds without exception is the conformal coverage guarantee\. Across every arm, both models, and target levels1−α∈\{0\.80,0\.90,0\.95\}1\-\\alpha\\in\\\{0\.80,0\.90,0\.95\\\}, empirical marginal coverage met or exceeded its target\. For example, the LLM \+ Combination arm achieved 0\.913 coverage against a 0\.90 target for Opus and 0\.921 for GPT\-5\.5\. In the internal sweep over the term\-frequency pipeline \(Section[5\.4](https://arxiv.org/html/2608.14617#S5.SS4)\), all 25 method\-by\-α\\alphacells satisfied coverage\. This is the model\-agnostic, distribution\-free promise of conformal prediction realised on real legal data, and it is the property that makes the downstream triage claims auditable\.

### 5\.4\.Pipeline internals and the failure of Dempster–Shafer

Evaluated on the full 1,000\-case test set with the term\-frequency reader, the aggregators separate sharply \(Table[3](https://arxiv.org/html/2608.14617#S5.T3)\)\. Belief propagation \(AUROC 0\.820\) and Bayesian odds \(0\.817\) are competitive with the mean baseline \(0\.834\), and belief propagation improves calibration over raw Bayesian odds \(Brier 0\.198 versus 0\.222\), consistent with its damping of correlated evidence\. Dempster–Shafer, by contrast, collapses to AUROC 0\.541 — barely above chance\.

Table 3\.Aggregator discrimination on the full 1,000\-case test set \(term\-frequency reader\)\. Brackets give 95% bootstrap intervals\.![Refer to caption](https://arxiv.org/html/2608.14617v1/x4.png)Figure 4\.Aggregator discrimination \(AUROC, 95% bootstrap CI\) on the full test set with the term\-frequency reader\. Dempster–Shafer’s interval sits just above chance\.The Dempster–Shafer failure is not merely weak discrimination; it is dangerous confidence\. Figure[5](https://arxiv.org/html/2608.14617#S5.F5)plots the accuracy of*committed*\(singleton\) predictions as a function ofα\\alpha\. For belief propagation and Bayesian odds, singleton accuracy is high \(≈0\.85\\approx 0\.85atα=0\.10\\alpha=0\.10\); for Dempster–Shafer it falls*below chance*— the method commits a confident singleton on a small fraction of cases and is wrong on roughly three\-quarters of them\. Sequential Dempster combination over 20\-plus high\-mass sources saturates toward certainty and concentrates that certainty on the wrong label\. The conformal wrapper limits the damage — it keeps Dempster–Shafer sets large so that marginal coverage is still met — but a component that must be neutralised by its wrapper to avoid harm has no place in the pipeline\. We remove Dempster–Shafer from all subsequent analysis\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x5.png)Figure 5\.Accuracy of committed \(singleton\) predictions versusα\\alpha\(full test set, term\-frequency reader\)\. Dempster–Shafer falls below chance; belief propagation and Bayesian odds remain reliable\.
### 5\.5\.A worked case: how the instruments interact

To make the mechanisms concrete, we trace a single real case \(test index 858; the Court found a violation\)\. The applicant, an elderly man, had a child with a married woman and maintained regular contact until the mother sought to relocate the child abroad; the domestic courts rejected his paternity claim as time\-barred without examining his reasons for delay or the child’s best interests — the classic fact pattern of a rigid procedural bar overriding a substantive family\-life interest, which the ECtHR has repeatedly found to violate Article 8\.

The Opus reader assigned the nine fact paragraphs the scores0\.56,0\.70,0\.81,0\.70,0\.91,0\.58,0\.97,0\.97,0\.920\.56,0\.70,0\.81,0\.70,0\.91,0\.58,0\.97,0\.97,0\.92\. The reading is legally sensible: biographical and procedural\-setup paragraphs sit near the middle, while the paragraphs describing the courts’ refusal to weigh the human factors \(the seventh through ninth\) are scored as strongly incriminating\. The reader’s holistic case probability was 0\.70 — a correct, if unstructured, lean toward violation\.

The aggregators then transform these nine numbers, and their divergence is instructive\. The base rate places the prior odds at roughly nine\-to\-one for violation, so a likelihood ratio exceeds one only for a paragraph scored above the 0\.898 prior; several genuinely pro\-applicant paragraphs \(0\.56, 0\.58, 0\.70\) therefore register, against this unusually high prior, as mild evidence*against*violation\. The undamped Bayesian\-odds product compounds these into a posterior of 0\.121 — a confident, and wrong, lean toward no\-violation\. Belief propagation with light damping recovers only to 0\.286, still on the wrong side\. The chain\-aware damping of Section[7](https://arxiv.org/html/2608.14617#S7)\(λ=1/9\\lambda=1/\\sqrt\{9\}\) produces 0\.688, and the simple mean produces 0\.791 — both correctly aligned with the LLM’s own 0\.70 and with the true outcome\. The case is a microcosm of the paper: the per\-fact reading carries the signal; undamped Bayesian fusion can destroy it by over\-counting weak evidence against a skewed prior; damping or simple averaging preserves it\.

## 6\.Results: selective prediction and human\-in\-the\-loop triage

### 6\.1\.From prediction to triage

The previous section establishes that the pipeline does not sharpen prediction\. We now show where it does add value\. We reframe the task as*triage*: rather than forcing a label on every case, the conformal layer routes each case to one of two destinations —Proceed\(the system commits to its label and the case is automated\) orReview\(the case is escalated to a human\)\. A good triage policy has two duties: the cases it proceeds on should be ones it gets right \(high accuracy on the automated subset\), and the cases it escalates should be the ones it would otherwise get wrong \(few errors escaping unreviewed, and little wasted human effort on cases the system had right\)\. With Dempster–Shafer removed, the “combination” in this section denotes belief propagation and Bayesian odds, wrapped by conformal selection\.

### 6\.2\.Risk–coverage curves and the operating\-point view

Figure[6](https://arxiv.org/html/2608.14617#S6.F6)shows risk–coverage curves on the balanced sample for both readers, with AURC annotated; Figure[7](https://arxiv.org/html/2608.14617#S6.F7)reports the Selective Accuracy Constraint — the maximum coverage attainable at each required selective\-accuracy floor\. These canonical selective\-prediction views make precise a subtlety that the discrimination numbers hide\. On the balanced sample, the raw LLM’s holistic probability is poorly suited to a confidence\-threshold sweep: it concentrates mass and yields a small SAC at high accuracy \(the raw\-LLM SAC at a 95% accuracy floor is near zero\), whereas the calibrated term\-frequency\-plus\-combination score sustains a meaningful SAC \(0\.42 coverage at a 95% floor; 0\.575 at 90%\)\. This is the first quantitative sign that the combination’s contribution lives in the*selective*regime, not in ranking\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x6.png)Figure 6\.Risk–coverage curves \(selective risk versus coverage\) for both readers, with AURC annotated\. Lower curves are better\.![Refer to caption](https://arxiv.org/html/2608.14617v1/x7.png)Figure 7\.Selective Accuracy Constraint \(Opus reader, balanced sample\): maximum coverage achievable at each required selective\-accuracy floor\.
### 6\.3\.The combination improves error\-catching, at a cost in coverage

Table[4](https://arxiv.org/html/2608.14617#S6.T4)reports the triage comparison at the 90% confidence setting \(α=0\.10\\alpha=0\.10\) for both models, with Dempster–Shafer removed\. Routing the LLM through the combination raises the fraction of would\-be errors caught for review from about 68% \(raw LLM\) to about 87%, and lowers the fraction of wrong decisions that escape to auto\-decision from about 7\.3% to about 4\.7%\. The pattern is near\-identical across Opus and GPT\-5\.5\. The cost is candour about coverage: the combination is more cautious, auto\-deciding about 21% of cases versus 51–59% for the raw LLM\. The no\-LLM arm \(C\) is the least safe of the three, letting the most errors escape — evidence that the LLM reader does contribute, even though that contribution is to safety rather than to ranking\.

Table 4\.Triage atα=0\.10\\alpha=0\.10\(balanced sample, Dempster–Shafer removed\)\. “Caught” = share of would\-be errors escalated to review; “Escaped” = share of wrong decisions auto\-issued; “Proceed” = auto\-decision rate\.![Refer to caption](https://arxiv.org/html/2608.14617v1/x8.png)Figure 8\.Triage across confidence settings, averaged over Opus and GPT\-5\.5\. \(a\) Fraction of would\-be errors caught for human review; higher is safer\. \(b\) Fraction of wrong decisions that escape to auto\-decision; lower is safer\. LLM \+ safe Combination is uniformly safer at every operating point\.
### 6\.4\.The combination roughly doubles abstention value on identical evidence

The cleanest attribution comes from comparing arm B against control B2 — the same LLM evidence with and without the fusion mathematics — using the abstention error\-reduction metric \(the drop in error rate when the system commits to a singleton rather than being forced to predict\)\. Across confidence settings, B reduces error by 0\.31–0\.35, roughly double the 0\.11–0\.17 of B2 and of the raw LLM, and the curves for Opus and GPT\-5\.5 are nearly coincident \(Figure[9](https://arxiv.org/html/2608.14617#S6.F9)\)\. The fusion mathematics, in other words, converts a frontier LLM into a*risk\-aware*one: it does not change the average prediction, but it sharpens the system’s sense of when its commitment is trustworthy\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x9.png)Figure 9\.Abstention error reduction versus target coverage, Opus and GPT\-5\.5\. The LLM \+ Combination arm roughly doubles the selective benefit of the raw LLM and of the no\-fusion control at every operating point; the two models are near\-identical\.

## 7\.A tuned, risk\-controlled triage engine

### 7\.1\.Design

The findings above imply a specific design\. Because the combination adds no discrimination, every lever for safe automation must act on*where a threshold sits on a well\-calibrated score*, not on more fusion\. We convened a structured design review across three frontier models \(Opus 4\.8, GPT\-5\.5, and Gemini 3\.1 Pro\) and an arXiv literature scout; their independent recommendations converged on a single pipeline, which we implement:

1. \(1\)Remove Dempster–Shafer \(Section[5\.4](https://arxiv.org/html/2608.14617#S5.SS4)\)\.
2. \(2\)Recalibrate the fused score — temperature scaling, optionally followed by isotonic or Venn–Abers calibration — to repair the prior mismatch of Section[5\.2](https://arxiv.org/html/2608.14617#S5.SS2)\(Guo et al\.,[2017](https://arxiv.org/html/2608.14617#bib.bib18); Vovk and Petej,[2014](https://arxiv.org/html/2608.14617#bib.bib27)\)\.
3. \(3\)Class\-conditional thresholds\. Because the no\-violation class is the rare, hard one, apply separate confidence thresholds per predicted class rather than a single symmetricα\\alpha\.
4. \(4\)Risk control with an explicit accuracy floor\. Choose the thresholds that*maximise*the auto\-clear rate subject to a class\-conditional binomial upper confidence bound \(Clopper–Pearson at confidence1−δ1\-\\delta\) on the auto\-cleared error rate, following the risk\-control programme ofBates et al\.\([2021](https://arxiv.org/html/2608.14617#bib.bib7)\)andAngelopoulos et al\.\([2021](https://arxiv.org/html/2608.14617#bib.bib5),[2022](https://arxiv.org/html/2608.14617#bib.bib6)\)\.

### 7\.2\.The over\-confidence pathology of raw fusion

A practical obstacle surfaced during implementation and is worth recording\. The undamped Bayesian\-odds product is numerically pathological on long chains: multiplying roughly 25 likelihood ratios against a high prior drives 96% of cases to a posterior of exactly 0\.000, collapsing AUROC to 0\.68 and rendering the score uncalibratable\. Two fixes restore good behaviour\. First, the chain\-aware dampingλ=1/K\\lambda=1/\\sqrt\{K\}prevents the saturation and recovers AUROC to≈0\.81\\approx 0\.81\. Second, and consistent with Section[5\.1](https://arxiv.org/html/2608.14617#S5.SS1), the simple mean of the per\-fact scores is itself a clean, well\-spread, equally discriminative score \(AUROC 0\.830\) with none of the pathology\. The tuned engine uses the damped score; the mean is an equally defensible substitute\.

### 7\.3\.Validation on the full 1,000\-case set

The accuracy floor is only meaningful if it holds out of sample, and certifying a*tight*per\-class floor requires a sufficiently large calibration set\. On the balanced 200\-case sample, the binomial upper bound on the rare no\-violation class is too weak to certify a 5% floor — with about 25 minority cases per split, a zero\-error calibration result still admits a 7% upper bound — and the realised floor\-violation frequency on the minority class is unacceptably high \(0\.46\)\. On the full 1,000\-case set, the certification becomes reliable\. Table[5](https://arxiv.org/html/2608.14617#S7.T5)presents the tuned engine against the untuned symmetric\-conformal baseline over 200 random 500/500 calibration/test splits\.

Table 5\.Tuned engine versus baseline on the full 1,000\-case set \(200 random splits\)\. “Auto\-clear accuracy” is accuracy on the automated subset; “Escaped” is the share of wrong decisions auto\-issued; “Caught” is the share of would\-be errors escalated; “Auto\-clear rate” is the automated fraction\.At a 95% target, the tuned engine auto\-clears cases at 96\.8% accuracy with only 0\.5% of errors escaping unreviewed and 96\.3% of would\-be errors caught for human review, against 85\.9% / 3\.8% / 72\.1% for the baseline \(Figure[10](https://arxiv.org/html/2608.14617#S7.F10)\)\. The price is a lower auto\-clear rate \(14% versus 29%\), the deliberate consequence of demanding a certified floor\. The realised floor\-violation frequency on the majority class is 0\.015 — comfortably below the nominalδ=0\.05\\delta=0\.05— confirming that the guarantee holds out of sample once the calibration set is large enough\.

We report candidly that on the minority no\-violation class the violation frequency remains high even atn=1000n=1000\(0\.46\), because the per\-class minority count per split is still on the order of 75; a production deployment that needs a certified minority\-class floor requires either a larger calibration corpus or a less stringent minority target\. This dependence of the guarantee on calibration\-set size is itself one of the paper’s practical contributions\.

![Refer to caption](https://arxiv.org/html/2608.14617v1/x10.png)Figure 10\.Tuned engine versus untuned baseline on the full 1,000\-case set, at 95% and 90% accuracy floors\. The tuned engine is markedly safer \(higher auto\-clear accuracy, far fewer escaped errors, more errors caught\) at the cost of a lower auto\-clear rate\.

## 8\.Discussion

#### Calibrated trust, not sharper prediction\.

The fusion proposal promises better prediction and does not deliver it: across two frontier models and a strong baseline, discrimination is flat at AUROC≈0\.83\\approx 0\.83, and the most accurate single system is a frontier LLM used directly\. What the pipeline delivers instead — once Dempster–Shafer is removed, the score is recalibrated, and risk control is applied — is*calibrated trust*: an auditable, guarantee\-backed ability to separate the cases a system should decide alone from the cases it should escalate\. On the automated subset the tuned engine is correct 96\.8% of the time with a documented coverage guarantee, and it escalates almost every case it would otherwise get wrong\. For high\-stakes legal work this property is worth more than a fraction of a point of AUROC, because the operative risk is not mediocre average accuracy but confident individual error\.

#### A deployment architecture that also answers the cost objection\.

A natural objection is economic: if a frontier LLM must read every fact paragraph of every case, the model cost may exceed the lawyer time it saves\. The literature supplies a well\-validated answer in the form of a cascade\. A cheap reader \(the term\-frequency model\) and the calibrated combination handle every case for negligible cost; only cases whose calibrated confidence falls in the uncertain middle are escalated to the LLM, which re\-scores them — and, for long cases, scores only the decision\-relevant paragraphs rather than the full record; only cases that remain uncertain after the LLM pass are escalated to a human\. This is the legal\-domain instantiation of confidence\-thresholded model cascades\(Chen et al\.,[2023](https://arxiv.org/html/2608.14617#bib.bib12)\)and the selective\-prediction cascade ofVarshney and Baral \([2021](https://arxiv.org/html/2608.14617#bib.bib24)\); crucially, conformal selection can be propagated through such a cascade without forfeiting the coverage guarantee\(Schuster et al\.,[2021](https://arxiv.org/html/2608.14617#bib.bib23)\)\. The long\-case strain is addressed by paragraph\-level rationale extraction\(Chalkidis et al\.,[2021](https://arxiv.org/html/2608.14617#bib.bib9)\)and by the chain\-aware damping of Section[7](https://arxiv.org/html/2608.14617#S7)\. The cascade spends the expensive model precisely where it changes the decision, which is the only regime in which an LLM pays for itself\.

#### The human\-review path is not free of risk\.

Deferring a case to a human is not a guaranteed improvement\.Hullman et al\.\([2025](https://arxiv.org/html/2608.14617#bib.bib19)\)show that the value of a prediction set to a reviewer depends on how it is presented and on the private information the reviewer holds, whileCresswell et al\.\([2024](https://arxiv.org/html/2608.14617#bib.bib13)\)provide randomised evidence that conformal sets, specifically, improve human accuracy relative to fixed\-size sets\. The implication for a legal\-AI triage system is that the review interface should surface the conformal set and the per\-fact evidence trace — the incriminating and exculpatory paragraphs that drove the decision — rather than a bare score, and that the deferral path should itself be monitored for the disparate error\-shifting that selective abstention can induce across applicant groups\.

#### A decision\-theoretic reading of the operating point\.

Letcerrc\_\{\\text\{err\}\}be the expected cost of a wrong auto\-cleared disposition,crevc\_\{\\text\{rev\}\}the cost of a human review, andcLLMc\_\{\\text\{LLM\}\}the marginal cost of an LLM evaluation\. For a case with calibrated violation probabilityss, auto\-clearing the arg\-max label incurs expected costcerr⋅min⁡\(s,1−s\)c\_\{\\text\{err\}\}\\cdot\\min\(s,1\-s\), while routing to review incurscrevc\_\{\\text\{rev\}\}\. Auto\-clearing is preferred whenmin⁡\(s,1−s\)≤crev/cerr\\min\(s,1\-s\)\\leq c\_\{\\text\{rev\}\}/c\_\{\\text\{err\}\}, i\.e\. when the calibrated confidence in the arg\-max label exceeds1−crev/cerr1\-c\_\{\\text\{rev\}\}/c\_\{\\text\{err\}\}\. Two consequences follow\. First, the optimal threshold is*class\-asymmetric*whenever the cost of wrongly clearing a violation differs from that of wrongly clearing a no\-violation — which is exactly why the tuned engine uses per\-class thresholds\. Second, the threshold rises withcerrc\_\{\\text\{err\}\}: the higher the stakes of an escaped error, the more the system should defer\. The risk\-control procedure of Section[7](https://arxiv.org/html/2608.14617#S7)can be read as enforcing a floor on accepted\-class error with a statistical guarantee rather than only in expectation\.

#### Implications for legal practice\.

Three practical implications follow for a firm or court considering such a system\. First, procurement claims should be scrutinised: a vendor claim of superior outcome*prediction*is, on this evidence, unlikely to survive a controlled comparison against a single strong model, whereas a claim of*calibrated selective automation with an auditable error floor*is both more modest and more defensible\. Second, the accuracy floor is a governance instrument: because it is certified out of sample, it can be written into a deployment policy and shown to a regulator or an insurer, converting “the system was confident” into “the system’s confident decisions are wrong at mostrrof the time, with confidence1−δ1\-\\delta\.” Third, the human\-review interface is not an afterthought: the evidence trace that drives each decision — which fact paragraphs were incriminating or exculpatory — is precisely what a reviewing lawyer needs to challenge or confirm the disposition\.

#### Why the fusion mathematics is, at best, redundant\.

Discrimination is a property of the information in the per\-fact scores, and neither Bayesian odds nor belief propagation adds information; they reparameterise it, and the reparameterisation is either calibration\-neutral \(belief propagation, after damping\) or calibration\-harmful \(Bayesian odds, via prior mismatch\)\. Dempster–Shafer adds a genuinely different combination rule, but on long, mutually reinforcing legal evidence chains that rule is the wrong one\. The conformal layer is the only component that contributes something the others cannot — a finite\-sample guarantee — and it does so as a wrapper around*any*score\. The practical engine is therefore thin: a good reader, an honest calibrated aggregate, and a risk\-controlled conformal gate\.

## 9\.Limitations and threats to validity

#### Single jurisdiction and task\.

Our evidence concerns binary violation prediction at the ECtHR\. The selective\-prediction conclusions should be re\-tested on contract\-obligation and patent tasks before being generalised; the cascade and calibration findings are likely more portable than the specific AUROC values\.

#### Weak per\-fact labels for the term\-frequency reader\.

The term\-frequency reader’s training labels are case\-level labels inherited by every paragraph, which is noisy\. This handicaps arm C in absolute terms, but the handicap is constant across the with\-/without\-fusion contrasts that carry our conclusions\.

#### Balanced\-sample size for the LLM arms\.

The LLM\-scored comparisons use 200 cases; confidence intervals on AUROC are correspondingly wide \(≈±0\.05\\approx\\pm 0\.05\)\. The full\-set validation of the tuned engine is not so limited\.

#### In\-sample calibration in the curve figures\.

The risk–coverage, reliability, and SAC figures apply isotonic recalibration in\-sample for curve*shape*; the rigorous out\-of\-sample guarantee is established separately in Table[5](https://arxiv.org/html/2608.14617#S7.T5)with proper calibration/test splitting\.

#### LLM exposure to ECtHR doctrine\.

The frontier models may have encountered ECtHR jurisprudence in pre\-training\. We mitigated by instructing facts\-only reasoning and by withholding labels, but we cannot exclude residual familiarity; this would, if anything, inflate the raw\-LLM arm and thus*strengthen*our finding that the fusion mathematics adds no discrimination on top of it\.

#### Minority\-class certification\.

As reported, certifying a tight floor on the rare no\-violation class is not achieved even atn=1000n=1000; deployments requiring such a guarantee need a larger calibration corpus\.

## 10\.Conclusion

We tested a popular proposal — that fusing an evidence graph, Bayesian odds, Dempster–Shafer combination, and conformal prediction yields a superior legal\-outcome predictor — directly, on 1,000 real ECtHR cases, with two frontier LLMs and a strong baseline, over roughly 4,750 controlled tests\. The proposal fails on its own terms: it does not improve discrimination, one of its components is actively unsafe, and a naive composition with an LLM destroys calibration\. But the exercise is not negative\. Once the unsafe component is removed, the score is recalibrated, and risk control is applied, the same machinery yields a deployable triage engine that auto\-clears cases at 96\.8% accuracy, lets only 0\.5% of errors escape, and catches 96\.3% of its would\-be mistakes for human review, with an auditable coverage guarantee\. The contribution of these pipelines to legal AI is therefore best understood not as sharper prediction but as*calibrated trust*: selective, guarantee\-backed automation that knows the edge of its competence and can prove it\. The right engine is thin — a capable reader, an honest calibrated aggregate, and a risk\-controlled conformal gate, deployed inside a cost\-aware cascade — and that thin engine is what we recommend building\.

## AI assistance disclosure

The author used AI assistance for portions of the experimental tooling, data processing, figure generation, and drafting\. Large language models \(Anthropic Claude Opus 4\.8 and OpenAI GPT\-5\.5\) were additionally objects of study and were used, as documented in the Methods, as per\-fact evidence estimators; a third model \(Google Gemini 3\.1 Pro\) contributed to the design review of Section[7](https://arxiv.org/html/2608.14617#S7)\. All quantitative results were produced by executable, reproducible code; all AI\-assisted text and analysis were reviewed, verified against the underlying results, and edited by the author, who is responsible for the final content\.

## Ethics statement

This study uses only publicly released, anonymised court judgments from the ECtHR HUDOC database via the LexGLUE and FairLex benchmarks; no private or personally identifying data were collected\. The work concerns a decision\-support system intended to operate with a human reviewer in the loop, and the paper explicitly cautions against unsupervised deployment and against assuming the human\-review path is risk\-free\.

## Data and reproducibility

All datasets are public: LexGLUE ECtHR Task A\(Chalkidis et al\.,[2022a](https://arxiv.org/html/2608.14617#bib.bib10)\)and FairLex ECtHR\(Chalkidis et al\.,[2022b](https://arxiv.org/html/2608.14617#bib.bib11)\), both sourced from the ECtHR HUDOC database\. All aggregators, the conformal and risk\-control procedures, the calibration methods, and the figure\-generation code are implemented in Python \(NumPy, scikit\-learn, NetworkX, Matplotlib\) and are organised so that each reported number traces to a saved result artefact\.

## References

- \(1\)
- Abbasi\-Yadkori et al\.\(2024\)Yasin Abbasi\-Yadkori, Ilja Kuzborskij, David Stutz, András György, et al\.2024\.Mitigating LLM Hallucinations via Conformal Abstention\.*arXiv preprint arXiv:2405\.01563*\(2024\)\.
- Alves et al\.\(2024\)Jean V\. Alves, Diogo Leitão, Sérgio Jesus, Marco O\. P\. Sampaio, Pedro Saleiro, Mário A\. T\. Figueiredo, and Pedro Bizarro\. 2024\.Cost\-Sensitive Learning to Defer to Multiple Experts with Workload Constraints\.*Transactions on Machine Learning Research*\(2024\)\.
- Angelopoulos and Bates \(2021\)Anastasios N\. Angelopoulos and Stephen Bates\. 2021\.A Gentle Introduction to Conformal Prediction and Distribution\-Free Uncertainty Quantification\.*arXiv preprint arXiv:2107\.07511*\(2021\)\.
- Angelopoulos et al\.\(2021\)Anastasios N\. Angelopoulos, Stephen Bates, Emmanuel J\. Candès, Michael I\. Jordan, and Lihua Lei\. 2021\.Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control\.*arXiv preprint arXiv:2110\.01052*\(2021\)\.
- Angelopoulos et al\.\(2022\)Anastasios N\. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster\. 2022\.Conformal Risk Control\.*arXiv preprint arXiv:2208\.02814*\(2022\)\.
- Bates et al\.\(2021\)Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I\. Jordan\. 2021\.Distribution\-Free, Risk\-Controlling Prediction Sets\.*J\. ACM*68, 6 \(2021\), 43:1–43:34\.
- Chalkidis et al\.\(2019\)Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras\. 2019\.Neural Legal Judgment Prediction in English\. In*Proceedings of ACL*\.
- Chalkidis et al\.\(2021\)Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis\. 2021\.Paragraph\-Level Rationale Extraction Through Regularization: A Case Study on European Court of Human Rights Cases\. In*Proceedings of NAACL\-HLT*\.
- Chalkidis et al\.\(2022a\)Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras\. 2022a\.LexGLUE: A Benchmark Dataset for Legal Language Understanding in English\. In*Proceedings of ACL*\.
- Chalkidis et al\.\(2022b\)Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Søgaard\. 2022b\.FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing\. In*Proceedings of ACL*\.
- Chen et al\.\(2023\)Lingjiao Chen, Matei Zaharia, and James Zou\. 2023\.FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance\.*arXiv preprint arXiv:2305\.05176*\(2023\)\.
- Cresswell et al\.\(2024\)Jesse C\. Cresswell, Yi Sui, Bhargava Kumar, and Noel Vouitsis\. 2024\.Conformal Prediction Sets Improve Human Decision Making\. In*Proceedings of ICML*\.
- El\-Yaniv and Wiener \(2010\)Ran El\-Yaniv and Yair Wiener\. 2010\.On the Foundations of Noise\-Free Selective Classification\.*Journal of Machine Learning Research*11 \(2010\), 1605–1641\.
- Galil et al\.\(2023\)Ido Galil, Mohammed Dabbah, and Ran El\-Yaniv\. 2023\.What Can We Learn from the Selective Prediction and Uncertainty Estimation Performance of 523 ImageNet Classifiers?\. In*Proceedings of ICLR*\.
- Geifman and El\-Yaniv \(2017\)Yonatan Geifman and Ran El\-Yaniv\. 2017\.Selective Classification for Deep Neural Networks\. In*Advances in Neural Information Processing Systems*\.
- Geifman and El\-Yaniv \(2019\)Yonatan Geifman and Ran El\-Yaniv\. 2019\.SelectiveNet: A Deep Neural Network with an Integrated Reject Option\. In*Proceedings of ICML*\.
- Guo et al\.\(2017\)Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q\. Weinberger\. 2017\.On Calibration of Modern Neural Networks\. In*Proceedings of ICML*\.
- Hullman et al\.\(2025\)Jessica Hullman, Yifan Wu, Dongping Xie, Ziyang Guo, and Andrew Gelman\. 2025\.Conformal Prediction and Human Decision Making\.*arXiv preprint arXiv:2503\.11709*\(2025\)\.
- Leitão et al\.\(2022\)Diogo Leitão, Pedro Saleiro, Mário A\. T\. Figueiredo, and Pedro Bizarro\. 2022\.Human\-AI Collaboration in Decision\-Making: Beyond Learning to Defer\. In*ICML Workshop on Human\-Machine Collaboration and Teaming*\.
- Madras et al\.\(2018\)David Madras, Toniann Pitassi, and Richard Zemel\. 2018\.Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer\. In*Advances in Neural Information Processing Systems*\.
- Mozannar and Sontag \(2020\)Hussein Mozannar and David Sontag\. 2020\.Consistent Estimators for Learning to Defer to an Expert\. In*Proceedings of ICML*\.
- Schuster et al\.\(2021\)Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay\. 2021\.Consistent Accelerated Inference via Confident Adaptive Transformers\. In*Proceedings of EMNLP*\.
- Varshney and Baral \(2021\)Neeraj Varshney and Chitta Baral\. 2021\.The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing\. In*Proceedings of ACL\-IJCNLP*\.
- Verma et al\.\(2023\)Rajeev Verma, Daniel Barrejón, and Eric Nalisnick\. 2023\.Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles\. In*Proceedings of AISTATS*\.
- Vovk et al\.\(2022\)Vladimir Vovk, Alexander Gammerman, and Glenn Shafer\. 2022\.*Algorithmic Learning in a Random World*\(2nd ed\.\)\.Springer\.
- Vovk and Petej \(2014\)Vladimir Vovk and Ivan Petej\. 2014\.Venn–Abers Predictors\. In*Proceedings of UAI*\.
- Wang et al\.\(2024\)Yumeng Wang, Sander Tilga, et al\.2024\.The Craft of Selective Prediction: Towards Reliable Case Outcome Classification — An Empirical Study on Legal NLP\.*arXiv preprint arXiv:2409\.18645*\(2024\)\.

## Appendix ADetailed experimental protocol

#### Reader training\.

The term\-frequency reader was fit on all fact paragraphs of the 9,000 LexGLUE training cases \(each paragraph labelled with its case’s binary outcome\), using a TF–IDF representation \(1–2 grams, 20,000\-term vocabulary, sublinear term frequency, minimum document frequency 3\) and L2\-regularised logistic regression\. The two LLM readers \(Claude Opus 4\.8, GPT\-5\.5\) scored the 200 balanced cases under an identical instruction that requested, per case, a holistic violation probability and one violation\-leaning score per fact paragraph, with explicit guidance to use the full\[0,1\]\[0,1\]range and avoid clustering at 0\.5, and without exposure to ground\-truth labels\. All 400 model outputs \(200 cases×\\times2 models\) were validated for JSON structure and for exact per\-paragraph count; there were no format failures\.

#### Aggregation\.

The base rateb=0\.898b=0\.898is the training positive rate\. The belief\-propagation damping in the internal analysis isλ=1/\(1\+0\.15⋅ln⁡\(1\+K\)\)\\lambda=1/\(1\+0\.15\\cdot\\ln\(1\+K\)\); the chain\-aware variant used by the tuned engine isλ=1/K\\lambda=1/\\sqrt\{K\}\. Dempster–Shafer uses an uncertainty mass of 0\.25 per fact and the pignistic transform for the decision; it is reported only in Section[5\.4](https://arxiv.org/html/2608.14617#S5.SS4)and excluded thereafter\.

#### Conformal and risk control\.

Split conformal prediction uses class\-conditional nonconformity with quantile level⌈\(n\+1\)​\(1−α\)⌉/n\\lceil\(n\+1\)\(1\-\\alpha\)\\rceil/n\. The tuned engine’s thresholds are selected by grid search to maximise the auto\-clear rate subject to a class\-conditional Clopper–Pearson upper bound \(confidence1−δ1\-\\delta,δ=0\.05\\delta=0\.05\) on the auto\-cleared error rate not exceeding the floorrr\. Validation uses 200 random 500/500 calibration/test partitions of the full 1,000\-case set; the curve figures additionally use isotonic recalibration in\-sample for shape and are not interpreted as guarantees\.

#### Resampling\.

AUROC confidence intervals use 200 bootstrap resamples; triage and tuned\-engine statistics are averaged over 30–200 random splits as indicated; abstention error\-reduction is averaged over 30 seeds per operating point\.

## Appendix BFull coverage and abstention tables

Table[B](https://arxiv.org/html/2608.14617#A2)reports empirical conformal coverage and abstention error\-reduction for all five arms, both models, at three target levels, on the balanced sample\. Coverage meets or exceeds target in every cell\. The abstention error\-reduction columns show that, on identical LLM evidence, the fusion arm B roughly doubles the selective benefit of the no\-fusion control B2; the term\-frequency control C2 also shows a large reduction, but this is an artefact of its poorer base calibration\.

Table 6: Empirical coverage \(cov\) and abstention error\-reduction \(red\) at target coverage1−α∈\{0\.95,0\.90,0\.80\}1\-\\alpha\\in\\\{0\.95,0\.90,0\.80\\\}, balanced 200\-case sample\. All coverage cells meet target \(the finite\-sample guarantee\)\.

## Appendix CSelective\-prediction summary

Table[C](https://arxiv.org/html/2608.14617#A3)reports AURC \(lower is better\) and SAC at three accuracy floors \(higher is better\) on the balanced sample for the Opus reader\. The calibrated term\-frequency\-plus\-combination score attains the lowest AURC and the highest SAC, consistent with the finding that the combination’s value is concentrated in the selective regime rather than in ranking\.

Table 7: AURC and Selective Accuracy Constraint \(Opus reader, balanced sample\)\.