JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Hugging Face Daily Papers Papers
llm-as-a-judge decision-only-judge confidence-threshold cost-efficiency cascade-evaluation

Summary

This paper introduces JEV-as-a-Judge, a cost-effective evaluation method for LLMs that uses a decision-only judge with confidence thresholds to accept certain verdicts and escalate uncertain ones, achieving comparable accuracy to state-of-the-art models at significantly lower cost.

LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.
Original Article
View Cached Full Text

Cached at: 09/23/26, 03:34 PM

Paper page - JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Source: https://huggingface.co/papers/2609.26550

Abstract

LLM-as-a-judgeenablesevaluationacrossdiversetasks,butinferencecostandconfidencereliabilitybecomecriticalatscale.Westudywhetheradecision-onlyjudgecanprovideaneconomicalfirstpassandidentifywhenstrongerevaluationisneeded.Comparingjev-as-a-judgewithsixteengenerativeandreward-modeljudges,withblindedhumanadjudication,wefinditwithinthreepercentagepointsofastate-of-the-artLLMjudge,ourstrongestcomparator,onordinarypreferenceandevidence-groundedfactualityat0.36%ofthecomparator’sfee.Largergapsarisewhenjudgmentsrequirecheckingaderivationorresistinganelaboratelywrittenwronganswer.Onseveralbenchmarks,JEV’sgaptothiscomparatorisconcentratedinlow-confidencedecisions.Afrozencascadethatacceptsconfidentverdictsandescalatesuncertainonesretains99%ofthecomparator’saccuracyatlowercost.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.26550

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.26550 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.26550 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.26550 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Codifying the Judge: Scalable Evaluation via Program Distillation

arXiv cs.AI

This paper introduces PAJAMA, a system that distills LLM-as-a-judge decision logic into programmatic judges, eliminating per-sample API costs while matching the performance of a 13B LLM judge, and providing transparency and efficiency improvements for scalable evaluation.