JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Hugging Face Daily Papers Papers
llm-as-a-judge decision-only-judge confidence-threshold cost-efficiency cascade-evaluation

Summary

This paper introduces JEV-as-a-Judge, a cost-effective evaluation method for LLMs that uses a decision-only judge with confidence thresholds to accept certain verdicts and escalate uncertain ones, achieving comparable accuracy to state-of-the-art models at significantly lower cost.

LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.
Original Article
View Cached Full Text

Cached at: 09/23/26, 03:34 PM

Paper page - JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Source: https://huggingface.co/papers/2609.26550

Abstract

LLM-as-a-judgeenablesevaluationacrossdiversetasks,butinferencecostandconfidencereliabilitybecomecriticalatscale.Westudywhetheradecision-onlyjudgecanprovideaneconomicalfirstpassandidentifywhenstrongerevaluationisneeded.Comparingjev-as-a-judgewithsixteengenerativeandreward-modeljudges,withblindedhumanadjudication,wefinditwithinthreepercentagepointsofastate-of-the-artLLMjudge,ourstrongestcomparator,onordinarypreferenceandevidence-groundedfactualityat0.36%ofthecomparator’sfee.Largergapsarisewhenjudgmentsrequirecheckingaderivationorresistinganelaboratelywrittenwronganswer.Onseveralbenchmarks,JEV’sgaptothiscomparatorisconcentratedinlow-confidencedecisions.Afrozencascadethatacceptsconfidentverdictsandescalatesuncertainonesretains99%ofthecomparator’saccuracyatlowercost.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.26550

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.26550 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.26550 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.26550 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

arXiv cs.CL

The paper evaluates Jev, a non-generative decision model, against a frontier LLM (gpt-6-astra) for RAG claim verification on 495 LLM-AggreFact claims, finding statistically indistinguishable headline accuracy at a 187-fold cost difference but highly unequal error profiles, and showing that a confidence-gated cascade can cut costs without beating a matched random-escalation control.

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Hugging Face Daily Papers

The paper introduces LLM-as-Jev, an architecture-preserving framework that extracts calibrated decision distributions from LLM next-token probabilities, and shows modern LLMs are already effective Jev-style decision models, with fine-tuning giving targeted gains on weaker models and specific tasks while KL anchors prevent conversational degradation.

Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers

Hacker News Top

A Red Hat Developer article benchmarks TypeSafe AI's Jev "System One" decision models against LLM-as-a-judge guardrails and traditional zero-shot text classifiers, concluding that decision models don't clearly outperform the established alternatives despite their schema-safe, low-latency, zero-shot benefits.