LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Hugging Face Daily Papers Papers
llm decision-making fine-tuning calibrated-decisions listwise-loss lora kl-divergence

Summary

The paper introduces LLM-as-Jev, an architecture-preserving framework that extracts calibrated decision distributions from LLM next-token probabilities, and shows modern LLMs are already effective Jev-style decision models, with fine-tuning giving targeted gains on weaker models and specific tasks while KL anchors prevent conversational degradation.

Jev-style decision models return categorical probability distributions over predefined options without generating free-form text, enabling software systems to act on their outputs directly. In this work, we investigate the extent to which general-purpose LLMs already possess this capability out of the box, and when fine-tuning is actually necessary. We present LLM-as-Jev, an architecture-preserving framework that extracts calibrated decisions directly from next-token probabilities over bracketed numeric identifiers. LLM-as-Jev provides both a training-free inference recipe and a fine-tuning objective that optimizes candidate selection via a tree-factorized listwise loss while anchoring auxiliary predictions to the base model using KL divergence penalties. Evaluating on Qwen3.5-4B and Qwen3-0.6B, we find that modern LLMs are inherently effective decision models: without training, the 4B model matches community Jev-style models built on the same backbone, outperforms letter-logit readouts, supports arbitrary option counts, and natively handles multimodal decisions over images. Fine-tuning provides targeted rather than universal benefits -- substantially improving weaker models and specific tasks (such as many-option intent routing), but offering diminishing returns for strong backbones. Crucially, our KL anchors prevent behavioral degradation in conversational text generation, with LoRA delivering the strongest performance on capable models.
Original Article
View Cached Full Text

Cached at: 10/06/26, 08:57 PM

Paper page - LLM-as-Jev: LLMs Are Already Jev-Style Decision Models – When and How to Fine-Tune Them

Source: https://huggingface.co/papers/2610.02076

Abstract

Jev-styledecisionmodelsreturncategoricalprobabilitydistributionsoverpredefinedoptionswithoutgeneratingfree-formtext,enablingsoftwaresystemstoactontheiroutputsdirectly.Inthiswork,weinvestigatetheextenttowhichgeneral-purposeLLMsalreadypossessthiscapabilityoutofthebox,andwhenfine-tuningisactuallynecessary.WepresentLLM-as-Jev,anarchitecture-preservingframeworkthatextractscalibrateddecisionsdirectlyfromnext-tokenprobabilitiesoverbracketednumericidentifiers.LLM-as-Jevprovidesbothatraining-freeinferencerecipeandafine-tuningobjectivethatoptimizescandidateselectionviaatree-factorizedlistwiselosswhileanchoringauxiliarypredictionstothebasemodelusingKLdivergencepenalties.EvaluatingonQwen3.5-4BandQwen3-0.6B,wefindthatmodernLLMsareinherentlyeffectivedecisionmodels:withouttraining,the4BmodelmatchescommunityJev-stylemodelsbuiltonthesamebackbone,outperformsletter-logitreadouts,supportsarbitraryoptioncounts,andnativelyhandlesmultimodaldecisionsoverimages.Fine-tuningprovidestargetedratherthanuniversalbenefits--substantiallyimprovingweakermodelsandspecifictasks(suchasmany-optionintentrouting),butofferingdiminishingreturnsforstrongbackbones.Crucially,ourKLanchorspreventbehavioraldegradationinconversationaltextgeneration,withLoRAdeliveringthestrongestperformanceoncapablemodels.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.02076

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02076 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02076 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02076 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers

Hacker News Top

A Red Hat Developer article benchmarks TypeSafe AI's Jev "System One" decision models against LLM-as-a-judge guardrails and traditional zero-shot text classifiers, concluding that decision models don't clearly outperform the established alternatives despite their schema-safe, low-latency, zero-shot benefits.

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Hugging Face Daily Papers

This paper introduces JEV-as-a-Judge, a cost-effective evaluation method for LLMs that uses a decision-only judge with confidence thresholds to accept certain verdicts and escalate uncertain ones, achieving comparable accuracy to state-of-the-art models at significantly lower cost.