Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

Hugging Face Daily Papers Papers
edge-computing llm-substitution low-latency service-orchestration decision-models edge-ai

Summary

This paper replaces LLM-based decision making in edge service orchestration with Jev decision models, reporting 22.7–64.5% lower median decision latency and far higher request completion rates under load, while noting wide contracts mark the limit of substitution.

Natural-language service requests can require a language-model decision before execution starts, consuming part of the request's latency budget. We integrate Jev's decision-oriented application programming interface (API) into edge service orchestration to reduce this overhead while retaining service completion. The integration extracts four to eight bounded intent fields and applies a shared validator, admission policy, and scheduler, accounting for decision waiting throughout the request timeline. We compare Jev, two self-hosted decision models, and three hosted large language models (LLMs) on 8,280 verified requests and on a live admission path with modeled execution and a real optical character recognition service. Across 33 test conditions, Jev reduces median decision latency by 22.7-64.5% relative to the fastest LLM. This latency barely moves with input size, contract width, or catalog size. On four-field contracts, Jev's API fees per correct decision are 59.7-80.9% lower at a cost of a few exact-match points, while wide contracts mark the limit of the substitution. Receiving the service catalog with each request, Jev names unseen services as accurately as known ones. On the live admission path, Jev keeps 0.91-0.95 of requests exact and on time at loads where the LLMs fall below 0.1. Since caching repeated descriptions gives the interpreters nearly the same latency, Jev's gain lies in fresh decisions. These results support decision-model substitution for latency-bound admission on bounded contracts.
Original Article
View Cached Full Text

Cached at: 10/03/26, 04:33 AM

Paper page - Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

Source: https://huggingface.co/papers/2609.22753

Abstract

Natural-languageservicerequestscanrequirealanguage-modeldecisionbeforeexecutionstarts,consumingpartoftherequest’slatencybudget.WeintegrateJev’sdecision-orientedapplicationprogramminginterface(API)intoedgeserviceorchestrationtoreducethisoverheadwhileretainingservicecompletion.Theintegrationextractsfourtoeightboundedintentfieldsandappliesasharedvalidator,admissionpolicy,andscheduler,accountingfordecisionwaitingthroughouttherequesttimeline.WecompareJev,twoself-hosteddecisionmodels,andthreehostedlargelanguagemodels(LLMs)on8,280verifiedrequestsandonaliveadmissionpathwithmodeledexecutionandarealopticalcharacterrecognitionservice.Across33testconditions,Jevreducesmediandecisionlatencyby22.7-64.5%relativetothefastestLLM.Thislatencybarelymoveswithinputsize,contractwidth,orcatalogsize.Onfour-fieldcontracts,Jev’sAPIfeespercorrectdecisionare59.7-80.9%loweratacostofafewexact-matchpoints,whilewidecontractsmarkthelimitofthesubstitution.Receivingtheservicecatalogwitheachrequest,Jevnamesunseenservicesasaccuratelyasknownones.Ontheliveadmissionpath,Jevkeeps0.91-0.95ofrequestsexactandontimeatloadswheretheLLMsfallbelow0.1.Sincecachingrepeateddescriptionsgivestheinterpretersnearlythesamelatency,Jev’sgainliesinfreshdecisions.Theseresultssupportdecision-modelsubstitutionforlatency-boundadmissiononboundedcontracts.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.22753

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.22753 in a model README.md to link it from this page.

Datasets citing this paper1

#### OniReimu/Edge-Computing-JEV Viewer• Updated1 day ago • 11.5k • 321

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.22753 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Hugging Face Daily Papers

The paper introduces LLM-as-Jev, an architecture-preserving framework that extracts calibrated decision distributions from LLM next-token probabilities, and shows modern LLMs are already effective Jev-style decision models, with fine-tuning giving targeted gains on weaker models and specific tasks while KL anchors prevent conversational degradation.

Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers

Hacker News Top

A Red Hat Developer article benchmarks TypeSafe AI's Jev "System One" decision models against LLM-as-a-judge guardrails and traditional zero-shot text classifiers, concluding that decision models don't clearly outperform the established alternatives despite their schema-safe, low-latency, zero-shot benefits.