When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents

# When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
Source: [https://arxiv.org/html/2607.18261](https://arxiv.org/html/2607.18261)
\(May 2026\)

###### Abstract

LLM agents are increasingly used as transaction compilers: a user states an intent in natural language, and the model emits a structured object that an API can execute\. JSON Schema and provider\-level structured\-output modes are useful because they remove a large class of parse failures, but they do not by themselves decide whether the object is a safe, faithful transaction\. We introduceOrderBench, a deterministic benchmark for restaurant ordering agents that separates syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances\. Across 2,400 Nebius Token Factory calls to four open models in prompt\-only and JSON\-schema modes, we find that schema\-valid output can still have large semantic error rates\. In the strongest model, both modes achieve 100% schema validity, yet semantic success remains near 80%; in weaker models, schema\-valid unsafe acceptances occur in double digits\. The result is a concrete engineering warning: structured output is a necessary interface layer, not a substitute for domain verification and fail\-closed execution\.

## 1Introduction

Production LLM agents often sit between unconstrained natural language and strict transactional APIs\. In food ordering, retail support, travel booking, health intake, and finance operations, the model is not merely formatting text: it is compiling a user’s request into an object that may reserve inventory, charge money, or trigger a safety\-relevant workflow\. The practical failure mode is therefore not only malformed JSON\. It is well\-formed JSON that says the wrong thing\.

Provider structured\-output modes and constrained decoding have made real progress on the formatting problem\. JSON Schema Draft 2020\-12 defines a standard contract language for JSON data\[[3](https://arxiv.org/html/2607.18261#bib.bib1)\], OpenAPI 3\.1 aligns API schemas with JSON Schema\[[6](https://arxiv.org/html/2607.18261#bib.bib2)\], and OpenAI’s Structured Outputs announcement reports perfect adherence on its internal complex\-schema evaluations for a supported model\[[5](https://arxiv.org/html/2607.18261#bib.bib3)\]\. Systems such as XGrammar further optimize grammar\-constrained generation for structured outputs\[[1](https://arxiv.org/html/2607.18261#bib.bib8)\]\. These advances are valuable, but their success criterion is usually structural: can the output be parsed, and does it match the declared type\-level shape?

This paper studies the next layer: semantic executability under a domain contract\. We focus on restaurant ordering because the domain is compact enough to evaluate exactly, but rich enough to expose production\-relevant edge cases: modifier negation, modifier scope over multiple items, unavailable catalog entries, allergen conflicts, dietary constraints, and cases where an allergen is safe only if a modifier is removed\. These are common business rules, not toy reasoning puzzles\.

Our contributions are:

- •OrderBench, a deterministic 300\-case benchmark with a small menu, hand\-coded oracles, and ten transactional edge\-case categories\.
- •A verifier that reports JSON validity, schema validity, status correctness, exact item semantics, constraint preservation, semantic success, and unsafe acceptance\.
- •A paired evaluation of prompt\-only JSON generation versus provider JSON\-schema mode using the same cases, models, prompts, and temperature\.
- •An empirical demonstration that schema validity can be perfect while semantic reliability remains insufficient for direct execution\.

## 2Related Work

### Tool and function calling\.

Gorilla and APIBench study whether models can select APIs and produce accurate arguments, emphasizing hallucinated or incorrect calls as a barrier to reliable tool use\[[8](https://arxiv.org/html/2607.18261#bib.bib5)\]\. The Berkeley Function Calling Leaderboard systematizes function\-calling evaluation across languages, APIs, SQL, relevance detection, and multi\-turn settings\[[7](https://arxiv.org/html/2607.18261#bib.bib7)\]\.τ\\tau\-bench moves closer to deployment by testing domain\-specific tool agents interacting with users and policies, and reports that even strong function\-calling agents remain inconsistent in realistic domains\[[9](https://arxiv.org/html/2607.18261#bib.bib6)\]\.OrderBenchis narrower than these benchmarks, but more surgical: it isolates the single\-turn contract compilation step and scores the difference between structurally valid and domain\-correct arguments\.

### Structured generation\.

JSON Schema is now a common target for structured LLM output\. Recent work benchmarks constrained decoding frameworks against JSON Schema test suites and structured\-output tasks\[[2](https://arxiv.org/html/2607.18261#bib.bib9)\], while XGrammar focuses on efficient grammar execution for structured generation\[[1](https://arxiv.org/html/2607.18261#bib.bib8)\]\. Our work assumes that structured generation is useful and asks what remains unsolved after the structure is valid\.

## 3Benchmark

OrderBenchdefines a menu with seven SKUs, item sizes, default modifiers, allowed additions/removals, allergens, and dietary tags\. Each example contains a customer utterance and a deterministic oracle object with five top\-level fields:status,items,constraints,clarification\_question, andreasons\. The status can beaccepted,needs\_clarification, orrejected\_safety\. The schema forbids additional properties and bounds quantities to executable integer values\.

The 300 examples are evenly distributed across ten categories: simple exact orders, quantity and size, negated modifiers, scoped modifiers, shared modifiers, allergen\-safe acceptances, allergen conflicts, dietary conflicts, unavailable items, and unavailable modifiers\. Utterances are deterministic paraphrases of hand\-written transactional templates; no model is used to create labels\. This design trades breadth for auditability: every oracle can be inspected and regenerated from source\.

## 4Evaluation

For each model and mode, we run all 300 cases at temperature zero through the Nebius Token Factory OpenAI\-compatible API\[[4](https://arxiv.org/html/2607.18261#bib.bib4)\]\. The two modes are:

- •Prompt\-only: the prompt asks the model to return only JSON and includes the schema as text\.
- •JSON schema: the same prompt plus provider strictjson\_schemaresponse formatting\.

We evaluate four models available from the provider at run time: GPT\-OSS 120B\-fast, Qwen3\-30B\-A3B, Llama\-3\.1\-8B, and Gemma\-2\-2B\. The exact provider model identifiers are recorded in the released JSONL outputs\. A compatibility pass excluded a fifth candidate that frequently exhausted the response budget and failed parsing under this prompt\. Raw model responses, provider usage, latency, parsed objects, and verifier results are written as JSONL\.

The main metric issemantic success\. For accepted orders, it requires correct status, exact multiset equality over SKU, quantity, size, additions, removals, and special instructions, plus exact preservation of stated allergens and dietary constraints\. For non\-accepted orders, it requires correct status, empty executable items, and exact constraints\. This is intentionally strict: it measures whether the model’s object can be used as a drop\-in transaction contract, not whether the answer is approximately useful\. Because exact equality does not imply equal severity, we also report a multi\-label error taxonomy: safety failures, catalog hallucinations, quantity/size errors, scope split errors, negation errors, and allergen\-preservation errors\.

The most important risk metric isunsafe acceptance\. We count an unsafe acceptance only when the model emitsstatus=acceptedfor an object that the verifier says must not be sent to execution: an allergen conflict, a dietary conflict, an unavailable item, an unavailable modifier, or an accepted item that still conflicts with a stated allergen\. Ordinary exactness misses, such as omitting a default size, are semantic failures but are not counted as unsafe\. For paired comparisons between prompt\-only and JSON\-schema modes, we report the schema\-minus\-prompt rate difference, a paired bootstrap 95% confidence interval over cases, and an exact McNemar test on discordant paired outcomes\.

## 5Results

Table[1](https://arxiv.org/html/2607.18261#S5.T1)reports the full paired evaluation\. The key pattern is that JSON and schema validity are much easier than semantic reliability\. Structured mode often improves schema validity for smaller models, but it does not guarantee better business decisions or safer executable objects\.

The strongest evaluated model, GPT\-OSS 120B\-fast, reaches 100% schema validity in both modes, but semantic success is 83\.0% in prompt\-only mode and 81\.3% in JSON\-schema mode\. Qwen3\-30B\-A3B also reaches 100% schema validity in both modes, while semantic success is only 31\.3% and 30\.7%, respectively, and unsafe acceptance remains around 15%\. For Llama\-3\.1\-8B, structured output raises schema validity from 68\.7% to 100\.0% and reduces unsafe acceptance from 13\.3% to 8\.3%, but semantic success remains 36\.0%\. Gemma\-2\-2B is the most severe counterexample: JSON\-schema mode produces 100% schema\-valid objects, yet semantic success is 2\.0% and unsafe acceptance is 41\.7%\.

Table 1:Main results\. Values other thannnare percentages\. “Semantic” is exact domain\-contract success; “Unsafe” is accepting an order that should not be executable\.Table[2](https://arxiv.org/html/2607.18261#S5.T2)aggregates semantic success by category and mode\. The hard categories are not generic JSON formatting tasks\. They are domain boundary tasks: item availability, modifier availability, and user constraints that must change the execution status\. Because this table averages over models, Appendix Table[6](https://arxiv.org/html/2607.18261#A1.T6)provides the model\-by\-category heatmap used to check whether a category\-level effect is broad or driven by one weak model\.

Table 2:Category\-level results aggregated across models\. Values are percentages\.Table[3](https://arxiv.org/html/2607.18261#S5.T3)separates exact semantic failures by operational severity\. The labels are multi\-label rather than mutually exclusive; for example, accepting a menu\-external modifier is both a catalog error and an unsafe acceptance\. The table makes the strictness of semantic success easier to interpret: quantity/size errors and scope errors are common, but the deployment critical categories are unsafe acceptances and catalog hallucinations\.

Table 3:Multi\-label error taxonomy aggregated across models\. Values are percentages of all cases in each mode\. Categories do not sum to 100%\.Table 4:Representative schema\-valid failures\. These examples illustrate why semantic exactness, unsafe acceptance, and catalog errors are reported separately\.Table[5](https://arxiv.org/html/2607.18261#S5.T5)reports the paired uncertainty estimates promised in the evaluation protocol\. The headline semantic differences for GPT\-OSS and Qwen are small and statistically indistinguishable from zero\. For Llama\-3\.1\-8B, JSON\-schema mode improves schema validity and lowers unsafe acceptance, but the semantic\-success gain is not significant at the 5% level\. Gemma’s semantic\-success improvement is statistically nonzero but remains practically inadequate at 2\.0% absolute success\. Unsafe acceptance shows a different pattern: JSON\-schema mode significantly reduces unsafe acceptance for Gemma and Llama\-3\.1\-8B, but does not eliminate the risk\.

Table 5:Paired prompt\-only versus JSON\-schema comparison\.Δ\\Deltais JSON\-schema minus prompt\-only percentage points\. Confidence intervals are paired bootstrap 95% intervals over cases;ppis an exact McNemar test\. Lower is better for unsafe acceptance\.
## 6Discussion

The experiments support a simple deployment rule: do not let schema\-valid model output directly execute a transaction\. A JSON Schema can ensure thatitemsis an array andquantityis an integer\. It cannot ensure that a requested allergen conflict was rejected, that “one without onion and one normal” was split into two item objects, or that “pineapple” was not silently invented as a modifier\. Those checks require a domain verifier\.

This does not make structured output unimportant\. It makes structured output the first layer in a larger contract stack\. A practical architecture is:

1. 1\.Require strict structured output to eliminate parse and shape failures\.
2. 2\.Validate the object against the business catalog and safety policy\.
3. 3\.Fail closed on verifier errors, using clarification rather than auto\-repair for safety\-critical fields\.
4. 4\.Log model output and verifier decisions as paired artifacts for regression testing\.

## 7Limitations

OrderBenchis synthetic and single\-turn\. Its strength is auditability, not coverage of all restaurant language\. The menu is intentionally small, so the reported rates should not be read as deployment estimates for a particular vendor or restaurant\. We also evaluate one prompt and one provider interface; prompt engineering, fine\-tuning, retrieval over a larger catalog, or a specialized planner\-verifier loop could improve absolute performance\. The central claim is narrower: schema validity alone is not a sufficient reliability metric for transactional execution\.

## 8Conclusion

Structured\-output APIs substantially reduce interface friction, but production agents need semantic contracts, not only syntactic contracts\.OrderBenchshows that models can satisfy a strict JSON shape while still emitting unsafe or incorrect executable orders\. For transaction\-compiling agents, the engineering unit of reliability should be the verified domain action\.

## Appendix AModel\-by\-Category Heatmap

Table[6](https://arxiv.org/html/2607.18261#A1.T6)breaks the category averages in Table[2](https://arxiv.org/html/2607.18261#S5.T2)down by model and mode\. Darker cells indicate higher semantic success\. The heatmap shows, for example, that the low aggregate score for scope is not solely a Gemma artifact: Qwen and Llama also struggle with split\-item semantics, while GPT\-OSS is substantially stronger\.

Table 6:Semantic success by category, model, and mode\. P = prompt\-only; S = JSON\-schema\. Values are percentages; darker cells are higher\.
## References

- \[1\]Y\. Dong, C\. F\. Ruan, Y\. Cai, R\. Lai, Z\. Xu, Y\. Zhao, and T\. Chen\(2024\)XGrammar: flexible and efficient structured generation engine for large language models\.External Links:2411\.15100Cited by:[§1](https://arxiv.org/html/2607.18261#S1.p2.1),[§2](https://arxiv.org/html/2607.18261#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]S\. Geng, H\. Cooper, M\. Moskal, S\. Jenkins, J\. Berman, N\. Ranchin, R\. West, E\. Horvitz, and H\. Nori\(2025\)JSONSchemaBench: a rigorous benchmark of structured outputs for language models\.External Links:2501\.10868Cited by:[§2](https://arxiv.org/html/2607.18261#S2.SS0.SSS0.Px2.p1.1)\.
- \[3\]JSON Schema Organization\(2022\)JSON Schema Draft 2020\-12\.Note:[https://json\-schema\.org/draft/2020\-12](https://json-schema.org/draft/2020-12)Accessed 2026\-05\-16Cited by:[§1](https://arxiv.org/html/2607.18261#S1.p2.1)\.
- \[4\]Nebius\(2026\)Nebius Token Factory API Reference\.Note:[https://docs\.tokenfactory\.nebius\.com/api\-reference/introduction](https://docs.tokenfactory.nebius.com/api-reference/introduction)Accessed 2026\-05\-16Cited by:[§4](https://arxiv.org/html/2607.18261#S4.p1.1)\.
- \[5\]OpenAI\(2024\)Introducing structured outputs in the API\.Note:[https://openai\.com/index/introducing\-structured\-outputs\-in\-the\-api/](https://openai.com/index/introducing-structured-outputs-in-the-api/)Accessed 2026\-05\-16Cited by:[§1](https://arxiv.org/html/2607.18261#S1.p2.1)\.
- \[6\]OpenAPI Initiative\(2021\)OpenAPI Specification v3\.1\.0\.Note:[https://spec\.openapis\.org/oas/v3\.1\.0\.html](https://spec.openapis.org/oas/v3.1.0.html)Accessed 2026\-05\-16Cited by:[§1](https://arxiv.org/html/2607.18261#S1.p2.1)\.
- \[7\]S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez\(2025\)The berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 48371–48392\.Note:[https://proceedings\.mlr\.press/v267/patil25a\.html](https://proceedings.mlr.press/v267/patil25a.html)Cited by:[§2](https://arxiv.org/html/2607.18261#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez\(2023\)Gorilla: large language model connected with massive APIs\.External Links:2305\.15334Cited by:[§2](https://arxiv.org/html/2607.18261#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan\(2024\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.External Links:2406\.12045Cited by:[§2](https://arxiv.org/html/2607.18261#S2.SS0.SSS0.Px1.p1.1)\.