Structured output reliability with LLMs — 3-month production learnings

Reddit r/artificial 07/14/26, 04:51 PM Tools

llm structured-output json production reliability schema-validation prompting

Summary

The article shares production learnings for reliably generating structured JSON output from LLMs, covering methods like JSON mode, schema validation, and retry loops, achieving 99.5% validity.

Been shipping structured JSON output from LLMs in production for a health app. Here's what I've learned about reliability. The problem: get a 70B model to return valid JSON matching a strict schema, every time. What I tried: Attempt 1: "Return JSON." No schema. 40% valid output. Attempt 2: Detailed schema in prompt. 75% valid. Attempt 3: JSON mode enabled (Groq/OpenAI/Anthropic all support). 92%. Attempt 4: JSON mode + schema validator + retry loop with error surfaced back. 99.5%. What still fails: - Emoji in fields (invalidates JSON parsing) - Very long generated fields (context length errors) - Rare "the model just doesn't return JSON" (0.5% baseline you can't kill) For production, my flow: LLM call in JSON mode with schema Parse. If fails, log the raw output for analysis Validate against Zod schema If schema fails, retry ONCE with the validation error in the prompt If still fails, use a static fallback Model tier matters less than I expected. Prompt scaffolding matters more. Question: anyone doing something more sophisticated? Curious about output-guided generation via Outlines or LMQL in production.

Original Article

Similar Articles

When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents

arXiv cs.AI

The paper introduces OrderBench, a benchmark for restaurant ordering LLM agents that evaluates semantic reliability beyond schema validity, demonstrating that structured output modes can achieve perfect schema validity while still having high semantic error rates.

@akshay_pachaar: https://x.com/akshay_pachaar/status/2064700531600458093

X AI KOLs Following

This article explains how to use GRPO to fine-tune an LLM (Qwen3-8B) for reliable JSON structured output, improving schema accuracy from 62% to 82%, surpassing GPT-4.1's 58%.

A cheap trick for reliable structured output: feed the validation error back into the retry

Reddit r/LocalLLaMA

A practical technique for improving structured output generation from LLMs by feeding validation errors back into retry prompts, allowing the model to self-correct rather than blindly retrying. The method involves describing the error in model-friendly terms and providing the previous output for editing.

Producing Structured Outputs from LLMs with Constrained Sampling

Reddit r/LocalLLaMA

Discusses methods for generating structured outputs from large language models using constrained sampling techniques.

I made a small local model (llama3.2 3B) reliably extract structured JSON from documents - the hard part wasn't the model, it was everything around it

Reddit r/AI_Agents

A developer shares lessons from building a local document-to-JSON extractor using llama3.2 3B on Ollama, highlighting that deterministic post-processing and schema-constrained outputs matter more than model size, while seeking feedback on hallucination and context truncation issues with long documents.