Structured output reliability with LLMs — 3-month production learnings

Been shipping structured JSON output from LLMs in production for a health app. Here's what I've learned about reliability. The problem: get a 70B model to return valid JSON matching a strict schema, every time. What I tried: Attempt 1: "Return JSON." No schema. 40% valid output. Attempt 2: Detailed schema in prompt. 75% valid. Attempt 3: JSON mode enabled (Groq/OpenAI/Anthropic all support). 92%. Attempt 4: JSON mode + schema validator + retry loop with error surfaced back. 99.5%. What still fails: - Emoji in fields (invalidates JSON parsing) - Very long generated fields (context length errors) - Rare "the model just doesn't return JSON" (0.5% baseline you can't kill) For production, my flow: LLM call in JSON mode with schema Parse. If fails, log the raw output for analysis Validate against Zod schema If schema fails, retry ONCE with the validation error in the prompt If still fails, use a static fallback Model tier matters less than I expected. Prompt scaffolding matters more. Question: anyone doing something more sophisticated? Curious about output-guided generation via Outlines or LMQL in production.