A cheap trick for reliable structured output: feed the validation error back into the retry

Reddit r/LocalLLaMA Tools
structured-output validation-error retry self-correction llm-output error-handling prompt-engineering

Summary

A practical technique for improving structured output generation from LLMs by feeding validation errors back into retry prompts, allowing the model to self-correct rather than blindly retrying. The method involves describing the error in model-friendly terms and providing the previous output for editing.

If you generate structured output from an LLM and validate against a schema, you know the failure mode: usually fine, occasionally a missing field or an unparseable response. The common fix is a retry, but a plain retry is the same prompt at the same temperature, so you are just re-rolling. What worked much better for me was making the retry self-correcting: when validation fails, put the validation error and the model's own previous output back into the next prompt and ask it to fix that specific thing. It edits instead of regenerating. python except ValidationError as e: attempts += 1 error_message = f""" The last response failed validation due to this error: <error>{format_error_for_llm(e)}</error> Fix the error and return the corrected data: <data>{serialize(response).decode()}</data> """ response = None # next loop appends error_message to the prompt Two details matter: describe the error for the model, not for a log ("field X must be an int, you sent a string"), and hand back its own prior output as the thing to correct. Tradeoffs: an extra call plus a longer prompt on failures (cap attempts); it only works when the bad output is parseable enough to feed back; and if you also fail over between providers, do not count the swap as an attempt. This is from a RAG platform I built. How are others handling this, constrained decoding / grammars, or feedback loops like this?
Original Article

Similar Articles

When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL

arXiv cs.LG

This paper frames LLM-generated reward shaping for sparse structured RL as a debugging problem, identifying failure modes like reward flooding and semantic misunderstanding. The authors propose diagnostic-driven iterative refinement, achieving dramatic success rate improvements (e.g., DoorKey-8×8 from 2.3% to 97.6%) compared to one-shot generation.

@jakevin7: Sharing a god-tier review prompt methodology. The LLM self-correction survey 'When Can LLMs Actually Correct Their Own Mistakes?' concludes that without reliable external feedback such as test results or tool outputs, a model relying only on self-reflection often cannot steadily correct errors...

X AI KOLs Following

Shared a prompt methodology based on LLM self-correction research, emphasizing that self-checking is limited without external feedback, and recommending progressively enhanced prompting strategies such as adversarial review.

Pigeonholing: Bad prompts hurt models to collapse and make mistakes

arXiv cs.CL

This paper introduces 'pigeonholing,' a phenomenon where bad prompts cause LLMs to collapse and repeat errors, leading to a 38-40% performance drop. Experiments across 10 tasks and 10 models show worsening with more conversation turns, and propose RLVR with synthetic errors as a mitigation.