The author shares findings on the lack of portability for JSON Schema structured outputs across AI providers like OpenAI, Gemini, and Anthropic, highlighting inconsistencies in constraint enforcement and offering practical advice for robust integration.
I write a lot of Structured Outputs code, and the annoying part is not the basic API call anymore. The annoying part is figuring out which parts of your JSON Schema are actually enforced, rejected, silently simplified, or accepted-but-not-enforced by each provider. A small example: OpenAI documents `anyOf` as supported for Structured Outputs, but the real story has caveats. The root schema cannot be `anyOf`, nested schemas must fit OpenAI's supported subset, and there are real-world issue threads where valid-looking `anyOf` schemas produce confusing 400s. One case I found: object variants inside `anyOf` sharing the same first key can fail with an unhelpful "Invalid response_format provided" error. That is manageable if you only use one provider. It gets messy when you try to run the same Pydantic/Zod schema across OpenAI, Gemini, Anthropic, and xAI. I did a small adversarial test suite for JSON Schema constraints: give the provider a schema, then prompt the model to violate a specific constraint, and check whether the output is actually constrained. Some examples where simple schema portability breaks: - `Field(min_length=5, max_length=8)` or `pattern` may be enforced by one provider, ignored by another, or stripped from the schema and validated client-side by an SDK. - `allOf` from inheritance is especially dangerous. OpenAI strict mode rejects it, Gemini/xAI returned `{}` in my tests, and Anthropic supports `allOf` only with limitations. - `anyOf` works in some places, but top-level unions, tool schemas, provider complexity limits, and variant shape can all break differently. - "OpenAI-compatible endpoint" does not mean "OpenAI-compatible schema behavior." A trivial Pydantic example may port cleanly, but a real schema with bounds, unions, refs, or inheritance often does not. A few practical takeaways from the tests: - Treat `strict: true` as mandatory for OpenAI Structured Outputs. Without it, the schema can look present but not actually constrain the generation. - Keep app-side validation even when the provider claims schema adherence. Refusals, truncation, SDK transformations, and unsupported keywords still exist. - Prefer flat provider-facing schemas over inheritance-heavy models. Inheritance often turns into `allOf`, and `allOf` is where portability gets ugly fast. - Use enums and explicit object structure for critical routing decisions instead of relying on regexes, string length, or numeric bounds across providers. - Test constraints adversarially: schema says one thing, prompt asks for a violation. If the provider lets it through once, assume you need validation or a different schema shape. The most useful mental model I ended up with: > The same schema can be accepted, rejected, silently simplified, or accepted-but-not-enforced depending on the provider. So for production I would not treat provider Structured Outputs as a generic JSON Schema runtime. I would keep a canonical semantic model, generate provider-specific schemas from it, and adversarially test the exact constraints I rely on. I wrote up the findings and also turned them into a coding-agent skill. The goal is to help agents stop generating plausible-but-wrong Structured Outputs code, like putting the schema in the prompt, forgetting `strict: true`, or using schema patterns that a target provider does not actually enforce. Curious how others are handling this: Are you keeping one canonical schema with provider adapters, separate schemas per provider, or just validating/retrying everything after the model response?
OpenAI introduces Structured Outputs in their API, enabling developers to reliably get valid JSON schema outputs from language models, improving integration with downstream systems and reducing parsing errors.
Open Envelope provides an open JSON Schema for defining AI agent teams with roles, handoffs, human checkpoints, and access policies, aiming for portability across frameworks.
This paper audits LangChain, AutoGPT, and OpenAI Agents SDK for architectural safety guarantees and finds no native compliance with containment principles, demonstrating that memory poisoning can cause persistent failures; it introduces lightweight mechanisms to eliminate such attacks.
This article highlights that many AI agent projects fail in production not because of model quality, but because teams launch without clearly defining what constitutes failure, missing critical edge cases that lead to confident incorrect outputs.
The article argues that open-weight AI models are catching up to closed ones not via distillation but due to the modularisation of the AI stack—stable interfaces (Transformer architecture, OpenAI-compatible APIs, agentic harnesses) allow innovations to diffuse rapidly across the ecosystem, shrinking the capability gap while keeping a massive price advantage, potentially leading to a commoditisation of frontier AI.