Tag
The article critically examines Jev, an AI model optimized for quick, structured outputs and cost-efficiency, while questioning the reliability of its judgments compared to larger models.
Jev is a generalist System One AI model designed for making focused, structured judgments with probabilistic outputs. This article provides a beginner's guide and introduces an interactive playground for experimentation.
Modal powers AI applications like 'Wet Claudes' and structured outputs, and is hosting the Runtime conference on October 1st to gather the community for discussions.
The post discusses verification patterns for AI agents to ensure reliability, suggesting techniques like separating actor and verifier, forcing structured outputs, and using evidence caps to prevent hallucinations and misbehavior.
The article reports benchmarking results for a deterministic AI financial verification engine, showing perfect performance on structured claims (66/66) but poor performance when LLM-generated claims are used (19/66), indicating a translation gap between LLMs and formal systems.
The article describes a technique for classifying e-commerce queries with LLMs by having the model hallucinate hypothetical classifications, then mapping them to the real taxonomy using embeddings, which is cheaper and simpler than constrained structured outputs.
A study showing that language models hallucinate when required to fill structured fields like JSON, even when they would honestly abstain in free text. The PhantomFill benchmark measures coerced fabrication rates.
Chiron is an exact-or-refuse evidence gate for structured outputs that verifies claims as VERIFIED, REFUTED, or REFUSED, with a public evaluation history and source-available code.
A detailed guide on using Pydantic AI and Pydantic Evals to ensure structured outputs from LLMs, covering shape validation, content correctness, and open-ended judgement.
This paper investigates how fine-tuning vision-language models to produce dense coordinate lists creates a controllable interference surface, finding that duplicate pressure can be removed without sacrificing localization accuracy.
Explores a common failure mode in recursive language models (RLMs) where free-text subagent responses cause issues, and presents a solution using structured outputs to improve reliability, illustrated with a long-context question-answering example from NarrativeQA.
Attended the Applied AI Conference in Berlin and gave a talk on building document agents, including a detailed walkthrough of LobsterX, a document-processing agent built with LlamaIndex that uses structured outputs and event-driven workflows.
This paper introduces the concept of 'constraint tax'—the accuracy loss caused by structured output constraints in small language models—and presents a measurement protocol to quantify the tradeoff between validity and correctness.
The author shares findings on the lack of portability for JSON Schema structured outputs across AI providers like OpenAI, Gemini, and Anthropic, highlighting inconsistencies in constraint enforcement and offering practical advice for robust integration.
This paper identifies a safety threshold in on-policy distillation with reward extrapolation, beyond which structured output tasks lose format preservation. Empirical validation shows that operating below this threshold allows a 1.7B student model to match an 8B SFT baseline on Amazon Fashion tasks with one-fifth the parameters.
OpenAI releases o1 model to API with production-ready features including function calling, structured outputs, vision capabilities, and 60% lower latency than o1-preview. Additional developer tools include Realtime API improvements, Preference Fine-Tuning, and new Go and Java SDKs.
OpenAI introduces Structured Outputs in their API, enabling developers to reliably get valid JSON schema outputs from language models, improving integration with downstream systems and reducing parsing errors.