Tag
Replit built ViBench to evaluate app-building success from natural-language specs and Telescope to cluster production failure traces, enabling harness-level and context-level continual learning for agents using closed frontier models.
Advocates using production traces as data for AI post-training, highlighting the growing scale of data spending.
Head of AI at Benchling discusses patterns for analyzing production traces in a tech talk.
LangChain launches LangSmith Engine in public beta, an autonomous agent that monitors production traces, clusters failures, diagnoses root causes, and proposes fixes and eval coverage to streamline agent development.
TRACER is an open-source system that trains lightweight ML surrogates on production traces from LLM classification endpoints, routing requests through a parity gate that activates surrogates only when agreement with the original model exceeds a specified threshold. This approach achieves 83-100% surrogate coverage on intent classification benchmarks while maintaining interpretability into handling boundaries and failure modes.