Tag
This paper analyzes a one-year production trace from Chutes to study LLM serving workloads, revealing temporal evolution and user-model interactions to improve serving system benchmarking.
A Microsoft paper analyzing 13.5M GitHub Copilot sessions shows that coding-agent workloads are dominated by autonomous LLM call chains, with KV-cache and container idle time strongly dependent on turn/session structure, arguing for workflow-level scheduling instead of request-level policies.
CreditGenie uses LangSmith to debug thousands of agent traces and generate targeted test questions from production data.
A blog post from Parlance Labs tests automated AI evaluation tools (Braintrust Loop, Arize Alyx, LangSmith Engine) on real production data, finding they catch 87% of issues humans flag but miss domain-specific failures and add noise, recommending iterative human-in-the-loop use.
Replit built ViBench to evaluate app-building success from natural-language specs and Telescope to cluster production failure traces, enabling harness-level and context-level continual learning for agents using closed frontier models.
Advocates using production traces as data for AI post-training, highlighting the growing scale of data spending.
Head of AI at Benchling discusses patterns for analyzing production traces in a tech talk.
LangChain launches LangSmith Engine in public beta, an autonomous agent that monitors production traces, clusters failures, diagnoses root causes, and proposes fixes and eval coverage to streamline agent development.
TRACER is an open-source system that trains lightweight ML surrogates on production traces from LLM classification endpoints, routing requests through a parity gate that activates surrogates only when agreement with the original model exceeds a specified threshold. This approach achieves 83-100% surrogate coverage on intent classification benchmarks while maintaining interpretability into handling boundaries and failure modes.