traces

This article explains how to use the LangSmith CLI to manage traces and build evaluation datasets through a quick tutorial.

The article explains a process for converting AI agent traces into a labeled training dataset, detailing steps like automatic capture, strategic sampling, and explicit labeling against criteria.

The author tested the Qwen3.8 27B model on a fluid simulation task, achieving success after iterative prompting, and shared the implementation with traces and a GitHub repository.

This article recommends best practices from the Langfuse team for designing Agent/LLM evaluation metrics using Traces, including goal metrics, guardrail metrics, and operational metrics, and emphasizes starting with error analysis to keep metrics concise, precise, and actionable.

Traceway is an open-source, self-hosted observability platform that unifies logs, traces, metrics, and exceptions using OpenTelemetry, with features like session replay and AI observability.

Advocates for treating agent failures as evaluation benchmarks rather than just trace logs, emphasizing the need for systematic testing of AI agent behaviors.

Own the agent learning loop and rent the model; private evals, traces, and memory compound over time.

At the aiDotEngineer World's Fair, Vtrivedy10 discussed how data mining from traces is a high-leverage practice for understanding AI agents, curating data at scale, and running improvement loops.

This tweet shares best practices for agent observability, covering metrics, logs, and traces to debug and optimize production AI agents.

A developer builds a debugging tool for AI agents that compares replays against reference runs to identify where behavior first drifted, expressing frustration with manual trace debugging.

The tweet recommends an article on agent architecture in production, highlighting the use of Traces to diagnose issues and implement an iterative improvement loop.

Yohei Nakajima highlights that building an agent on activegraph automatically produces first-class traces, unlike bolted-on solutions, demonstrated with a coding agent experiment.

A tweet by Martin Casado highlighting a solution to the difficult problem of exposing traces at scale to AI agents, balancing cost and AI leverage.

Adam Łucek discusses how LangChain uses trace data to build evaluations for production agents.

Respan introduces an AI observability platform that automatically catches issues in traces, aiming to replace manual debugging for agent-based workflows.

Engine is a new tool that connects agent observability traces to automated fixes and evaluations, closing the agent improvement loop for engineering teams.

LangSmith Engine is an agent that sits on top of traces, automatically identifies issues, and suggests action items like code changes or evaluators to add.