Using an agent to understand agent traces

Reddit r/AI_Agents Tools

Summary

The article describes using Databricks Genie to query MLflow agent traces with natural language, enabling non-SQL users to analyze agent performance metrics like latency and errors.

Been running an agent in production for a few months and logging everything to an MLflow experiment (spans, tool calls, latencies, token counts, errors). The traces are great but querying them meant writing pandas/SQL by hand every time I wanted to answer a question like "which tool calls are timing out most" or "what's my p95 latency on multi-step runs." So I pointed Databricks Genie at the trace tables. Genie turns natural-language questions into SQL over your data, so now I just ask things in plain English: - "Show me the 10 slowest traces this week and which tool dominated the latency" - "What % of runs hit an error, broken down by tool?" - "Average tokens per trace, trending by day" It generates the SQL, runs it against the MLflow trace data which lives in UC, and hands back a table or chart. Setup was basically: Traces already landing in an MLflow experiment (autolog handles most of this) Flatten the trace/span data into queryable tables Create a Genie space over those tables with a bit of context (what a "span" is, what the tool names mean). For a quick start you can also use genie code directly. Biggest win is that non-SQL folks on the team can now interrogate agent behavior themselves instead of pinging me.
Original Article

Similar Articles

Building data agents

Reddit r/AI_Agents

Discusses the evolution from text-to-SQL to autonomous data agents, comparing custom-built agents using LangGraph with managed platforms like Snowflake Cortex Analyst, Databricks Genie, and PowerBI Copilot.

Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents

arXiv cs.AI

This paper introduces the Insights Generator, a multi-agent system for systematic corpus-level trace diagnostics of LLM agents, which generates evidence-backed insights by proposing and testing hypotheses across execution traces. Experiments show that using Insights Generator reports improves scaffold performance by 30.4 percentage points.

AI Agent Intelligence tool - Incident debugging, Cost spike detection

Reddit r/AI_Agents

Building a tool for AI Agent incident debugging and cost spike detection without additional instrumentation, covering issues like prompt injection, reasoning loops, and data exfiltration. Asking if customers in production environments see this as a pain point worth paying for.