Using an agent to understand agent traces
Summary
The article describes using Databricks Genie to query MLflow agent traces with natural language, enabling non-SQL users to analyze agent performance metrics like latency and errors.
Similar Articles
Building data agents
Discusses the evolution from text-to-SQL to autonomous data agents, comparing custom-built agents using LangGraph with managed platforms like Snowflake Cortex Analyst, Databricks Genie, and PowerBI Copilot.
Anyone actually doing pattern analysis across their agent's traces, or are we all just eyeballing dashboards?
The author questions why engineers are not performing automated pattern analysis on agent traces, arguing that current observability tools like LangSmith and Langfuse lack the 'connection' step needed to compound knowledge from agent behavior, unlike personal knowledge systems.
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
This paper introduces the Insights Generator, a multi-agent system for systematic corpus-level trace diagnostics of LLM agents, which generates evidence-backed insights by proposing and testing hypotheses across execution traces. Experiments show that using Insights Generator reports improves scaffold performance by 30.4 percentage points.
AI Agent Intelligence tool - Incident debugging, Cost spike detection
Building a tool for AI Agent incident debugging and cost spike detection without additional instrumentation, covering issues like prompt injection, reasoning loops, and data exfiltration. Asking if customers in production environments see this as a pain point worth paying for.
Signals: finding the most informative agent traces without LLM judges [R]
Katanemo Labs introduces 'Signals,' a lightweight method for identifying informative agent traces without using LLM judges or GPUs, achieving higher efficiency in trajectory analysis.