A developer discusses the lack of suitable observability tools for AI agents, expressing disappointment with existing solutions like Opik and hoping for a service that supports OpenTelemetry for analyzing agent sessions and failure modes.
Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something? So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems. I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope. For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that. Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case. Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes. What are y'all doing for this? Are y'all doing this?
A developer building multi-agent financial workflows seeks community advice on observability and reliability tooling for AI agents in production, sharing frustration with fragmented landscape and cascading failures.
A developer argues that most production AI agents lack essential observability like session traces and cost tracking, comparing it to deploying a web app without monitoring. The article questions whether agent observability is an unsolved problem.
Opik is an open-source platform for AI agent observability that goes beyond tracing to automatically diagnose failures, propose fixes, and verify them, closing the debugging loop without manual intervention.
The article critiques the lack of observability in the memory layer of AI agents, which determines what the agent believes, and questions why this layer remains a black box despite advances in other system observability.