We keep adding “skills” to our agents and have no idea which ones actually work. Solved problem?
Summary
A PM at an internal developer platform highlights the challenge of tracking which AI agent skills are actually invoked and effective, and asks the community if there are existing tools or solutions for this observability problem.
Similar Articles
How to evaluate a skill for building better agent tools
An article discussing methods for evaluating skills to build better agent tools, likely focusing on practical approaches for AI agent development.
How does your company measure the impact of agents and skills in real production, not just benchmarks?
A discussion on how companies should measure the real-world impact of AI agents and skills in production environments, rather than relying solely on benchmark results.
Unpopular opinion: most production AI agents are flying blind and their developers don't know it
A developer argues that most production AI agents lack essential observability like session traces and cost tracking, comparing it to deploying a web app without monitoring. The article questions whether agent observability is an unsolved problem.
What are you using for observability?
A developer discusses the lack of suitable observability tools for AI agents, expressing disappointment with existing solutions like Opik and hoping for a service that supports OpenTelemetry for analyzing agent sessions and failure modes.
How are you keeping track of what your AI agents are actually doing in production?
The article highlights the challenges teams face in monitoring AI agents in production, particularly regarding compliance and health, and invites discussion on current practices and gaps.