I built an open-source eval + monitoring tool
Summary
LitmusAI is an open-source tool for testing AI agents pre-deployment and monitoring them in production, with YAML/Python test cases covering correctness, consistency, latency, and token cost, plus live traffic monitoring for prompt injection and policy violations with webhook alerts.
Similar Articles
How to go about evaluation and Observability while building AI agents?
The author discusses challenges in evaluating and monitoring AI agents in production, including offline vs online evals, LLM-as-a-judge, tracing, and cost tracking, while citing tools like Langfuse and LangSmith but focusing on underlying processes.
telemetry.dev
telemetry.dev is an LLM observability tool for monitoring and debugging AI agents, launched on Product Hunt.
An open-source tool for testing AI agent behavior before they go into production.
The author plans to open-source a tool for testing AI agent behavior before production, supporting OpenAI Agents SDK, PydanticAI, and custom Python agents, and is seeking developer contributions and feedback.
AI Observability by OpenObserve
OpenObserve launches an AI-native, open-source observability platform designed to trace AI agents and LLMs, offering detailed insights into performance, cost, and quality for developers.
I built AgentLighthouse, a local “Lighthouse for AI agents” that scans repos/docs/APIs for agent readiness
AgentLighthouse is a local-first tool that scans repositories, docs, and APIs to assess how well AI coding agents (like Codex, Claude Code, Cursor) can understand and use a project. It checks for agent instruction files, documentation quality, setup clarity, OpenAPI operation quality, MCP tool descriptions, and more.