@ArizePhoenix: Who judges the evaluators? When you use LLM-as-a-judge, you’re trusting a model to decide whether your agent, workflow,…

Who judges the evaluators? When you use LLM-as-a-judge, you’re trusting a model to decide whether your agent, workflow, or prompt did the right thing. But that raises the obvious question: how do you debug and evaluate the judge? In Arize Phoenix, every evaluator run is automatically traced via OpenTelemetry and sent to a dedicated Phoenix project. That means you can inspect exactly how your evaluator made its decision: → the input data → the exact prompt sent to the judge LLM → the model’s reasoning → the final score → execution timing, token usage, and cost This is especially useful if you have a production agent because your evals need to evolve as well. It becomes increasingly important to check for systematic evaluator bias and to align evaluation with human judgment. In the same way that your agent must improve over time, so must your evals.