@LanLance24: Recommends this article by a Langfuse team member, which discusses best practices for designing Agent/LLM evaluation metrics when we have Traces. A good evaluation metric system should be concise, based on real online failure cases, decision-guiding, and continuously updated. > Metrics are divided into three categories: 1…

I highly recommend this article from the Langfuse team, which discusses best practices for designing evaluation metrics for agents/LLMs when working with traces.

A good evaluation metric system should be:

• Minimal and focused
• Based on real failures from production
• Actionable in guiding decisions
• Iterative and continuously updated

Metrics can be categorized into three types:

1. Target Metrics: The core quality aspects you truly aim to improve (e.g., whether the response actually solves the problem, or if the correct tools were used).
2. Guardrail Metrics: Non-negotiable red lines (e.g., safety, compliance, format requirements), where any failure is severe.
3. Operational Metrics: System performance indicators like cost, latency, and request volume.

Best Practice: Begin by conducting error analysis (carefully reviewing a set of real traces to identify actual failure points), then align with product goals to decide which metrics to monitor. When selecting metrics, ensure they directly guide action (e.g., if a metric degrades, would you roll back code, revise prompts, or investigate further?).

Metrics are flexible and should not remain static. After system updates, they need re-evaluation. Metrics with consistently high scores over time can be removed, while also avoiding over-optimization for any single metric (Goodhart’s Law).

Lotte (@lotte_verheyden): Before you write good evals, you have to pick the right things to evaluate. Main take-aways:

• source metrics from failures you see in traces
• keep the set small and maintainable
• only keep a metric if you’d take action when it moves