Tag
Joseph Decker evaluates 16 AI models on truthfulness for his product Condensr, discovers that the leaderboard winner fabricated content five times in an audit, and instead ships the second-place model which had zero fabrications. The post details the evaluation process, a bug in the LLM judge that penalized accurate summaries due to truncated transcripts, and the importance of custom evals over generic benchmarks.