@omarsar0: Learn to build evals, folks. Highly recommended reading on using decision models for evals. From our internal evals, Sy…
Summary
Omar Santos recommends resources on using decision models (like System One model Jev) for LLM-as-a-judge evaluations, building on Hamel Husain's blog post about choosing and tuning decision models for evals.
View Cached Full Text
Cached at: 10/21/26, 04:35 PM
Learn to build evals, folks.
Highly recommended reading on using decision models for evals.
From our internal evals, System One models like Jev are a great fit for LLM Judges.
If you want to go beyond Hamel’s great write-up, I’ve also put together a few resources on applying decision models to LLM judges.
(bookmark it)
-
Jev-as-a-Judge for Agent Evaluations - https://academy.dair.ai/dashboard/resources/jev-as-a-judge…
-
Jev-as-a-Judge: Evaluating Agents with a Decision Model - https://academy.dair.ai/labs/jev-as-a-judge-for-agent-evals…
Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - https://arxiv.org/abs/2609.26550
Hamel Husain (@HamelHusain): New blog post 🎉: How To Use Decision Models For Evals
In this post, we show you how to choose a decision model for your task, tune its threshold (don’t assume the 0.5 default is good!), and interpret its scores.
Lots of cool interactive charts here.
Similar Articles
@omarsar0: https://x.com/omarsar0/status/2107472222398886011
Omar Santos presents a guide on using Jev, TypeSafe AI's decision model, as an LLM judge for agent evaluations, showing how reading full trajectories (tool calls, results, errors) rather than just final replies increases reliability and consistency for production-grade evaluation and monitoring systems.
@omarsar0: If you use LLM-as-judge, this one is worth reading. (bookmark it) It's actually one of the most effective ways to use L…
BinEval is a new framework that decomposes LLM evaluation criteria into atomic binary questions, improving interpretability and enabling targeted prompt optimization, achieving strong results on factual consistency benchmarks.
@ArizePhoenix: Who judges the evaluators? When you use LLM-as-a-judge, you’re trusting a model to decide whether your agent, workflow,…
The article discusses the challenges of debugging and evaluating LLM judges using Arize Phoenix, which traces evaluator runs via OpenTelemetry to inspect decision logic, costs, and potential biases.
Decision models like Jev don't beat LLM-as-a-judge or traditional classifiers
A Red Hat Developer article benchmarks TypeSafe AI's Jev "System One" decision models against LLM-as-a-judge guardrails and traditional zero-shot text classifiers, concluding that decision models don't clearly outperform the established alternatives despite their schema-safe, low-latency, zero-shot benefits.
@sermakarevich: An article for everyone who ships, buys, or signs off on software built on large language models (LLMs): engineers, pro…
A plain-language, source-backed guide to LLM evaluations: how to build repeatable eval loops with inputs, gold answers, and graders, why unchecked judge models are the most common way AI teams fool themselves, and how to get honest numbers with error bars for code-based and model-based grading of retrieval and agent systems.