Tag
Omar Santos recommends resources on using decision models (like System One model Jev) for LLM-as-a-judge evaluations, building on Hamel Husain's blog post about choosing and tuning decision models for evals.
作者对比了 Jev 与三种 LLM 评审（Sonnet、Haiku、Gemini Flash Lite）在智能体评测中的表现：Jev 成本低 180 倍（每 1k 条裁决 $0.07）但准确率低 2.6 个百分点（90.6% vs 93.2%），建议采用 Jev 先筛 + Sonnet 处理约 16% 不确定样本的级联方案，以 1/6 成本达到 Sonnet 级准确率。
A plain-language, source-backed guide to LLM evaluations: how to build repeatable eval loops with inputs, gold answers, and graders, why unchecked judge models are the most common way AI teams fool themselves, and how to get honest numbers with error bars for code-based and model-based grading of retrieval and agent systems.
EvalSeal is an open-source tool that enhances LLM evaluation reproducibility through multiple runs, flip rate measurement, and tamper-evident sealing, highlighting instability in LLM judges.
Dan Luu critiques claims that dynamic languages are more token-efficient for LLMs, pointing out flaws in existing evals and emphasizing the need for better benchmarking methods.
smevals is an open-source framework for running evals against small and large models, with a structured vocabulary for tasks, configs, runs, and graders, plus a CLI for execution and grading.
A detailed guide on using Pydantic AI and Pydantic Evals to ensure structured outputs from LLMs, covering shape validation, content correctness, and open-ended judgement.
Discusses a blind spot in AI agent reliability: cached facts that were true when ingested but become stale by the time they are used, leading to coherent but incorrect actions. Proposes separating consistency (match with source) from currency (source still true now), and asks how the community handles this.
This article explores why independent AI evaluation startups rarely succeed, citing talent drain to more lucrative parts of the stack, a narrow customer base, and optimization pressures that degrade eval utility.
A critique of large Rails codebases in the AI agent era, proposing a shift to skill-based development with agents, markdown skills, and TypeScript for deterministic I/O.
Spotify Engineering discusses using LLM evals as a funnel before A/B experiments, improving hit rates and creating a feedback loop between evals and experiments.