Tag
推文展示了一个有趣的 AI 评测：名为 Jev 的模型仅凭目的地地址和当前所在街道名称，在每个转弯处选择方向，就能仅靠路牌穿越美国，从纽约一路导航到洛杉矶完成 Cannonball Run。
A practitioner outlines 15 must-build projects for AI Evals Engineers, covering trajectory grading, shadow routing, calibrated LLM-as-a-Judge, CI/CD regression gates, adversarial RAG testing, DPO fine-tuning flywheels, statistical significance, agent red-teaming, drift monitoring, cost-quality dashboards, counterfactual replay debugging, synthetic edge-case generation, context-window eviction tests, dataset contamination checks, and public eval methodology teardowns.
作者分享了一套基于真实会话失败用例构建 AI agent 回归测试集的流程：先手动标注 traces，再按失败类型分组、写廉价检查项，并用 Kitaru 的 MCP server 与重放机制加速评测集的维护。
Course notes for Chapter 3 of AI Engineering, covering evaluation methodologies (Evals) in depth: starting from the challenges posed by open-ended outputs and black-box models, it introduces language modeling metrics such as cross-entropy, perplexity, BPC/BPB, and compares two mainstream evaluation approaches — AI-as-a-judge and comparative evaluation.
The tweet highlights the essential role of evals in automating AI for enterprises, emphasizing that measurement is key to understanding and improving non-deterministic processes like agent performance.
This article discusses the importance of evaluations in AI systems, explains why traditional testing is insufficient, introduces three main types of evaluations, and provides implementation suggestions.
The article outlines a 12-stage learning path to become an AI Evals & Reliability Engineer, covering foundational skills to advanced production techniques.
A team describes how their prompt-injection eval suite caught a regression in a document assistant before shipping, emphasizing the importance of maintaining a strict hierarchy between system instructions and retrieved data.
oqoqo is a tool for building evals and custom benchmarks for real-world tasks.
Hamel Husain shares flashcards and insights from an AI evaluation course, advocating for binary judges over Likert scales for practical LLM evaluation.
The author predicts that evals/analytics startups will transition into continual learning platforms in 2026, with some failing and the tasteful ones succeeding.
Abliteration launches a made-to-order synthetic training data workflow that generates negative, rare, and adversarial examples for classifiers, with schema, real-world facts, labels, provenance, and export to platforms like Hugging Face.
Arize AI is hosting the Observe 2026 conference in San Francisco focused on AI agents and evaluations with speakers from OpenAI, Cursor, and Uber. The event features talks on multi-agent systems and frontier agentic AI.
A curated list of 11 links shared daily to help people learn AI evaluation techniques, covering evals, observability, LLM-as-judge, and agent evaluation.