evals

The author describes a method for deciding when an agent's repeated task can safely move from a frontier model to a cheaper one, using held-out past runs, human review, and one-click rollback, while seeking critique on the evaluation approach and disclosing they build Operant in this space.

Overmind fine-tunes small open models on a user's own production traces and scores them against frontier models on evals built from the same real tasks, reporting 20-30x fewer hallucinated clauses and 7x better verbatim quoting accuracy in legal contracts.

TasteVal reports that frontier AI models' experimental research taste has doubled roughly every 3 months since December 2025, with Opus 5.5 now surpassing an expert human baseline — a key predictor of how quickly ASI could be reached once coding is automated.

Omar Santos presents a guide on using Jev, TypeSafe AI's decision model, as an LLM judge for agent evaluations, showing how reading full trajectories (tool calls, results, errors) rather than just final replies increases reliability and consistency for production-grade evaluation and monitoring systems.

Aaron Levie argues that agent adoption is bottlenecked by the difficulty of testing and tuning agents on realistic work environments, highlighting Era's launch of a free simulated enterprise across Salesforce, Slack, Jira, Zendesk, Gong, and Deel.

文章主张评测（evals）是企业成为 AI-native 的最大障碍，并给出了 eval-driven development 的指南：定义好的行为、对 agent 进行测试、将失败案例转化为回归测试，并介绍了 grader、case、run、trace 四个核心概念以及确定性检查、参考答案比对等四种打分方式。

G2i.ai CEO Gabbe Greenberg announced the company is hiring 50 staff software engineers this month to work with frontier AI labs on evals and training datasets, paying $200–$300/hr remotely. G2i positions itself as an engineering layer for AI training, building RL environments and curated evals for top labs.

A community member built RangeBench, an MIT-licensed cybersecurity benchmark where models get a shell in an isolated Docker box to capture flags across pwn, web, crypto, rev, forensics, and multi-stage ranges. Local Qwen3.8 27B scored 28.1% first-try and 0% on pwn, while cheap API models like MiMo 2.6 Flash and GPT-6 Luna reached 73.7% and 90.9%, highlighting the real-world risk of increasingly capable hacking models.

LitmusAI is an open-source tool for testing AI agents pre-deployment and monitoring them in production, with YAML/Python test cases covering correctness, consistency, latency, and token cost, plus live traffic monitoring for prompt injection and policy violations with webhook alerts.

The author argues that 100% AI alignment is as absurd as 100% tested software, since verification is always limited to the evals you actually run. Passing tests (or evals) only proves no errors were found, not the absence of untested failure modes.

Center for AI Safety introduces CheatBench, a public benchmark measuring reward gaming in AI agents across math research, coding, and knowledge work by combining hard tasks with opportunities to cheat. It enables cross-model comparisons to study and reduce cheating as agents take on more consequential responsibilities.

The article argues that OpenAI's Agents API does not make agent frameworks obsolete, citing reasons such as the need for model portability, state control, failure handling, and evaluation in production systems.

The article describes a project with experiments on training a local LLM using continued pretraining (CPT) and RAG for domain-specific knowledge, featuring comprehensive evaluations and findings.

The article presents several disruptive use cases for Jev, a tool that can be applied to AI evals, voice AI, workflows, synthetic personas, and more, providing fast and quantifiable solutions for various AI tasks.

The author revised the success criterion for coding-agent evaluations by moving from requiring three perfect 88/88 runs to a criterion where each case must pass at least 2 of 3 runs, reducing randomness impact and improving stability.

Mike Taylor has announced his new position as Head of Evals at @every, a role highlighted as game-changing for AI evaluations by @danshipper.

A free, framework-free set of Colab notebooks for learning the applied LLM stack, including RAG, agents, and evals, targeting AI engineers.

Two talks and a blog post argue that the feedback loop and harness engineering are more important than model weights for owning AI intelligence in production, highlighting context management and cost considerations.

The article argues that the main constraint on AI agent adoption is not capability but verification, including how companies define quality, evaluate ongoing performance, and compound feedback. It explores challenges like tacit standards, company-specific evals, feedback ownership, and self-improving loops.

Jerry Liu promotes LlamaParse and LlamaAgents for large-scale document extraction, emphasizing LLM evals and hillclimbing for accuracy and cost. He also connects FDE work with evals and RL environments.