evals

The article describes a project with experiments on training a local LLM using continued pretraining (CPT) and RAG for domain-specific knowledge, featuring comprehensive evaluations and findings.

The article presents several disruptive use cases for Jev, a tool that can be applied to AI evals, voice AI, workflows, synthetic personas, and more, providing fast and quantifiable solutions for various AI tasks.

The author revised the success criterion for coding-agent evaluations by moving from requiring three perfect 88/88 runs to a criterion where each case must pass at least 2 of 3 runs, reducing randomness impact and improving stability.

Mike Taylor has announced his new position as Head of Evals at @every, a role highlighted as game-changing for AI evaluations by @danshipper.

A free, framework-free set of Colab notebooks for learning the applied LLM stack, including RAG, agents, and evals, targeting AI engineers.

Two talks and a blog post argue that the feedback loop and harness engineering are more important than model weights for owning AI intelligence in production, highlighting context management and cost considerations.

The article argues that the main constraint on AI agent adoption is not capability but verification, including how companies define quality, evaluate ongoing performance, and compound feedback. It explores challenges like tacit standards, company-specific evals, feedback ownership, and self-improving loops.

Jerry Liu promotes LlamaParse and LlamaAgents for large-scale document extraction, emphasizing LLM evals and hillclimbing for accuracy and cost. He also connects FDE work with evals and RL environments.

Jerry Liu shares thoughts on how forward-deployed engineer (FDE) work will shift toward defining goals, environments, and evals while automated optimization processes handle tactical implementation.

Armature is a new product analytics tool for AI agent sessions, capturing user intent, agent thinking, and success scores via an SDK for MCP servers, with automatic PII redaction.

Omar recommends reading Chamath Palihapitiya's AI investing guide, emphasizing the importance of harness engineering and evals for AI builders.

Agent Behavior is a format for writing behavior specs for AI agents in Markdown, enabling teams to define, review, and evaluate expected agent conduct across interactions.

OpenAI reminds developers that eval results depend on API settings and harness design, recommending the Responses API, retaining reasoning, and using compaction for best performance.

A user comments that you could hook Codex up to the Pangram API to rewrite text until it passes a human evaluation.

A developer shares hard-won lessons from building AI agents: focusing on tool design over model choice, using small loops instead of giant prompts, logging agent context, adding guardrails early, and creating small evals to catch bugs.

At the AI Engineer World's Fair, Phil Schmid gave a talk on why vibe-checking agent skills break in production and how to build reliable automated evals using negative test cases, skill limits, and ablation tests.

Advocates for treating agent failures as evaluation benchmarks rather than just trace logs, emphasizing the need for systematic testing of AI agent behaviors.

oqoqo is a tool for building evals and custom benchmarks for real-world tasks.

The article compares popular developer tools for agent reliability across four layers: tracing/evals, runtime guardrails, and gateway. It finds that no single open-source tool covers all layers, and most developers use a combination.

This article argues that alignment evaluations (evals) need to be properly calibrated to be meaningful, discussing common pitfalls and techniques for improving calibration.