Tag
A developer discusses how Anthropic's backend rewrite to Rust involved building robust evals and parallel-running systems to verify correctness, arguing that constraining non-deterministic AI tools is the key challenge in AI-assisted coding right now.
作者与 Antaripa Saha 合作发布了一篇关于 AI 代理调试的深度教程，讲解如何阅读代理 trace、定位失败原因，并将失败案例转化为 evals 以提升生产环境可靠性。
作者指出 AI 功能在 demo 阶段表现优异却在生产环境中失灵的常见原因：示例是由人工挑选的理想输入。建议在写第一个 prompt 前先收集 50-100 个真实（尤其是糟糕的）输入并定义理想答案，用作回归评测集。
Anthropic's Applied AI team发布指南，指出许多企业多智能体设计只是照搬组织架构图（康威定律），并建议先建立端到端评测、再逐个折叠子智能体，仅在折叠后性能下降时才保留子智能体。
LangChain CEO Harrison Chase's Interrupt London session breaks agents into model, harness, and context, showing how each improves via LangSmith Fine-Tuning, LangSmith Engine, and memory tools—closing with a North Star vision of an agent that catches its own mistakes, writes its own evals, and proposes its own fixes.
该论文研究前沿语言模型的概率判断是否满足全概率律（含因果 do-演算的后门调整），发现 GPT-6.1 Sol 和 Claude Opus 5.5 最一致，而推理长度与一致性无关，且一致性残差可在无真值情况下标记不可靠回答。
Arize Phoenix evals (TS) now supports running any classification evaluator on a decision model (e.g. gpt-6-luna via OpenAI's Decisions API), returning labels and probabilities per option with each call traced as a decision span.
A practitioner shares hard-won lessons from moving agentic workflows from demos to production, highlighting quiet tool-call failures, goal drift over long chains, missing evals, retrieval issues over model quality, and the value of human approval over autonomy.
An experienced backend/distributed-systems engineer argues that AI agent failures in production usually stem from plumbing issues—non-idempotent tool calls, missing approval steps, uncapped costs, and poor state persistence—rather than model limitations, and shares practical hardening tips.
作者认为AI驱动的软件开发中，评测系统（evals）正在取代传统自动化测试，成为提升开发效率和系统质量的关键，并提到Claude Code团队和Google的harness2等大厂都在加大投入。
一篇关于企业AI落地策略的观点帖，主张与其花重金让AI专家学习业务，不如让业务运营人员快速掌握AI执行能力，并分享了从工作流操作者入手实施AI转型的实践经验。
Devin can now automate QA evals by reading real production conversations, rebuilding agent failures in a full copy of the environment, and turning them into eval tests as PRs for human review.
Rox 介绍了「Living Evals」：由 Devin 每周基于真实生产流量自动复现失败案例并生成评测集，团队只需审查 PR，评测套件已覆盖 72% 的真实使用场景。
Nancy from Arize and Phoenix engineer Elizabeth demonstrate in the first AI Builder webinar how to use the open-source Harbor framework to run offline benchmarks on AI agents and tools in a sandbox, and use the Arize Phoenix plugin to log, version, and compare scores, errors, and traces.
Secludy 发布了「Who Sells Data」免费目录，收录 159 家向 AI 开发商出售训练数据、强化学习环境和评测的公司，每条信息均附来源与核查日期，且未知信息会明确标注而非猜测。
作者开源了 agent-engineer 插件，适用于 Claude Code、Codex CLI、Cursor 等，包含 11 个技能、4 个子代理和 5 个钩子；盲测 A/B 显示 Claude Code (Opus 5.5) 使用插件后任务需求满足率从 34% 提升到 53%。
DigitalOcean 推出 Signals（Agent Learning Services 与 Agent Data Services），可从生产环境的 agent 轨迹中自动标记失败会话并一键转化为评测集，在真实运行环境中回放验证修复效果。
讨论随着AI代理的普及，产品信息与质量评价将演变为新的SEO/AEO博弈场，公司会针对代理偏好的评测、基准与人类偏好进行优化，而真正客观的评测难以实现。
LlamaIndex CEO Jerry Liu argues that AI development is shifting toward defining evals and hill-climbing over them, rather than hand-coding deterministic or agentic workflows, and that agent interfaces will evolve to capture goal-and-eval-driven instructions.
作者在 AI Engineer World's Fair 上的 evals 演讲总结，记录了如何将一个 LLM 评审器从完全不可用提升到与人类判断 30 次中有 27 次一致：包括改用点式评分避免偏好偏差、校准上下文、更换模型以及应对谄媚倾向等步骤。