Evals in the new world of software development

Reddit r/AI_Agents News

Summary

作者认为AI驱动的软件开发中,评测系统(evals)正在取代传统自动化测试,成为提升开发效率和系统质量的关键,并提到Claude Code团队和Google的harness2等大厂都在加大投入。

Its becoming increasingly clear to me that the world will see the next step improvements in harness design and efficiency through evals. Its critical that you constantly do it on what you build and create a recursion loop to pick further improvements. The more this is automated the faster you are going to move. For ex. recently Claude code team published info around how they are trying to make their review process faster by running an eval for every real world run were something might have gone wrong and then suggest what needs to change to make the whole system better. It's like how in the past automated test were important for shipping a quality and stable product for users with the right speed. In this new world its replaced by these evaluation systems. Here is where I am going to be spending most amount of time this quarter. Even more promising since I see almost all big players investing here - like the google's harness2.
Original Article

Similar Articles

@yibie: https://x.com/yibie/status/2102619356874117594

X AI KOLs Timeline

This article discusses the importance of evaluations in AI systems, explains why traditional testing is insufficient, introduces three main types of evaluations, and provides implementation suggestions.

@Ryrenz: Folks, you no longer have to fumble around blindly when evaluating AI apps. evals-skills lets a coding agent guide you through building an evaluation suite tailored to your product. evals-skills has 1,500+ stars on GitHub, and the project says this approach comes from helping over 50 companies with evaluations, …

X AI KOLs Timeline

evals-skills enables coding agents to help developers build AI application evaluation workflows tailored to their products, identifying failure modes before designing scoring criteria. It has earned 1,500+ stars on GitHub. The approach draws on experience conducting evaluations for more than 50 companies and training thousands of learners, and includes skill modules such as error-discovery and validate-evaluator.

@thinkszyg: The AI Coding Speed Paradox: Coding 48% Faster, Review 6x Slower. How to Rebuild the Review Process? SD Times Analyzes Data from 250,000 Developers: AI Boosts Coding Speed by 48-58%, But AI-Generated PRs Get Stuck in Review for 4-6x Longer…

X AI KOLs Timeline

The article points out that AI coding increases coding speed by 48-58%, but code review time increases by 4-6x, and security vulnerabilities also rise. It proposes a three-step plan to rebuild the review process: AI pre-review, focusing on architectural decisions, and using Microsoft's open-source ASSERT framework for behavioral verification.

@sodawhite_dev: https://x.com/sodawhite_dev/status/2067413032544940062

X AI KOLs Timeline

The article analyzes Anthropic's 400,000-session report on Claude Code, pointing out that AI programming tools are changing the division of labor between humans and AI. Domain knowledge is more important than coding ability. Expert users can enable AI to perform more complex tasks, while verification and task decomposition capabilities become core competitive advantages.