Tag
The article discusses techniques to identify prompt degradation upon model updates, such as using pinned evaluation cases and model testing matrices, and inquires about best practices for evaluating AI agents in production.
The tweet argues that AI language tutors should shift from chatbots to agents that focus on practical language usage through testing, akin to continuous integration in software development.
The article discusses hidden states that an AI agent should track when diagnosing CI failures, such as flaky tests, real bugs, and configuration errors, and seeks feedback on weaknesses and missing states.
Uber open-sourced SubmitQueue, a high-performance speculative merge queue that validates multiple changes in parallel against predicted future states of HEAD to keep trunk green at scale.
This article argues that traditional CI fails for AI agents and proposes replacing it with a merge queue where all tests run before merging, allowing agents to fix issues before breaking the build.
PyTorch introduces a Cross-Repository CI Relay (CRCR) that automatically triggers and tracks CI in out-of-tree backends, providing a unified dashboard for CI health.
The author shares a TDD skill for AI agents to improve test writing, based on Kent Beck's Canon TDD, and provides a GitHub link.
The article discusses the problem of flaky tests in software development and proposes a simple mechanical habit: when using a merge queue, continue to run the full test suite on main and maintain a visible list of recent main failures to help identify and eradicate flaky tests.
Tophat by Shopify is a tool that lets you test mobile CI builds on any device without building locally.
GoogleTest, Google's C++ test framework, has released version 1.18.0 which requires at least C++17, along with documentation updates and ongoing improvements.
Jenkins is a leading open-source automation server built with Java, used for CI/CD, testing, deployment, and automating development workflows.