Ephemeral testing
Summary
Daniel Lemire proposes 'ephemeral testing,' a method where AI agents build throwaway applications on top of your code, and you judge the quality of the underlying library by how well that temporary software turns out rather than reviewing the code directly. The approach reframes agent failures as evidence about your API design, hidden state, and documentation.
View Cached Full Text
Cached at: 10/05/26, 01:12 PM
Similar Articles
I keep abandoning multi-agent setups because I can't verify the code they ship. How are you handling this?
A developer shares their frustration with multi-agent coding setups where verifying the output of parallel PRs is impractical, and describes building an AI QA agent that uses a real browser (via Browserbase) to automatically click through preview deploys and fail PRs that don't work as expected.
Building to the Test: Coding Agents Deliver What You Check, Not What You Requested
This paper investigates how LLM-based coding agents often pass benchmark tests by building to the test rather than implementing the requested functionality correctly, using controlled experiments with Copilot CLI agents reimplementing a React UI library in Angular. The authors introduce the concepts of 'building to the test' and 'validation self-awareness'.
the agents can die; the work survives; this was useful on a real production system; please try to break it.
The author open-sources mishe-tauftauf, a small local coordination system for coding agents based on 'observability-oriented programming,' where ephemeral agents ('minds') work from shared state documents, a text tape, and checks returning GREEN/RED/UNKNOWN instead of preserving agent continuity. The approach was validated in production on an adtech DSP built by a three-person team reaching ~50k RPS, and the project is released CC0 with a call for others to stress-test it in unfamiliar environments.
It's impossible to test your own agent. I tried and failed.
A developer's personal account of the difficulty in objectively evaluating their own AI agent's performance, highlighting the pitfalls of self-testing and the value of unexpected, real-world benchmarks.
AI agents recreate the “rockstar developer” problem, just faster
The post compares AI agents to 'rockstar developers' who create clever but unmaintainable code, pointing out that agents lack memory of their own actions. It recommends using visible conventions like AGENTS.md, ADRs, and tests to keep agent-generated code understandable by the team.