My mutation test run killed 252 of 252 mutants. Every kill was fake.

I use Claude Code to build small Python tools that run parts of my home setup, and for anything that matters I have it mutation test the test suite: change the code on purpose, one small break at a time, and check that some test fails for each one. A mutant that no test notices is a hole in the tests. The run that looked perfect: three hours, 252 mutants, 252 killed. Then the check I run after every long run failed. That check pushes the unchanged code through the same runner, and it should pass. It didnt. What happened: the runner copies the code and the tests into a fresh temp folder for each mutant. Three days earlier a test had started reading a small data file that sits next to the code. Nobody added that file to the runner's copy list. So in every copy, mutated or not, that one test failed, and every mutant "died" of the same missing file. The runner's own baseline didn't catch it, because it ran on the source folder, where the file exists. It was worse than one wasted morning. The same bug had already voided an earlier "11 of 11 killed" for a change I'd shipped, and that number was sitting in my change log as evidence. With the runner fixed, the 11 held up, and a full rerun of the 252 killed them all for real. But for three days that number proved nothing. It wasn't the first time either. An earlier run crossed midnight with its test venv under /tmp. The venv lost its config file overnight (I think macOS's tmp cleanup got it), pytest couldn't start anymore, and a suite that can't start fails. So everything after midnight read as killed. What I do now: Every run has a control: a comment-only change that must survive. If the control dies, the run is void. The control goes through the runner's own copy step, before and after the run. A baseline on the source folder hides a file the copy is missing. "Killed" isn't evidence on its own. The log names the failing tests for each kill. If one test name shows up in every kill, thats not a strong suite, it's one broken test. When a test starts reading a new file, the file goes on the copy list in the same change. Long runs go through a wrapper that fingerprints the source folders before and after (names, sizes, hashes) and marks the run VOID if anything changed. Claude once added a test in the middle of a run, so the later mutants ran against different tests than the first ones. If you run evals or an LLM judge, same idea: a case that has to pass and a case that has to fail, every run, through the exact same harness. 100% should make you suspicious, not happy.