After going through ~15 agentic-loop papers (the wins and the failures), the thing that predicts success is the verifier, not the model

I've built a few agent loops and do consulting with various teams in Berlin and got curious why some teams have agents that work great and others fall apart. So I went through a pile of papers, the success stories and the failure ones, and the same thing kept showing up. The loops that win all have a real, hard-to-game check on the work, and they're willing to spend compute to hit it: - ComPilot wraps an off-the-shelf LLM around a compiler. The compiler reports legality and measured speedup, the model retries. 2.66x speedup on a single run, 3.54x best-of-5, no fine-tuning. - AlphaCodium runs generated code against tests in a loop. GPT-4 went from 19% to 44% on CodeContests just from adding that loop. - DeepSeek-R1 trains on verifiable math/code rewards. The R1-Zero variant climbed from 15.6% to 71.0% on AIME over training, and to 86.7% with majority voting. - o3 hit 87.5% on ARC-AGI... at the high-compute setting, which cost on the order of hundreds of thousands of dollars for the run. The score is real and so is the bill. The failures almost always lack a verifier, or have one the model can game: - The "AI Scientist" agent, given control of its own runtime, tried to edit its own timeout instead of making the code faster. - Without external feedback, models asked to self-correct their reasoning often get worse, not better. - On open-ended tasks the gap is still brutal: GAIA, humans 92% vs an agent 15%; WebArena, 14% vs 78%. And single-run scores lie, reliability across repeated trials drops fast. Two things I took away for building: The verifier is the actual product. If you can't check the output cheaply and in a way the model can't talk its way around, you don't have a loop, you have a vibe with extra steps. Tests, a compiler, a metric on a held-out set, a second model grounded in evidence, whatever. Find the best one you have. The wins are bought with compute, so the metric that matters is cost per successful outcome, not per run. And sandbox the thing, because a loop with access to its own constraints will edit them. Curious what everyone's using as their verifier. What's actually worked for you, and what's gotten gamed? Where do you put a human in the loop?