New Engineering Paradigm in the Age of Agents: Three Articles Point to the Same Answer

Over the past two weeks, I read three articles from independent sources. They discuss different specific problems—one academic paper, one engineering blog, one startup handbook—yet they all point to the same answer. It’s not “write better prompts.” It’s “build better loops.”

I string them together.

Article 1: Regimes – Academic Validation

Yohei Nakajima’s paper An Auditable, Held-Out-Gated Improvement Loop introduces a system called Regimes.

Core problem: AI Agents repeatedly make the same mistakes. These mistakes are classifiable, diagnosable, and routable to specific pipeline locations. But existing “self-improvement” schemes aren’t trustworthy—failures aren’t logged, diagnoses can’t be replayed, and each “promote or discard” decision lands in an external database that can’t be audited.

Regimes’ answer: Build the entire improvement loop on an event-sourced runtime (ActiveGraph). Every step of the agent’s state is a deterministic projection of an append-only log. Four gates—static checks, sandbox execution, in-sample evaluation, and held-out validation—ensure a fix is only promoted when it truly works on unseen data.

Key finding: In-sample +0.18 collapsed to held-out +0.04. The gates keep all the overfitting out.

More important finding: The promoted prompt is not the endpoint. A prompt is a high-bandwidth probe. When independent improvement loops repeatedly discover the same principle, and the prompt’s regressions precisely reveal when that principle should not fire—the prompt points at a candidate deterministic operator. The prompt says “execute when tags match”; the operator says “execute only when detected structure matches.”

Article 2: Warp’s Buzz – Engineering Practice

Zach Lloyd (Warp CEO) wrote a blog about how they built Buzz—an agent for handling social interactions.

The first version was a pile of rules. “If someone mentions a bug, say this. If someone compares competitors, say that.” Brittle and robotic.

They did what the Regimes paper argued in theory: replaced rules with principles. Rules overfit; principles transfer.

Then they did something Regimes couldn’t do in academia: embedded the feedback loop into the team’s daily workflow. The team doesn’t need to become prompt engineers. They just react to Buzz’s drafts in Slack with an emoji indicating what they actually did. Once a day, Buzz collects all feedback, compares suggestions vs. actuals, extracts lessons, updates the skill file—then opens a PR.

Agent skills are managed like code. Repository, version history, review, rollback. The agent can’t directly change production behavior—it opens a PR, humans review. This keeps the useful part of self-improvement without giving up control.

Zach summed it up in one sentence: “The best teams don’t just write better prompts. They build better loops.”

Article 3: No Coding Before 10 AM – Team Restructuring

Michael Bloch’s article describes how another startup completely rewrote their engineering handbook because of Claude Code.

Rule #1: No coding before 10 AM.

For the past twenty years, engineering culture has been “maximize coding time”—cut meetings, lock calendars. This team does the exact opposite. Every morning, engineers pair-prompt: sit together, draft a prompt, define the goal, set up the agent. The agent does the work. The engineer ensures the agent can do the work well.

Other hard rules:

• Design for the agent, not for humans – every system treats the agent as the primary consumer
• Spec results, not process – AI figures out the how; you judge the output
• Review outputs, not code – don’t read every line the agent writes. If it passes tests, ship it
• Build something new, kill the old way immediately – dead code is noise for agents
• Assume everything is obsolete in three months

The Intersection of Three Threads

These three pieces together reveal not three independent findings, but the same paradigm unfolding at different levels:

Four-layer consensus:

1. Prompt is the starting point, not the endpoint

Regimes says the promoted prompt points to an operator. Warp says principles over rules. No Coding says spec results, not process. All three say the same thing: the prompt is a probe. The truly durable improvement isn’t in the prompt text itself, but in the deeper principle or operator that the prompt reveals.

2. Improvement needs gates, not intuition

Regimes uses four gates to ensure improvement isn’t overfitting. Warp’s daily PR is a human gate—every improvement is reviewed. No Coding’s “review outputs, not code” is a result-oriented gate. Self-improvement without gates is a random walk. Self-improvement with gates is engineering.

3. Feedback loops must live in the team’s workflow

Regimes’ loop is algorithm-driven: diagnose-route-repair. Warp’s loop is two Slack interactions: one emoji + one daily PR. No Coding’s loop is one to two hours every morning. The best loops aren’t the most complex. They’re the ones the best teams will actually use.

4. In the age of Agents, the human role changes

It’s not “humans are replaced.” It’s human judgment becomes a scarcer resource. Regimes says humans diagnose failure modes and design action seams. Warp says humans express judgment with emojis and review PRs. No Coding says humans define goals and set up the agent. The shared portrait across all three: humans shift from operators to supervisors, from executors to judges.

One Sentence

Engineering in the age of Agents is no longer “I write prompts, the agent executes.” It is “I design loops, the agent improves inside the loop, and my judgment keeps the loop from going off the rails.”

Three originals:

• Regimes: https://arxiv.org/html/2606.10241v1…
• Warp Buzz: https://warp.dev/blog/agents-need-feedback-loops-not-perfect-prompts…
• No Coding Before 10 AM: https://michaelxbloch.substack.com/p/no-coding-before-10am…

#AIEngineering #Agent #Survey #SoftwareEngineering

An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph

Source: https://arxiv.org/html/2606.10241v1 (June 8, 2026)

Abstract

Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and each promote‑or‑discard decision lands in a side database rather than in the agent’s own history. We show that an event‑sourced agent runtime removes that friction and turns controlled improvement into a first‑class workflow. When the agent’s state is a deterministic projection of an append‑only event log, failures are recorded, a recorded run replays exactly from its log, candidate patches scope to typed pipeline seams, gates are auditable, and every promotion or discard is itself an event. We demonstrate this with Regimes, an improvement loop built on the ActiveGraph runtime. Regimes diagnoses failed evaluations, proposes a repair at a specific point in the agent’s pipeline, and promotes that repair only when it passes static checks, sandbox execution, in‑sample evaluation, and validation on held‑out examples. The loop is target‑agnostic: the same control flow runs against different agent tasks through a common interface. ActiveGraph is what makes this auditable rather than anecdotal: the runtime is event‑sourced, so a recorded run replays deterministically from its log, model and tool responses are cached for exact replay when the cache is available, and the loop’s own history (diagnosis, proposed repair, gate outcomes, promotion or discard) is itself an auditable event log rather than a side database. We demonstrate the loop on LongMemEval‑S, where the dominant failure is not retrieval but reconciliation: the relevant evidence is already present in the assembled context, yet the reader answers incorrectly. Across five seeded held‑out splits, Regimes discovers reader‑prompt repairs that improve final held‑out accuracy by +0.05 to +0.10 in four splits and by +0.01 in one over‑promotion split. Two splits are individually significant under a simple paired test (seed 5’s value is not adjusted for its four‑step sequential promotion structure); the pooled flip count is positive but should be read as descriptive, because the splits come from the same 500‑question pool. The detailed statistics (per‑split McNemar tests, pooled discordances, and the same‑pool caveat) are in the results section rather than here. The case study shows both the promise and the limits of autonomous improvement. The held‑out gate rejects many overfit candidates, including nearly every retrieval‑reweighting repair, while the weak split reveals a concrete next improvement: a stricter promotion threshold and a plateau‑aware stopping rule. More broadly, the discovered prompt repairs point past themselves: their successes reveal which evidence‑use behaviors matter, and their regressions reveal when a behavior should not fire. Durable agent improvements should convert those behaviors into guarded deterministic operators that condition on detected structure rather than on coarse prose rules. The durable contributions are ActiveGraph as an auditable substrate that makes controlled improvement loops tractable, the reproducible held‑out‑gated loop it supports, the failure‑regime taxonomy that routes each failure to a pipeline location (whose marginal value over an unrouted baseline is the primary open question), and the prompt‑as‑discovery‑probe hypothesis.

1 Introduction

Long‑running agents fail in repeated, diagnosable ways. As an agent accumulates history, retrieves over it, assembles a context, and reads that context to answer, each stage can break, and the same kinds of breakage recur. What is missing is not another point fix but a loop: something that can detect a recurring failure, propose a repair, test the repair safely, and keep it only if it works on examples it was not tuned on. This paper presents such a loop, called Regimes, and uses one agent task as a detailed case study. Two plain‑language commitments shape the design. First, a repair is kept only if it works on held‑out examples, not just on the examples used to write it. We call this a held‑out‑gated promotion, and it is what separates a real improvement from a candidate that merely overfits the cases the loop happened to see. Second, the same loop should run against different agent tasks without rewriting its core. We call this target‑agnostic: the diagnosis, gating, and rotation logic talk to any task through a common interface, and only the task‑specific pieces (how to evaluate it, which parts of its pipeline may be edited, how to label its failures) are swapped in. ActiveGraph is part of the core story, not a deployment detail. The loop runs on the ActiveGraph runtime (Nakajima, 2026), an event‑sourced system whose state is a deterministic projection of an append‑only event log, with a cache that records model and tool responses. This is what makes autonomous improvement auditable rather than anecdotal: a run replays exactly from its log without new model calls, the evaluation is controlled and repeatable, and the loop’s own history (each diagnosis, each proposed repair, each gate outcome, each promotion or discard) is itself an auditable event log rather than a side database. A self‑improvement claim is only as trustworthy as the substrate that records it, and an event‑sourced substrate lets a reader audit every step. This is where the substrate, not the algorithm, is the lever. The case study is LongMemEval‑S (Wu et al., 2024), a long‑context memory benchmark. A prior analysis on the same substrate found that deterministic retrieval already gets the right evidence into context most of the time, so the dominant remaining failure is not retrieval but reconciliation: the relevant evidence is present in the assembled context, and the reader still answers incorrectly. We give this failure a short name for reuse, assemble‑internal, meaning the evidence was assembled into context but used incorrectly. Re‑ranking retrieved turns cannot fix it, because the turns are already in context; only a change to how the reader uses the evidence can. The loop has a small vocabulary, defined here once and used consistently after. A failure regime is the diagnosed reason a question failed (for example, evidence dropped at the context budget, or assemble‑internal). An action seam is the part of the pipeline the loop is allowed to edit in response (re‑weight retrieval scores, reorder the assembled turns, or edit the reader’s prompt). The regime‑to‑seam mapping is the fixed routing from a diagnosed regime to the seam that can address it. So the loop, in one sentence: diagnose the dominant failure regime, route it to its seam, have a language model author a repair at that seam, and promote the repair only if it survives static checks, sandbox execution, in‑sample evaluation, and held‑out validation. Repairs are executable patches over named pipeline points; for the reconciliation case study the seam that matters edits the reader‑prompt fragment, so the repairs that work here are reader‑prompt edits, not arbitrary generated runtime code. One idea runs through the whole paper and is worth stating plainly up front: the promoted prompt is not the destination. A prompt repair is a high‑bandwidth probe. When independently discovered repairs keep encoding the same evidence‑use behavior, and their occasional regressions reveal exactly when that behavior should not fire, the prompt has pointed at a candidate deterministic operator. The lasting value of the loop is less the prompt it promotes than the operator that prompt reveals. This reframes a modest accuracy gain as a discovery instrument, and it connects the case‑study result (Section 5), the failure analysis (Section 5.9), and the forward agenda (Section 8). It is also why the substrate matters past the experiments here: the operators these probes reveal belong as typed projections on the event log itself (Section 8.4). Regimes sits at the intersection of four maturing threads: self‑improving agents (SICA, Robeyns et al. (2025); GRASP, Moll et al. (2026)), held‑out prompt and program optimization (DSPy, Khattab et al. (2023)), executable failure‑mode analysis (MAST, Cemri et al. (2025); AgentDebug, Zhu et al. (2025)), and long‑context memory benchmarks (LongMemEval and LongMemEval‑V2; Wu et al., 2024; Wu et al., 2026). The contributions, ordered from the enabling substrate to the specific case study: 1. 1. ActiveGraph as an auditable substrate for controlled improvement loops: event sourcing, deterministic replay from the log, cached model and tool responses, and the loop’s own history recorded as events are what make autonomous improvement easy to build, replay, inspect, constrain, and audit. The empirical sections demonstrate that this affordance is real: the loop re‑homes cleanly across tasks (Section 4) and produces a held‑out‑validated, fully auditable improvement run (Sections 5, 6). 2. 2. Regimes as one improvement loop built on that substrate: diagnose, propose, gate, promote, rotate, with the full history recorded as events, demonstrated to improve one task (LongMemEval‑S) under one reader. Its target‑agnosticism is established at the level of interface and control flow, not as a multi‑task empirical result. 3. 3. A held‑out promotion gate that rejects overfit repairs, installed before the action space was widened to free‑text prompt edits. The gate is the loop’s trust guardrail and the component it shares with prior gated optimizers (DSPy, GRASP) rather than its novelty: one run’s in‑sample +0.18 collapsed to held‑out +0.04, and the gate kept the overfit portion out. 4. 4. A failure‑regime taxonomy that routes each failure to the pipeline location that can address it (score, assembly, or reader‑prompt). The routing is the organizing heuristic of the loop; whether the diagnosis step adds value beyond supplying failed examples to a held‑out‑gated author is supported here only indirectly and is the primary open question (Section 5.8, Threat 11). 5. 5. A LongMemEval‑S case study showing modest but consistent improvement: held‑out gains of +0.05 to +0.10 in four of five seeded splits, near zero in the fifth, with the fifth split itself an informative over‑promotion finding. 6. 6. A hypothesis, induced from the discovered repairs and two analyzed regressions, that prompt repairs are useful discovery probes but should later become guarded deterministic operators that fire on detected structure rather than coarse prose rules. No operator is built or evaluated here; this is a forward bet, not a result. The substrate‑supported loop is target‑agnostic at the level of interface and control flow, demonstrated by re‑homing the loop onto a second, structurally different task (text‑to‑SQL) with byte‑identical loop behavior. We do not claim that the loop improves multiple tasks empirically; the measured improvement is specific to LongMemEval‑S under a single reader (claude‑sonnet‑4‑6). The calibrated claim, stated once so it frames what follows: an event‑sourced runtime makes the diagnose‑route‑repair loop tractable and auditable, and on LongMemEval‑S that loop produces a modest, directionally consistent held‑out improvement, with two of five splits individually significant and the pooled count significant only as a descriptive same‑pool summary. The lasting contribution is the substrate‑supported machinery plus the evidence‑use operators its prompt repairs reveal, not a new record on the benchmark. Reader’s map. The paper has one spine: an event‑sourced runtime makes controlled improvement loops tractable, Regimes is one such loop, LongMemEval is the stress test, and guarded operators are the implication. Section 2 places the work among self‑improving agents, prompt optimization, and failure taxonomies. Section 3 defines the loop and its vocabulary. Section 4 tests whether the loop is genuinely target‑agnostic by re‑homing it onto a second task. Section 5 reports the LongMemEval case study, including the multi‑seed replication and the seed‑101 over‑promotion finding. Section 6 audits the measurement itself, including the regime class