Paper page - Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review

Source: https://huggingface.co/papers/2608.12440 The spec ambiguity: The spec is not ambigous UX wise. The UX requirements are clearly stated in a 20-line prompt at the begining of the job. How it could be considered ambigous, depending on how the word is used, the prompt did not contain any hint about how the codebase is built, how to do what is asked, which invariants to respect, which race conditions are involved, etc. The agent inspected the codebase in real time, and produced autonomously a 55-page long, extremely complex specification, describing how to translate the user’s intent into code.

One concrete example, section 4: on reopening there’s a three-way contention between panel initialisation, replay of already-generated content, and live streaming of new tokens, and nothing in the functional description dictates the order to resolve it in, the agent had to pick one and write it into the spec.

Since there is no oracle, it’s not like the bun port or the SWE benchmark, there is no way to prove mathematically that the result is correct. Because the behaviour, the tests, did not exist. It is checked by running the app manually, under stress test, and observing the result. This is described in the paper, and was done across 30 coding sessions with the agent, while testing manually that the refactor produced a working software. This is as much proof you can get on a system of that complexity and that size. And this is very close to what you have when given a real life complex ticket on an existing app.

How do I know it picked the right one? I don’t. This task is way too complicated for me. Again, this is not a formal method. There is no possibility of using a formal method for a new feature, given an app of almost 4000 files and thousands of features. It looks fine, it’s modular, it seems serious. I see it as a tech lead would see a task of this size, treated by a squad. He would not prove correct the code using a formal method. I guess he would check how it looks, execute it, and probably validate, given the complexity.

To show how complex it is, this is the specification generated and refined 14 times by the agent:https://aisovereignlabs.ai/docs/case-study/liveSession/logs/liveSession-logs-phase2-specification-en.pdf

Convergence: The methodology uses loops. They are not like loop engineering, which reprocess the job until it works, more or less depending what variant is used. They are correction loops. For example, the agent writes the spec, then it fixes it iteratively, like explained in table “Table 2. Specification refinement cycles”. It searches mistakes in the specification, and fixes them. We do this, in this example case study, until it can’t find any mistake, at iteration #14.

Then we let the agent code. Then we do the same after coding with an automatic code review loop, 17 times, until convergence = no more mistakes found during 2 iterations. In the end, after 31 total audit loops, the agent fixed 201 mistakes. This is all self evaluated by the agent itself, using different agentic sessions to remove as much bias as possible.

This is an empirical study yielding working testable software. There is no mathematical proof, if that’s what you expect. No one can prove a real life original task on a complex system of 750 000 lines. It can be tested, put to the test. Even better, the agent could be used in another study, with or without an oracle, independently from me.

Why did I publish this? I created the whole application, AICode, 100% dogfooded, with thousands of features, equivalent IMHO in complexity to Claude Code and Codex, alone, in 8 months, using this exact technology. I can’t prove I did, because I can’t prove anything so large and instrument 30 000 agentic sessions. But i know for a fact that it works, because the app works, and it’s far from trivial. However, I was able to take one of the hardest 4000 commits, tp instrument it fully, exposing 100% of the agentic sessions logs, and to explain everything in a 14 pages paper.

I hope I answered your legitimate and tough questions. I tried to.