Anthropic Engineer Reveals: Long-running AI Agents Can Now Auto-generate Complete, Usable Apps in 3-5 Hours

📌 Source: https://www.youtube.com/watch?v=mR-WAvEPRwE

⚡ Key Takeaways: AI can now build a complete, fully functional app in 3-5 hours. After a year of iteration, long-running AI agents have improved continuous stable operation by 12x. Multi-role development has a far higher success rate than single-agent iterative loops. Self-evaluation is the biggest trap in AI development; an independent adversarial evaluator is the correct approach.

In the first keynote of a recent global AI engineering conference, a member of Anthropic’s applied AI team shared hard-won lessons from a year of developing long-running AI agents. They not only deconstructed the undisclosed scaffolding implementation details behind industry demo videos but also openly discussed the pitfalls they encountered and the viable solutions they validated. Long-running AI agents are AI systems capable of running continuously for hours or even days, automatically completing complex, lengthy tasks. The most typical application is fully automated code writing and app development.

Previously, many believed that AI development only required stronger models and larger context windows. However, a year of practice at Anthropic has proven that an AI agent’s capabilities are always the result of the model and its outer scaffolding evolving together. Distributing tasks among different AI roles yields results orders of magnitude better than having a single AI do everything.

❓ What Makes Long-running AI Development So Difficult?

Long-task development cannot be solved by simply enlarging the context window. There are three core issues that must be addressed.

The long-running agents discussed here refer to AI agents that can run continuously for 5-6 hours or even several days. Anthropic engineers summarized three core categories of problems:

First, context issues. Beyond the well-known context window limit (the maximum amount of text a large model can process at once; exceeding it causes loss of earliest input information), two other frequently overlooked problems exist: context decay (also called context rot, where the longer the conversation, the worse the model’s understanding of early content, making it more prone to errors) and context anxiety (the model senses it’s approaching the window limit and subconsciously rushes to a conclusion, ending the task early even if features are incomplete).

Second, planning issues. Native large language models are inherently bad at breaking down long tasks, easily deviating from the original requirements.

Third, output judgment issues. Large models often mistake half-finished features for completed ones, and they are unable to find their own errors when checking.

🔄 The Secret to a 12x Improvement: Model and Scaffolding Co-evolution

Don’t assume that as models get stronger, outer scaffolding will disappear. Scaffolding will always exist, evolving only alongside the model’s capabilities.

First, a definition: scaffolding here refers to the peripheral framework code wrapped around the large model, providing additional capabilities like memory, loops, tool calling, and state management. It is the core foundation for a long-running agent’s stable operation.

Anthropic’s iteration over the past year has a very clear pattern: whenever the model’s capabilities are upgraded, the scaffolding is updated simultaneously. If a capability gap is found in the model, it is first supplemented with scaffolding. Then, real data generated by the scaffolding is used to optimize the model, creating a virtuous cycle of co-evolution.

The results of this approach are significant: to achieve the same 50% task completion rate, Opus 3.7 from a year ago could only run stably for 1 hour, while the current Opus 4.6 can run for 12 hours — a 12x improvement.

The earliest Claude Code often made errors even in simple bash commands and string escaping, running stably for only 20 minutes at most. Now, most of Claude Code’s own code is written by Claude Code itself, and it can run stably and continuously for days.

Today, Anthropic’s long-running agents can take a user’s request and produce a complete, out-of-the-box functional application in 3-5 hours.

🤝 Single Agent vs. Multi-Agent Team? Copy This Division of Labor Logic

Breaking a long task into different roles and assigning them to different AIs, simulating the division of labor in a human development team, yields far higher success than having one AI do all the work.

The traditional single-agent loop development has an unavoidable problem: if the generated content doesn’t meet standards, the AI only applies patches on top of the original, making things messier and messier, with errors never truly being fixed.

Anthropic’s validated solution is to directly split development into three roles, each with its own independent context window, fully simulating a human development team structure:

• Planner (Project Manager): Responsible only for breaking down the user’s vague requirements into high-level iterative tasks. Avoids getting into fine-grained technical details upfront, preventing early erroneous decisions from being amplified in subsequent iterations.

• Generator (Developer): Responsible for writing the code to implement features.

• Evaluator (QA Tester): Responsible for checking whether the features meet the requirements.

The core innovation of this approach, which is a key step beyond the previously popular Ralph Loop, is that the Generator and Evaluator first negotiate clear “completion criteria” before writing any code. These are written as testable acceptance contracts, and the Evaluator only checks against the contract, not against the Planner’s vague requirements.

A very clear comparison test: using the same prompt “make a retro game creation tool” and the same model, the two approaches yielded wildly different results:

• Single-agent loop: The UI looked okay, the sprite editor was barely usable, but the core game playback mode was completely broken — no keys responded.

• Multi-role framework: It took 6 hours to automatically produce a complete product, Retro Forge. It included a new project dialog, a full 54-color 8-bit palette, and even automatically added an AI sprite generation assistant. The core playback mode worked perfectly: physics loops, directional controls for characters, and collision detection all functioned. The entire development was broken down into 27 fine-grained acceptance contracts, allowing all issues to be localized and fixed.

Anthropic engineers validated a counter-intuitive but very practical conclusion: tuning an independent evaluator to be strict and critical is far easier than making the Generator self-critical. It’s like how most people can tell if a painting is good or bad, but drawing a good one yourself is much harder. This capability gap can be directly exploited in engineering. Self-evaluation itself is a trap in AI development; you must use an independent adversarial evaluator.

⚔️ Latest Progress: The Evaluator Actually Opens Web Pages to Test for Bugs

Anthropic’s latest experiment with the Generator-Evaluator adversarial model has the Evaluator actually run the code for testing. This has a much lower miss rate than simply reading the code to find errors.

The latest approach being tested by Anthropic engineers draws on the idea of Generative Adversarial Networks (GANs) (a classic AI architecture that splits the model into a content generator and a content discriminator). It completely separates generation and evaluation: the Generator focuses on building features and writing code, while the Evaluator specializes in finding errors and scoring.

Unlike previous methods that only checked code changes, the Evaluator in this scheme uses automated tools to open the running page, click on all features for testing, and then return feedback to the Generator for iterative optimization.

Evaluation scores come from four dimensions: Design, Originality, Craftsmanship, and Functionality. Currently, the weights lean towards Design and Originality, which nicely addresses the common complaint about AI-generated interfaces being all the same.

Engineers also observed an unexpectedly positive phenomenon: the latest version 4.6 model is very willing to discard all previous work and start over. It doesn’t get stuck in a situation where the Evaluator misses a problem and the Generator stubbornly holds onto a flawed output. This behavior is exactly the same as a human programmer: when code becomes messy, deleting and rewriting is cleaner than patching. This solves the single-agent loop’s problem of accumulating bad patches.

This model doesn’t conflict with multi-agent team architecture. It is itself a subset of the multi-agent architecture, embeddable into every step of any complex workflow, where any fine-grained task can be assigned its own dedicated evaluation agent.

💡 Key Quotes

• In an uncertain world, making a determined failure is better than an uncertain success.

• It’s much easier to criticize a painting than to paint one. The same goes for large models.

• The improvement of a long-running agent’s capabilities is the result of the co-evolution of the model and the outer scaffolding. Scaffolding won’t disappear; it will only iterate alongside the model.

• Self-evaluation is a trap in AI development. You must use an independent adversarial evaluator.

• The technological frontier of AI will not shrink; it will only keep moving.

• A project that looks finished doesn’t mean it actually works.

• The core skill in developing an AI agent is learning to think from the perspective of the large model, correctly identifying the problem, and then making adjustments.

• We don’t aim to build an AI that requires human cleanup; our goal is to create a fully autonomous development agent.