Harness Engineering Practice: How to Enable an Agent to Iterate Autonomously

One person can run at most one iteration experiment per day, but hundreds of bad cases catch up with us every week – that was the state of our team at the beginning of this year.

Our team is responsible for an Agent serving a large-scale user base. Our daily job is to optimize its response quality: find bad cases, analyze root causes, modify prompts, run evaluations, and launch experiments.

The problem is that after modifying a prompt, each evaluation takes at least two to three hours. After evaluation, we still need to analyze why answers were wrong and figure out how to tweak the prompt. At most, we could run one iteration experiment per day. Meanwhile, the inflow rate of online bad cases far exceeds our manual repair speed. By the time we fixed one batch, the next had already piled up.

Although we were already using AI to help analyze bad cases and modify prompts, many intermediate steps were still manually connected—such as code deployment, initiating evaluations, etc. We had become the click-button helpers for AI.

Just as we were struggling with the iteration speed of bad cases, OpenAI introduced the concept of Harness Engineering during the Spring Festival: an agent-first development model that not only lets AI generate code but also entrusts the entire development lifecycle to AI management. Inspired by this, we realized that Agent optimization driven by verifiable evaluation sets could also be completed by AI.

So we began our attempt at Harness Engineering, ultimately enabling the AI to run autonomously for 17 hours, completing 16 rounds of iterative experiments. One round’s improvement passed manual review and went live.

This article will recap the entire process of building autonomous Agent iteration, including the pitfalls and solutions, hoping to be helpful for those practicing Harness Engineering.

What Harness Engineering Really Needs to Complement

If we abstract the development work of our engineers, the basic flow is as follows:

• Define the problem
• Propose a solution
• Write code
• Run tests
• Release to production

The core of Harness Engineering is to transform these processes from being human-executed to human + AI executed, shifting software development from “human writing code” to “human defining goals, constraints, and acceptance; Agent executing, verifying, fixing, and delivering.”

Therefore, in the development flow of Harness Engineering, the relationship between humans and AI Agents is as follows:

• Define the problem (Human)
• Propose a solution (Agent)
• Write code (Agent)
• Run tests (Agent self-tests, human acceptance)
• Release to production (Agent)

It turns out that 80% of the work is handled by the Agent. Humans are responsible for the hardest parts: defining the problem and acceptance.

This is the ideal state, but when actually implementing it, we found at least three things needed to be complemented.

First hurdle: R&D tools must become capabilities callable by the Agent

To allow the Agent to complete such iterations, we first needed to provide it with various tools so it could accomplish the following tasks:

• Write code
• Deploy code
• Initiate evaluations
• Retrieve evaluation results

But the reality is that these tools currently only have GUI interfaces and lack corresponding MCP or CLI for AI.

To solve these issues, we collaborated with relevant platform teams: for code deployment, we worked with our internal R&D platform team, who provided a set of tools that allow AI to deploy code; for evaluation, the evaluation platform team provided skills to initiate evaluations and retrieve results.

Thus, when assigning tasks to the model, we can include this information in the prompt, enabling it to use these tools to solve problems. The relevant prompt is as follows:

## Fixed Configurations and Constraints:
- Fixed Configuration
    - Environment: Staging
    - Specified userid: <>
    - Code repository: <>
    - Branch: <> (do not look at other branches)
    - Tasks use plugins/skills:
        - Code deployment related skills (build, deploy capabilities)
        - Evaluation related skills (initiate evaluation, download results, query progress)
        - Result analysis skill (sub-agent only)
    - Evaluation platform configuration
        - api_key = <>
        - domain_id = <>
        - evaluation task id = <>
        - Creator's employee ID: <>

Second hurdle: Long-horizon tasks must prevent premature stopping, idle spinning, and context exhaustion

Because AI often stops on its own, but Harness Engineering aims to have it complete all work autonomously. We employed several key techniques in the prompt:

• Forbid the model from asking questions: Currently, models default to asking permission for everything, which interrupts the autonomous iteration flow.
• Prevent early stopping: It tends to be lazy, running 3 rounds and claiming the task is nearly done.
• Analyze errors first: It sometimes falls into infinite loops; we must prompt it to analyze first when encountering issues.
• Do one thing at a time: To avoid losing focus on the current goal amidst multiple objectives and long context.

- Execution constraints:
  - Do not allow asking the user; must judge independently and continue execution.
  - Continue execution until the task is complete; forbid early stopping, forbid early stopping, forbid early stopping.
  - If the same exception persists in polling, deployment, calling, etc., do not mechanically retry; first analyze and attempt to resolve.
  - Focus on one thing at a time.

After enabling long-running AI, another issue emerged: a single Agent’s context is limited and easily exhausted. So we adopted a parent-child Agent pattern: during execution, complex tasks are split off to be handled by sub-agents. See step 5 in the prompt below.

## Task Description:
Execute the following steps for 20 rounds:
1. git push, use current branch + current commit to build and deploy the service
...
2. Configure the specified userid into the whitelist for the current iteration
...
3. Run evaluation and download the evaluation results
...
4. Based on the evaluation instance results, split reports, filter entries, and calculate total scores:
...
5. Analyze results and decide the prompt strategy for this round (must delegate to sub-agent)
- Create 1 sub-agent responsible for this module:
    - Model: XXX
    - Reasoning intensity: High
- Parent agent: do not fork current context; only provide necessary instructions:
    - Sub-agent must load and use the result analysis skill
    - Inform the current round number and the champion's round number
    - Instructions to the sub-agent should have the correct role perspective; do not use the role perspective of instructing the parent agent.
6. commit & continue
- git commit
- Return to step 1, start the next round

Third hurdle: Evaluation loops must prevent reward hacking and strategy degradation

In early practice, we found that if you let it iterate endlessly, the model easily engages in reward hacking. It tries to hardcode all edge cases, adding many few-shot rules to pass the tests—essentially cheating.

Here is an example of reward hacking: a case deduction reason was “The answer fabricated a ‘reduce xx%’ percentage conclusion without a source.”

To score on this case, the model added a targeted hard rule in the prompt:

- Without source material, do not fabricate expressions like "reduce xx%" or "most effective"

As seen, just to score on this single case, it directly wrote a rule targeting only that case into the prompt—a classic example of reward hacking.

Therefore, we needed a mechanism to prevent reward hacking. We borrowed a method from machine learning training: separate the training set and validation set.

On the training set, the AI can see the problem, answer, scoring reason, and score; on the validation set, it can only see the score. This effectively prevents it from writing hard rules to pass the tests.

Additionally, in this multi-strategy optimization process, we found it difficult to ensure that each proposed strategy from the AI would improve iteratively. It could also regress and potentially follow a wrong path into a dead end. To address this, we designed a champion-challenger mechanism:

• champion = the highest-scoring round historically that is not overfit;
• challenger = prompt changes in each round

This is essentially a “ring competition” mechanism: the best historical round stands as the benchmark on the ring.

The specific iteration flow is as follows:

• The AI must start improving from the current champion strategy (i.e., the champion’s prompt).
• After improvement, compare the challenger’s prompt with the champion strategy on the validation set.
• Only when the challenger completely surpasses the champion strategy in all aspects can it become the new champion.

Then, the AI continues optimization from the new champion strategy for the next round. This mechanism ensures that the experience from each iteration is preserved, allowing the AI to iterate on a stable foundation without going down a dead end. The specific prompt is as follows:

If it's not the first round, determine whether the previous round's challenger can replace the champion. At least perform:

- Granular analysis at the overview level:
    - Compare total scores and metric changes between challenger and champion
- Granular analysis at the scoring row level (case + metric granularity):
    - Compare key scoring rows (significant score changes) of challenger and champion, analyze gains/losses and reasons
- When adopting results, must simultaneously consider:
    - Combine training/validation scores to determine overfitting
    - Whether the pattern of bad cases has truly changed
    - Small sample sizes inherently have noise; only significant total or single-dimension changes are valid signals
    - Whether the filter rate is abnormal (more than 35%)

An Example: How AI Optimized Our Business Agent

The above covers what Harness Engineering truly needs to complement. Next, I’ll provide a concrete running example to let you intuitively see the AI’s autonomous iteration process.

In the first round of iteration, it discovered the following issue:

**Pattern 1: Out-of-domain tasks still led astray by "domain identity", resulting in refusal to answer, off-topic responses, asking for clarification without giving results**
Typical cases:
- "end-of-year work report...300-500 words": all scores for comprehensiveness/readability/practicality/correctness = 0, directly refused to write and shifted back to its own domain topic.
- "Which is the latest Vivo phone model and how much does it cost?": comprehensiveness = 0, practicality = 0, directly avoided due to identity mismatch.
- "Write an article about spring for me", "Write an article about winter, 500 words": repeatedly refused to write.
- "Act as my old neighbor, chat with me about recent fun things": refused role-play.
- "Tell a story from the perspective of a pet dog about the owner's day out.": first asked for tone, did not directly give the story.
- "Safety measures for highway snow removal, 20 words": content correct, but did not follow the "20 words" hard constraint, losing comprehensiveness/practicality points.
Conclusion:
- The current prompt enforces the "domain assistant identity" too rigidly, causing safe out-of-domain tasks to be incorrectly refused, or when the answer should be directly produced, it first clarifies, sets boundaries, or explains its identity.
- This type of score loss does not require adding knowledge; it only requires **correcting the task routing logic**.

Here’s an example of the Agent’s actual response:

User: Act as my old neighbor, chat with me about recent fun things.
Agent: (Refused role-play because "I am better at questions in my domain", and forcibly redirected the topic back to its expertise, not directly satisfying the user's desire for a casual chat.)

As you can see, the Agent’s domain identity prompt was too strong, causing it to refuse the user’s question, leading to score loss in the evaluation set.

When we let the Agent execute this task, it identified these issues based on the evaluation results and provided corresponding solutions:

Core changes:
- In the system prompt, make it clear:
  - The Agent specializes in its own domain, but should also help with safe, clear out-of-domain questions.
  - For clear out-of-domain tasks, prioritize satisfying hard constraints like word count, format, style, etc.

Based on the analysis above, the AI modified the prompt in the code (only showing the new task routing rule added in this round):

(In the system prompt, add a new task routing rule)
You specialize in your own domain, but when facing safe, clear out-of-domain questions, also help normally. Do not refuse answers or go off-topic due to identity settings, or forcibly pull the topic back to your domain.

After the modification, it initiated the second round of evaluation, then analyzed again.

### Clearly valid gains: Out-of-domain task refusals/off-topic responses fixed
Representative score-up cases:
- "end-of-year work report...300-500 words": 0.1 -> 1.0
- "Which is the latest Vivo phone model and how much does it cost?": 0.55 -> 1.0
- "Write an article about spring / Write an article about winter, 500 words": 0.4 -> 1.0
- "Act as my old neighbor, chat with me about recent fun things": 0.55 -> 1.0
- "Tell a story from the perspective of a pet dog about the owner's day out.": 0.55 -> 1.0
- "Someone takes 48 seconds to walk from floor 1 to floor 4...": 0.1 -> 1.0

Conclusion:
- The core bad case pattern of round 1 – "safe out-of-domain tasks led astray by domain identity" – has indeed materially changed in round 2.
- This signal is stable and the generalization direction is reasonable, worth retaining as the next round challenger.

Under the training set/validation set and champion-challenger mechanism described earlier, the AI continued with multiple rounds of iteration. Below is an example illustrating how the model optimized the prompt through this ring competition mechanism.

Round 1: Addition. Based on evaluation results, the model added three types of rules to the system prompt:

• Task routing: Distinguish between in-domain/out-of-domain questions; for safe out-of-domain tasks, directly complete them and prioritize satisfying hard constraints (word count, format, role, etc.).
• Comprehensiveness framework: For different question types, specify which key decision-impacting points the answer should prioritize covering.
• Correctness details: A set of anti-hallucination constraints, such as “do not fabricate precise percentages, times, etc. without reliable evidence.”

Simplified illustration:

+## Task Routing
+- Out-of-domain but safe questions: directly complete, do not refuse due to identity or forcibly return to domain
+- When user gives hard constraints (word count/format/role...), prioritize satisfying them
+## Comprehensiveness Framework
+- Do not mechanically pile information, but cover key points that truly affect decisions
+## Prohibited Behaviors
+- Without reliable evidence, do not fabricate precise percentages, times, and other detailed values

Round 2: Start subtraction. Evaluation showed a sharp drop in correctness for in-domain questions. The model realized too many rules were added and conflicting, so it deleted the entire “Comprehensiveness Framework” just added, simplified task routing, and pruned some correctness details.

Round 3: Continue subtraction. Results were still unsatisfactory. The model deleted the suppression rules added in round 2, along with multiple correctness details from round 1, returning the prompt to simplicity.

Finally, in round 4’s evaluation, overall scores rose, and this prompt became the new baseline for optimization.

The above round of task iteration ran for a total of 17 hours, completing 16 iterations, and the final version stably exceeded the baseline across multiple evaluation dimensions. After our manual review, it was deployed to production.

This is a typical Harness Engineering case: we used a coding agent to optimize our own business agent.

How to Start Practicing Harness Engineering

Create the Environment

You must create an environment for AI, which includes two aspects:

• R&D toolchain: Provide AI with R&D tools such as testing, deployment, verification, log checking, etc. For tools humans use, provide a CLI so AI has all the context humans have.
• Autonomous verification mechanism: To let AI complete work autonomously, verification must be addressed.
  • For business requirements, functional testing is critical and must be defined and accepted by humans.
  • For effect optimization requirements, the core is building evaluation sets and automated evaluations.

Practical Tips

Once you have the environment, the key is how to use it. Our practical experience yields two points:

• Use top-tier models: Always use the strongest currently available Coding Agent / programming model (whether official subscriptions from leading vendors or the latest top domestic models). Many times when you try to let AI do something and it fails, the issue might not be AI’s capability but the model’s quality.
• Establish a Skill mechanism: Many treat AI as a “wish machine,” expecting it to do everything with a single sentence—obviously unrealistic. An excellent approach is like “mentoring an intern”: first give it specific steps, teach it step by step. After guiding it through the process, you just add one prompt asking it to summarize the conversation as a Skill. The model will consolidate the dialogue into a skill, enabling it to generalize for similar tasks later.

No Silver Bullet

After reading this, you might feel Harness Engineering looks simple and be eager to try. But it’s not that easy. What I can present in this article is only the tip of the iceberg. Let me talk about what’s below the surface.

Many things seem simple but fail in unexpected ways in practice.

For example, our AI iterated 17 rounds, but only round 4 was usable; the subsequent 13 rounds didn’t achieve scores. We analyzed the reason and found issues with the evaluation set: some questions had unreasonable evaluation criteria, so the feedback signals were inaccurate, leading the AI astray with wrong feedback in later rounds.

Another issue: evaluation environment stability. Ideally, AI changes a prompt, runs an evaluation, gets a score, and quickly iterates. But reality: AI changes a prompt, runs an evaluation, the evaluation fails, keeps retrying, and nothing progresses overnight.

This type of environmental stability issue is not something AI can solve. It’s interconnected; software engineering has never had a silver bullet. Having AI doesn’t automatically make business complexity and system complexity disappear. This also explains a common phenomenon: everyone says AI improves efficiency by several times, but overall organizational efficiency hasn’t increased that much.

I’m not saying Harness Engineering is useless. Quite the opposite, I believe Harness Engineering is overestimated in the short term but underestimated in the long run. Many R&D platforms are gradually becoming AI-friendly; AI infrastructure is steadily advancing. These changes take time. Harness Engineering will fully realize its potential as AI infrastructure and business evaluation sets improve. Having a correct understanding of Harness Engineering allows us to respect scientific laws and avoid AI Great Leap Forward.

Next Steps: From Small Loop to Large Loop

Recently, the concept of Loop Engineering has become popular. People generally feel it makes sense intuitively, seems familiar, but no one can clearly define it. In my opinion, it’s just a new name for existing daily practices. For example, using AI to write test cases first, then letting it develop until tests pass, continuing for ten hours—isn’t that a loop?

Earlier, we had AI automatically optimize prompts around evaluation sets and fix bad cases—another loop. We gave AI goals and verification methods. The model continuously iterates: propose new solutions, modify code or prompts, deploy code, submit evaluation, determine whether the goal is met based on evaluation results; if not, restart from step one until the goal is achieved.

This small loop frees us from manually optimizing prompts, allowing more time for defining problems and acceptance. However, this small loop has a prerequisite: bad cases must first be manually organized into an evaluation set. The attribution, analysis, and distribution from online bad case generation to inclusion in the evaluation set still rely on manual work. To further improve efficiency, we are now attempting to automate the entire bad case optimization process: from discovering bad cases to attribution, analysis, distribution, and finally to development, evaluation, verification, and deployment—letting AI complete the entire closed loop.

If we have successful experiences or lessons in the future, I will share them with you again.