The most insightful talk on Harness Engineering I’ve seen — and I highly recommend it.

Video link: https://podwise.ai/dashboard/episodes/8013289…

The entire talk is only 20 minutes. IBM engineer Tejas Kumar uses a single example to elegantly and clearly explain the value of a Harness.

The core takeaway of the talk is actually just one sentence: The model is only the brain; what truly brings AI into real-world work is the Harness.

We hear this line often, but this example makes it immediately obvious why it’s true.

I’ve shared my notes below, hoping they offer some insight.

1 We give an Agent with little Harness Engineering experience the task of upvoting a post on Hacker News. The workflow looks very simple.

Open the link. If not logged in, go to the login page. After logging in, return to the post page, click the upvote button — task complete.

But when a bare-model Agent hits the login page, it has no idea what to do. It fails to log in, yet confusingly tells the user the task is done and the upvote succeeded.

In reality, it never clicked anything. The Agent lied. Anyone who used Agents last year has definitely encountered something like this.

Faced with this problem, making the prompt more detailed or switching to a stronger model may not help. The issue is that this Agent has no Harness.

Speaking of which, let’s clarify what a Harness actually is.

You can think of it this way: A Harness is a complete infrastructure layer wrapped around the model. A full Harness generally includes the following parts:

1. A tool registry that determines which tools the model can call.

2. Context management that controls which information stays in the context window and which should be compressed or discarded.

3. Guardrails that set boundaries for the Agent’s operation — for example, a maximum number of execution rounds, a maximum number of tool calls, or immediate termination upon encountering an abnormal state.

4. An Agent Loop that allows the model to observe, think, call tools, and receive results round after round.

5. A verification step that checks after the task is complete whether the result was truly achieved.

Why is this layer necessary? Because models are not like programs — given a fixed input, they won’t necessarily execute along a fixed path.

They might complete the task correctly, or they might go off course halfway, forget the goal, misjudge the state, or even claim success when the task actually failed.

The role of the Harness is to add a deterministic control system around the model. Which tools can be called, when a task should stop, how to manage context, and whether a result is actually considered complete — all of this is managed by deterministic code.

Products like Claude Code, Cursor, and Codex are all Agents wrapped in a Harness. They work well not just because the underlying models are smart, but because the infrastructure layer outside is solid.

2 Let’s return to the earlier example. We start by adding layers of Harness around the Agent, one by one.

First, we add the two most basic operational controls.

One is an iteration limit. An Agent with no constraints can easily fall into an infinite loop. Sometimes when hitting a bottleneck, it will keep clicking the same button and get stuck in a loop.

The Harness can set a clear operational boundary for the Agent. For example, if the number of execution rounds exceeds a threshold, it immediately interrupts and marks the task as failed. This ensures that even a failed task ends within a controllable cost.

The other is context management. The HTML returned by the browser is already long, and stacking a few rounds of tool call logs quickly fills up the context window.

Once it’s full, the Agent starts drowning in its own history, becoming more confused the further it goes, even forgetting the original goal.

The Harness can periodically compress historical context, retaining only information valuable for the next decision. This prevents the Agent from being dragged down by excessively long context.

After adding these two layers, the Agent at least becomes controllable. But there’s still a more critical issue unsolved: when it says the task is done, how can the system be sure?

3 So the next layer of the Harness addresses the problem of verifying the Agent’s results.

In the previous workflow, when the Agent finished its loop and reported success, the system simply assumed the task was actually completed.

But what the model says is done may be a hallucination, disconnected from what it actually did.

So this layer of the Harness shifts the judgment from the model’s single output to a check of the real execution record.

In the Hacker News example, it’s actually simple: add a separate verification logic. After the task ends, the Harness runs an additional verification round, reviewing the tool call history from the execution.

Which URLs did it visit? Which buttons did it click? Did the DOM state actually change? If the DOM changed — for instance, the upvote button lit up — the task is considered successful.

If nothing changed, the task definitely failed, regardless of what the model concluded.

This step seems simple, but it’s an essential addition for every Agent. What the model says doesn’t matter; what matters is having a objective standard for whether the task was completed.

I know many people’s first reaction to unreliable Agents is that the model isn’t strong enough. But often the problem isn’t the model.

No matter how smart the model is, without a mechanism to verify whether it actually completed the task, the problem is unsolvable.

4 With verification in place, we go back to solve the Hacker News login issue. The earlier Harness strategies — iteration limits, context compression, and verification — all add constraints and oversight to the Agent itself.

But logging in requires the Agent, like a human, to actually input a username and password and complete the login step.

The specific approach: the Harness, in each loop, independently checks the browser’s current URL. If it detects that the page has entered a login state, it doesn’t hand control back to the model. Instead, it triggers a pre-written login logic.

This logic relies entirely on deterministic program execution, not model reasoning. It fetches credentials from environment variables, locates the input fields, fills them in, and hits submit.

After successful login, it navigates the browser back to the original page and returns control to the Agent, allowing it to continue the original task.

Throughout this process, the Agent has no idea what happened. It just finds itself on page A, still on page A, but its authentication status has changed, so it can proceed.

Behind this design is a very important insight: not everything should be done by the model.

Models are good at handling open-ended problems. They excel at scenarios requiring understanding, judgment, and generalization. But login is not one of them.

More precisely, the Harness does one thing: it identifies scenarios where the model is weak, takes over those scenarios with code, and then restores the world to a state where the model can continue to perform well.

With this layer added, the entire Agent finally works. The same model, the same prompt, no changes — but now it can reliably open Hacker News, go to the login page, log in, return to the original post, click upvote, and confirm completion.

It transforms from an Agent that couldn’t even handle a login into one that can run a complete workflow.

5 Looking back at these four layers of the Harness, something interesting emerges.

The first layer, iteration limits, deals with whether the system goes out of control. The second layer, context compression, deals with whether it can run to completion.

The third layer, verification, deals with whether the task was actually accomplished. The fourth layer, login takeover, deals with what tasks should not be left to the model itself.

Together, these four layers do the same thing: use a deterministic engineering system to constrain and support a non-deterministic model.

This is why the statement “the model is the brain, the Harness is the body” holds true.

The model itself is only responsible for reasoning. But the real world is a system composed of states, permissions, pages, networks, and edge cases. Reasoning ability alone cannot stably enter a real production workflow.

What the Harness does is re-converge all that complexity into an operating environment that the model can handle stably.

The most thought-provoking point of this example is this: from start to finish, the Agent’s model never changed, the prompt never changed — only the execution framework around it changed.

Yet the Agent’s performance went from a system that couldn’t even get past a login to one that can reliably run a full workflow.

Translated into broader terms: in many cases, what truly determines an Agent’s stability is the Harness.

Model improvements are certainly important. But many Agent problems are not model capability problems.

More importantly, the Harness is not magic. It’s a complete set of software engineering problems that can be decomposed, tested, and optimized.

Harnesses in AI: A Deep Dive — Tejas Kumar, IBM | AI Engineer | Podwise

Source: https://app.podwise.ai/dashboard/episodes/8013289 AI harnesses serve as essential infrastructure for grounding non-deterministic, black-box AI models in stable, controllable environments. By implementing a harness, developers enforce reliability through structural components like tool registries, guardrails for step limits, and automated verification loops. Rather than relying on complex prompt engineering, these systems provide a deterministic framework that manages context and ensures agents adhere to specific operational boundaries. A practical demonstration shows how a simple harness resolves common agent failures, such as login errors or hallucinated successes, by programmatically verifying outcomes and injecting necessary logic. As agentic workflows evolve, the industry is shifting toward dynamic, self-generating harnesses that allow agents to autonomously establish their own safety and operational constraints, representing a critical step toward more robust and reliable AI systems.

Sign in to continue reading, translating and more.

Continue (https://app.podwise.ai/signin?redirect=%2Fdashboard%2Fepisodes%2F8013289)

mindmap screenshot

Preview

preview episode cover

How to Get Rich: Every EpisodeNaval