Why The Harness Matters More Than The Model | YC Paper Club

**Why The Harness Matters More Than The Model | YC Paper Club** **TL;DR:** This article explores the underestimated importance of the "Harness" (i.e., prompt engineering, context engineering, and agent frameworks) in AI. By reviewing its development history—from simple loops to self-improving systems—it demonstrates how improvements to the Harness alone can significantly boost model performance and introduces cutting-edge exploration into self-improving Harnesses. --- ### **The Harness: The Underestimated "Scaffolding"** The Harness is essentially a wrapper, scaffolding, or prompt engineering framework. For a long time, it was considered "second-class research." However, improvements from Harness 1 to Harness 2 alone can yield an 18% performance boost, directly determining the success of projects like ArcGI. A classic comment from a month ago argued that prompt engineering isn't top-tier machine learning research. Yet, practice shows that the evolution of the Harness is a key driver of AI agent performance. This leads to the distinction between the "Static Harness Era" (where the Harness itself doesn't improve) and the recent emergence over the last six months of the "Self-Improving Harness Era." ### **From Perplexity to Test-Time Experience** A current paradox in AI development is that while we continuously push the intrinsic intelligence of models (akin to "perplexity" or "IQ"), we fail to fully leverage the "experience" generated by models during test-time to rapidly adapt to new distributions. Traditional in-context learning (ICL) saturates after about 50 examples, necessitating a switch to more complex processes like LoRA or fine-tuning. The Harness is the key to bridging this gap and enabling rapid adaptation. Take the ArcGI benchmark as an example: Claude Opus has only a 30% pass rate on a strict private test set. However, through some "unremarkable" Harnesses (wrappers and scaffolding), performance can be boosted to 95%, with Nvidia's AVO even reaching 100%. ### **Practical Case: Automated Research Agent Swarm** The author presents a practical case demonstrating the power of self-improving Harnesses. The core goal was to investigate whether diffusion language models could surpass autoregressive models through integration and sharded aggregation. The workflow is as follows: 1. **Set Goal & Seed Ideas**. 2. A **Scoping Agent** searches for relevant papers and codebases. 3. A **PI (Principal Investigator) Agent** oversees progress. 4. A **Research Agent** conducts research, seeking feedback from an advisory group (including humans) and making revisions when encountering difficulties. 5. An **Author Agent** writes the paper. The entire process is monitored and interacted with via a "cockpit" interface. The Harness's scaffolding layer allows the author to simply propose six distinct ideas, assign them to eight H100 nodes, let them run autonomously, and periodically check the generated papers. ### **A Brief History of Harness Evolution** The development of Harnesses can be summarized as continuously enriching functionality upon a "Static Harness" foundation. - **V0 (2019)**: Starting with GPT-2, it was a simple "non-terminating sequence loop" with Top-P sampling and environment interaction, featuring no tool calls or skill learning. - **Enriching Context & Action Space**: - **Few-shot Learning**: Providing examples within the context. - **Chain-of-Thought (CoT)**: Distributing predictions across multiple tokens to guide step-by-step reasoning. - **Tool Calling**: Like Toolformer and WebGPT, enabling models to call external tools/APIs. - **Context Read/Write**: Like MegPT, allowing models to perform CRUD operations on their own context. - **Skill Learning**: Like Voyager, distilling experiences from combining tools to complete tasks into reusable skills (often stored as `skills.md`). - **Code Generation as Instant Tools**: Like Intercode, directly outputting code to create tools. - **Collaboration & Reflection**: - **ReAct, Self-Refine, Reflection**: Introducing multi-agent concepts, allowing agents of different roles to collaboratively improve context, correcting outputs through evaluation and reflection loops. - **Sub-agents & Recursion**: Tools can be a set of sub-agents; for example, RLM allows recursive invocation, ultimately managed by a central orchestration agent. ### **Harness V1: The Static Framework** The structure of a Static Harness V1 includes: 1. **Agent Specification**: Defining system prompts, allowed turns, and tool call limits. 2. **Tool List, Skill List, Sub-agent List**. 3. **An Execution Loop**: This loop can be triggered by prompts (e.g., a Slack message) or a scheduled task. It includes steps like session management, context compilation, model invocation, tool execution, and context updating. ### **Towards a Self-Improving Harness** The most exciting direction is enabling the Harness itself to learn. - **DSPY**: Learns optimal system prompts from training sets, using methods like genetic programming for iterative optimization. - **Darwin Machines**: Allow improvement of the Harness code while it's running. One can envision a repository storing different agents (Harness + system prompt) that evolve through sampling, fitness function evaluation, and updates. - **Self-Modification & Meta-Harness**: A "Meta-Harness" exists that allows agents to modify their own Harness, thereby evolving into different versions. This cycle ultimately aims to yield better agents. The core function of a Meta-Harness is precisely to generate new Harnesses. --- Source: Why The Harness Matters More Than The Model | YC Paper Club (https://www.youtube.com/watch?v=n9xKblqyQ28)