What are you actually evaluating these days: prompts, context, or the whole harness?
Summary
A discussion about the focus of AI evaluations, questioning whether practitioners are optimizing prompts, context, or the entire harness, and noting a shift toward holistic optimization.
Similar Articles
Do you eval the whole harness or each of its parts?
A discussion question about whether to evaluate a machine learning harness as a whole or evaluate its individual components separately.
The Answer to the Harness Question (2 minute read)
The article argues that AI harnesses serve two distinct purposes—providing context about what the user wants (intent) and instructions on how to achieve it (execution)—and that these age differently: execution instructions become less valuable as models improve, while intent context becomes more valuable.
@rohanpaul_ai: This paper shows that agent performance depends less on prompts alone and more on the harness around them. “Agent intel…
This paper argues that AI agent performance depends more on the harness (control layer) than on prompts alone, proposing natural-language agent harnesses to make design choices inspectable and portable.
A lot of conversation around Harness Engineering, What does that even mean?
A discussion about the buzz around Harness Engineering and AI agent Harness, questioning whether the industry is moving away from LLMs deciding agent responses.
Rethinking the Evaluation of Harness Evolution for Agents
This paper rethinks how automatic harness evolution for agents should be evaluated, showing that gains may be due to increased compute rather than genuine improvements, and that evolved harnesses transfer poorly to unseen tasks.