Tag
This paper introduces a controlled evaluation framework that separates answer coverage from repeatable task specialization in LLM-generated harnesses on MATH-500, finding that additional coverage largely comes from repeated executions of the same program rather than genuine specialization gains.
The paper introduces Meta-Harness, a system that frames LLM harness design as a multi-objective search over accuracy, safety, and token cost, achieving superior performance with an agentic proposer using full filesystem access.
A perspective on LLM harnesses, tracing their origins to early multi-step tool use research like HotPotQA and GoldEn, with commentary from Christopher Potts on debates about whether LLMs alone would suffice.