Tag
HarnessOpt-Bench is a benchmark for evaluating LLMs' ability to optimize the harness—the prompts, tools, control flow, memory, and orchestration code—around a target agent, using a fixed evaluation budget. Experiments with five frontier LLMs show that optimizer models separate more than the coding harnesses they act through, with substantial room for improvement.
A blog post describes how automatic harness optimization enabled DeepSeek V4 Pro to achieve Sonnet 4.6 performance on the Legal Agent Benchmark at one-seventh the cost.
Self-Harness introduces a new paradigm where LLM-based agents iteratively improve their own operating harness by mining model-specific weaknesses, proposing harness modifications, and validating them through regression testing, achieving substantial performance gains on Terminal-Bench-2.0 across multiple base models.
Retrospective Harness Optimization (RHO) is a self-supervised method that improves LLM agent performance using only past trajectories, achieving a 78% pass rate on SWE-Bench Pro without external grading.