@joelniklaus: New blog post on harness optimization. We hit Sonnet 4.6 performance with a 7x cost improvement. Fable 5 was the first …
Summary
A blog post describes how automatic harness optimization enabled DeepSeek V4 Pro to achieve Sonnet 4.6 performance on the Legal Agent Benchmark at one-seventh the cost.
View Cached Full Text
Cached at: 07/02/26, 02:17 AM
New blog post on harness optimization. We hit Sonnet 4.6 performance with a 7x cost improvement.
Fable 5 was the first frontier model release that evaluated on legal tasks. It only scored 13%, the worst performance among all benchmarks evaluated.
@Harvey released this benchmark called Legal Agent Benchmark (LAB) just a month prior. It contains a set of realistic legal matters. Each task gives the agent a closed workspace of documents (contracts, emails, spreadsheets, slide decks) and asks for a concrete deliverable: a diligence memo, an issue list, a redline, a draft. An LLM judge grades the deliverable against a long rubric containing 61 distinct binary criteria each on average.
Many frontier models such as Gemini 3.1 Pro don’t surpass 0% all-pass rate (all rubric criteria passed). With automatic harness optimization, we manage to push DeepSeek V4 Pro from 0% to 5% all-pass rate, achieving parity with Sonnet 4.6 for 1/7 of the price.
Read the blog post for the details: https://huggingface.co/spaces/joelniklaus/harness-optimization…
Don’t Train the Model, Evolve the Harness - a Hugging Face Space by joelniklaus
Source: https://huggingface.co/spaces/joelniklaus/harness-optimization Fetching metadata from the HF Docker repository...
Similar Articles
@akshay_pachaar: Sam Altman made the case for open-source harnesses in July. a month later, someone shipped it, and it's more efficient …
TrueForge is an open-source agent harness that optimizes prompt context and model calls, reducing token costs significantly compared to managed solutions, as demonstrated in benchmarks.
HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
HarnessOpt-Bench is a benchmark for evaluating LLMs' ability to optimize the harness—the prompts, tools, control flow, memory, and orchestration code—around a target agent, using a fixed evaluation budget. Experiments with five frontier LLMs show that optimizer models separate more than the coding harnesses they act through, with substantial room for improvement.
@omarsar0: Build your own harness, folks. This is absolute banger paper from NVIDIA on self-evolving agent harnesses. (bookmark it…
The paper introduces SoL-Pi, an automated harness discovery method that reduces token usage by nearly half and cuts API costs by about a third while maintaining performance on benchmarks using models like GPT-5.6 Sol and Opus 5.
@joelniklaus: Codex is overoptimised for large models: it ranks 2nd out of 10 for GLM 5.2 but drops to 9th place for Gemma-4! Almost …
A study comparing 10 coding agent harnesses on SWE-bench Pro shows harness choice dramatically affects pass@1 and cost, revealing vendor harnesses are overfitted to large models and do not transfer to smaller ones.
@levie: At Box, we've been testing Sonnet 5.5 in early access on our complex work eval with the Box Agent, and Sonnet 5.5 is an…
Box tested Claude Sonnet 5.5 in early access and reported significant performance gains in complex work evaluations across financial services, legal, life sciences, and public sector, with faster processing and reduced token usage.