@joelniklaus: New blog post on harness optimization. We hit Sonnet 4.6 performance with a 7x cost improvement. Fable 5 was the first …
Summary
A blog post describes how automatic harness optimization enabled DeepSeek V4 Pro to achieve Sonnet 4.6 performance on the Legal Agent Benchmark at one-seventh the cost.
View Cached Full Text
Cached at: 07/02/26, 02:17 AM
New blog post on harness optimization. We hit Sonnet 4.6 performance with a 7x cost improvement.
Fable 5 was the first frontier model release that evaluated on legal tasks. It only scored 13%, the worst performance among all benchmarks evaluated.
@Harvey released this benchmark called Legal Agent Benchmark (LAB) just a month prior. It contains a set of realistic legal matters. Each task gives the agent a closed workspace of documents (contracts, emails, spreadsheets, slide decks) and asks for a concrete deliverable: a diligence memo, an issue list, a redline, a draft. An LLM judge grades the deliverable against a long rubric containing 61 distinct binary criteria each on average.
Many frontier models such as Gemini 3.1 Pro don’t surpass 0% all-pass rate (all rubric criteria passed). With automatic harness optimization, we manage to push DeepSeek V4 Pro from 0% to 5% all-pass rate, achieving parity with Sonnet 4.6 for 1/7 of the price.
Read the blog post for the details: https://huggingface.co/spaces/joelniklaus/harness-optimization…
Don’t Train the Model, Evolve the Harness - a Hugging Face Space by joelniklaus
Source: https://huggingface.co/spaces/joelniklaus/harness-optimization Fetching metadata from the HF Docker repository...
Similar Articles
@sethkarten: https://x.com/sethkarten/status/2072034978112889328
Continual Harness is a reset-free, self-improving agentic harness that achieves 20.54% on ARC-AGI-3 at a cost of $774 by storing memories, reusing skills, and refining its prompt, outperforming prior baselines like Hermes and OpenClaw with greater efficiency.
@AlphaSignalAI: https://x.com/AlphaSignalAI/status/2074130508833845396
Self-improving harnesses enable AI agents to autonomously rewrite their operating rules by analyzing execution traces, leading to a 60% performance boost. Research from Shanghai AI Lab introduces the Self-Harness framework, allowing lightweight models to outperform larger ones without manual engineering.
@tonygentilcore: https://x.com/tonygentilcore/status/2075234683202531403
Glean's engineering blog details their new agent harness using 100% programmatic tool calling via code execution, which reduces token usage by 24% compared to standard tool calling. The harness manages context with tool truncation and a sandbox filesystem for long-running complex workflows.
Self-Harness: Harnesses That Improve Themselves
Self-Harness introduces a new paradigm where LLM-based agents iteratively improve their own operating harness by mining model-specific weaknesses, proposing harness modifications, and validating them through regression testing, achieving substantial performance gains on Terminal-Bench-2.0 across multiple base models.
Claude Code improved my agent harness by 40% overnight
The author introduces 'Autoharness', a tool that uses Claude Code to autonomously optimize agent harnesses by iterating on prompts and hyperparameters. This resulted in a 40% performance increase on the tau2-airline benchmark.