60-82% accuracy swing on 4B model classification task: the only variable was harness design
Summary
A pre-registered ablation on a 4B model classification task shows that harness design (rule placement, evidence order, turn structure) swings accuracy from 60% to 82%, demonstrating that poor harness design is often mistaken for poor model capability.
Similar Articles
Same model, different harness: 30-50 point performance swing. But teams still pick agents by model name.
The article highlights that agent harnesses cause a 30-50 point performance swing compared to model selection, arguing that teams should focus on instance-level verification rather than just model names.
The harness matters more than the model. A 27B behind good critics changed my mind.
A developer argues that the harness (critics, scaffolding) around an AI model is more important than the model itself, sharing an example where a 27B model with good critics became usable for coding work.
@omarsar0: Harness choice is a big deal. So much room to advance and improve results across the board with agent harnesses. Great …
This tweet highlights the new DataSpace benchmark for data agents, showing that harness choice significantly impacts accuracy across multimodal models and agent harnesses, with the best accuracy reaching 66.34% and the benchmark remaining unsaturated.
Same Model, Different Harness: Different Coding-Agent Results
This paper investigates how changing the harness configuration in a coding agent impacts performance on coding benchmarks when the model remains fixed. The study shows that a treatment harness, which shortens older tool results to manage context, improves task completion rates, especially under tight context constraints.
Harness does matter
The author emphasizes that the evaluation harness significantly impacts the DeepSeek V4.1 Flash AI model's performance, indicating the critical role of harness choice in AI testing.