AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Summary
This paper studies strong-to-weak capability transfer at test time, showing that stronger models can build inference-time harnesses that nearly double weaker models' performance without parameter updates.
View Cached Full Text
Cached at: 08/13/26, 03:34 PM
Paper page - AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
Source: https://huggingface.co/papers/2608.12307
Abstract
Stronger models can build inference-time harnesses that substantially improve weaker models’ task performance without parameter updates by offloading reasoning into structured code and routing.
Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter’s parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We studystrong-to-weak scaffolding: whether a stronger builder model can constructinference-time harnessesthat help a weaker target model solve tasks more reliably without any parameter updates. Using four representativeTheory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form oftest-time capability transferis highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning intodeterministic code,benchmark-specific routing, and strictanswer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model’s own capability, and weaker target models receive the largest gains. These results suggest thatinference-time harness designis an important complement to conventionaltraining-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.12307
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.12307 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.12307 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.12307 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@AlphaSignalAI: https://x.com/AlphaSignalAI/status/2074130508833845396
Self-improving harnesses enable AI agents to autonomously rewrite their operating rules by analyzing execution traces, leading to a 60% performance boost. Research from Shanghai AI Lab introduces the Self-Harness framework, allowing lightweight models to outperform larger ones without manual engineering.
Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
This paper introduces Meta-Skill, a method letting a frozen Builder model learn reusable principles from Target execution feedback to construct better agent harnesses for unseen tasks, improving performance by 8.95 points on Harness-Bench and NewtonBench. It suggests a path toward system-level self-improvement by learning to build better environments rather than changing model weights.
Harness does matter
The author emphasizes that the evaluation harness significantly impacts the DeepSeek V4.1 Flash AI model's performance, indicating the critical role of harness choice in AI testing.
Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
The paper introduces an on-policy expert-correction pipeline to co-evolve harnesses and models, enabling weaker AI models to improve by correcting failing turns without full imitation, preserving compatibility and enhancing performance on enterprise agent tasks.
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
This paper empirically tests the common assumption that more structured harnesses universally improve LLM agent reliability, finding a non-monotone relationship across model tiers. It introduces the HEAT-24 benchmark and reveals that strict harnesses can harm frontier chat models while benefiting reasoning models.