AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

Hugging Face Daily Papers Papers

Summary

This paper studies strong-to-weak capability transfer at test time, showing that stronger models can build inference-time harnesses that nearly double weaker models' performance without parameter updates.

Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
Original Article
View Cached Full Text

Cached at: 08/13/26, 03:34 PM

Paper page - AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses

Source: https://huggingface.co/papers/2608.12307

Abstract

Stronger models can build inference-time harnesses that substantially improve weaker models’ task performance without parameter updates by offloading reasoning into structured code and routing.

Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter’s parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We studystrong-to-weak scaffolding: whether a stronger builder model can constructinference-time harnessesthat help a weaker target model solve tasks more reliably without any parameter updates. Using four representativeTheory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form oftest-time capability transferis highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning intodeterministic code,benchmark-specific routing, and strictanswer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model’s own capability, and weaker target models receive the largest gains. These results suggest thatinference-time harness designis an important complement to conventionaltraining-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.12307

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.12307 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.12307 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.12307 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

@AlphaSignalAI: https://x.com/AlphaSignalAI/status/2074130508833845396

X AI KOLs Timeline

Self-improving harnesses enable AI agents to autonomously rewrite their operating rules by analyzing execution traces, leading to a 60% performance boost. Research from Shanghai AI Lab introduces the Self-Harness framework, allowing lightweight models to outperform larger ones without manual engineering.

Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI

Hugging Face Daily Papers

This paper introduces Meta-Skill, a method letting a frozen Builder model learn reusable principles from Target execution feedback to construct better agent harnesses for unseen tasks, improving performance by 8.95 points on Harness-Bench and NewtonBench. It suggests a path toward system-level self-improvement by learning to build better environments rather than changing model weights.

Harness does matter

Reddit r/LocalLLaMA

The author emphasizes that the evaluation harness significantly impacts the DeepSeek V4.1 Flash AI model's performance, indicating the critical role of harness choice in AI testing.