self-evolving harnesses are here.

(100% open-source)

today you pick a fixed harness, and every task runs through it. a deep research question and a file-renaming task get the same memory strategy, the same planner, and the same tools exposed at every step.

JIT-Agent is a 27B open-source model that writes the harness instead. you hand it the task and the tool registry, and it emits four Python files plus a prompt config.

that happens once, before the task starts. then a second model, one you don’t train or modify, does the actual work by being called step by step from inside those files. it never writes any of the code and never sees it.

each file covers one part of a fixed four-module contract:

→ memory decides what slice of the history that second model sees at the current step. → planning turns that view into a directive for the next action. → tool policy decides which tools are exposed for that directive. → action assembles the prompt from all of it, calls the model, and interprets the reply as either a tool call to run or a final answer.

because the code is written per task, the same generator produces structurally different agents.

here are a few tasks from the JIT-Agent paper, and the harness each one produced:

1. for a request that had to find contact records, build a workbook, and email it, it compiled the requirements into a dependency graph where delivery waits on artifact verification, and stored intermediate artifacts so later nodes consume finished results instead of rebuilding them from the transcript.

2. for a multi-hop identity question, a fixed graph would commit too early to one evidence path, so it synthesized a delegate tool that opens a private research subagent with its own memory and a five-step budget. the returned answer re-enters the parent loop as an ordinary observation.

3. for a task built around comparing numbers and ranking the results, it kept the arithmetic out of the model. the harness computes the numbers by running code, stores the results as fixed values, and hands those same values to every later step, so the model reads figures instead of recomputing them from text.

holding the working model fixed and changing only the harness, JIT-Agent’s harnesses matched or beat hand-built runtimes on search and instruction benchmarks while spending well under half the tokens, and roughly a third less per case on average. writing the harness costs tokens too, and that cost sits outside those per-case numbers.

so harness design stops being one choice you commit to and becomes something written fresh for each task. JIT-Agent also keeps the harnesses that beat its current best in an archive, and pulls from it when a later task looks similar. its own weights never change while this happens.

this work is fully open-source.

repo: https://github.com/bingreeky/JIT paper: https://arxiv.org/abs/2608.25593

i have written a detailed article on what a harness actually contains before a model starts generating one.

the article is quoted below.

bingreeky/JIT

Source: https://github.com/bingreeky/JIT

Scaling Harness Intelligence via Just-in-Time Harness Evolution

What is JIT-Agent?

JIT-Agent is a compact meta-agent that writes your agent harness on the fly. Instead of precompiling one general-purpose scaffold and hoping it transfers, JIT-Agent takes a task spec, a protocol, a tool/skill registry, and a few retrieved prior harnesses, and emits an executable, task-specific harness that wraps any off-the-shelf agentic LLM — Model-as-a-Harness.

Every harness is factored into four modules — memory, planning, action, capability orchestration — implemented against the shared interfaces in HarnessFactory, so generation means emitting structured code rather than free-form agent programs. As traces and feedback come back, JIT-Agent revises the harness and updates the archive: harnesses keep improving at test time while the generator itself stays frozen.

Results. The resulting JIT-Agent-27B lifts a wide range of backbone agents across deep research, daily work, planning, and workspace tasks.

Building the scaffold turns out to be a trainable, transferable axis of agent intelligence — orthogonal to scaling the base model.

Repository layout

Each directory has its own README with the details.

Setup

1. Clone the repository

git clone https://github.com/bingreeky/JIT.git
cd JIT

2. Environment (Python 3.11)

conda env create -f environment.yml && conda activate jit

or, in an existing environment: pip install -r requirements.txt. Serving a local meta model (vLLM/SGLang + torch) is deliberately not included — the pipeline only ever talks HTTP to it.

3. Credentials

cp .env.example .env   # then fill it in

Anything already exported in the shell wins over .env, and every model role can also be overridden per run on the command line.

4. Data

Small datasets ship in the repo; anything large is a documented download.

python scripts/check_datasets.py        # present / partial / missing, per benchmark
bash scripts/fetch_datasets.sh travel   # one benchmark ("all" ≈ 1 GB)

Usage

All modes share the same benchmark adapters, execution model, judge, and scoring path. The meta model writes a harness, the execution model runs it, and the judge model grades the result. Configure credentials in .env; CLI flags override them.

1. Test a fixed HarnessFactory design. No meta model is called; the selected harness is executed and scored directly.

python -m scripts.run_seed_harness --bench xbench --list-harnesses
python -m scripts.run_seed_harness --bench xbench \
    --harness plan_and_execute --max-samples 5

See the HarnessFactory guide for the eleven included designs.

2. Use a hosted API model as the meta-agent. Hosted APIs usually do not expose prompt_logprobs, so use judge selection explicitly. META_API_KEY is read from .env.

python -m scripts.run_jit --bench xbench \
    --meta-model provider-model --meta-base https://api.provider.com/v1 \
    --selector judge --rollouts 3 --max-samples 5

3. Evaluate the JIT checkpoint. Serve the checkpoint, then use its tokenizer for the published log-probability selector.

MODEL=JIT-Agent/jit-27b SERVED_NAME=jit TP=4 \
    bash scripts/serve_meta_model.sh

python -m scripts.run_jit --bench xbench \
    --meta-model jit --meta-base http://127.0.0.1:8000/v1 \
    --selector logprob --tokenizer JIT-Agent/jit-27b \
    --rollouts 3 --meta-temperature 1.0 --max-samples 5

Drop --max-samples for a full run. MODEL may also be a local checkpoint path; SERVED_NAME must match --meta-model. The shell wrapper bash scripts/run_jit.sh xbench reads the same settings from the environment.

Key arguments

Supported benchmarks are xbench, deepsearchqa, agentif, officebench, odyssey, shopping, and travel.

Output and resume. JIT runs separate generation, selection, and execution artifacts:

summary.json    headline metrics + how the run was configured
generate/       the N candidate harnesses per case, with prompts and responses
select/         the pick per case, the rule that produced it, per-candidate scores
execute/        the harness that actually ran, its trajectory, and the numbers you report

Fixed-harness runs write summary.json, scores.jsonl, and per-case reports directly. Re-running an identical command resumes completed work and retries only infrastructure failures; --skip-generate and --skip-select reuse earlier JIT phases.

Detailed documentation: JIT pipeline · HarnessFactory · CLI and runtime. Run either entry point with --help for the full flag list.

Citation

If you find JIT-Agent useful, please cite:

@misc{zhang2026jitagentscalingharnessintelligence,
      title={JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution},
      author={Guibin Zhang and Leo Lu and Fangzhou Xie and Kang Zhu and Junhao Wang and Zhifei Xie and Zhaochen Yu and Zihang Liu and Zhongxiang Sun and Qiankun Li and Yue Liao and Heng Chang and Xiaobin Hu and Qibing Ren and Wangchunshu Zhou and Shuicheng Yan},
      year={2026},
      eprint={2608.25593},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.25593},
}

License

See LICENSE.