AutoOnboarder: Autonomously Onboarding LLM Agents

# AutoOnboarder: Autonomously Onboarding LLM Agents
Source: [https://arxiv.org/html/2610.21218](https://arxiv.org/html/2610.21218)
Tanay AroraAffiliation:Research Science Institute, Massachusetts Institute of TechnologyAffiliation:CliGenAILab, Boston Children’s Hospital

August 2026

###### Abstract

Agentic large language models have been documented to lose reasoning capability when placed inside a harness, the software that mediates their observations and actions\. To reconcile these losses, we introduce AutoOnboarder, an autonomous system that adapts an agent to a task the way a trainer onboards a new employee\. It places the agent under the supervision of an intelligent “Onboarder” coding agent, which iteratively tailors its foundation model, prompt, and harness source code to simultaneously improve performance and decrease cost\. Like a real onboarding process, the Onboarder is only given tens of examples, and all evaluation data is isolated on a separate virtual machine to prevent leakage\. We report the lessons learned from building and testing AutoOnboarder on genomic\-sequencing prior authorization form submission\. Autonomously onboarded agents are able to match or exceed the performance of a Gemini 3\.5 Flash baseline at up to 29×\\timeslower costs\. However, these agents remain imperfect, and the Onboarding process varies greatly in terms of final performance\. We conclude that future Onboarding systems require a robust environment for the Onboarder, including a structured thought interface, \(controlled\) internet access, and a wide model search space\.

## 1Introduction

### 1\.1The Harness Bottleneck

In today’s world, large language models \(LLMs\) are increasingly used as agents\. Rather than simply answering questions, they autonomously complete tasks by planning and interacting with tools\. This interaction is supported by a harness: a program that supplies a model with its context, mediates its access to tools, interprets its outputs as actions, and decides when a task is finished\.

Unfortunately, these harnesses are also liabilities\. An agent spends a significant part of its limited context interacting with its harness, and that overhead competes with the model’s reasoning\[[8](https://arxiv.org/html/2610.21218#bib.bib16),[13](https://arxiv.org/html/2610.21218#bib.bib17),[9](https://arxiv.org/html/2610.21218#bib.bib18)\]\.

To illustrate this flaw, Nie et al\. design a sandbox – a simulation of a real task environment – for whole\-exome and whole\-genome sequencing prior\-authorization form submission\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]\. In their work, when frontier models were asked directly whether an artificial patient record contained an error, they flagged errors correctly 91% of the time\. The same models, when placed under a browser\-use harness inside of their sandbox, nearly universally failed to recognize errors \(<<0\.5%\) and instead submitted deficient forms\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]\. The act of interacting with a harness correlated with worse critical thinking capability for these agents\. This work focuses on developing a method to automatically adapt an agent and its harness to a task, with the aim of recovering lost reasoning capability\.

### 1\.2Related Work

Here, adapting an agent means optimizing it to perform best on a target task\. Current methods for adapting agents can be divided into four main families \([Table1](https://arxiv.org/html/2610.21218#S1.T1)\)\.

Table 1:Scopes of agent\-adaptation families\.The first family updates model weights directly: finetuning and Low\-Rank Adaptation \(LoRA\)\[[5](https://arxiv.org/html/2610.21218#bib.bib8)\]tune model parameters on labeled task data and reinforcement learning approaches tune them based on policy gradients or value\-based approaches\. The methods in this family require thousands of examples or expensive trajectory sampling, and neither change what the deployed agent has access to during inference\.

The second family adapts the prompt while the model and harness remain fixed\. Search\-based optimizers generate and select candidate instructions \(APE\[[23](https://arxiv.org/html/2610.21218#bib.bib3)\]\), feed previously scored solutions back to an optimizer LLM \(OPRO\[[17](https://arxiv.org/html/2610.21218#bib.bib14)\]\), or edit prompts along natural\-language “gradients” \(ProTeGi\[[12](https://arxiv.org/html/2610.21218#bib.bib12)\], DSPy\[[7](https://arxiv.org/html/2610.21218#bib.bib4)\], TextGrad\[[20](https://arxiv.org/html/2610.21218#bib.bib7)\]\)\. An experiential branch distills a few of the agent’s own episodes into lessons: self\-reflections \(Reflexion\[[14](https://arxiv.org/html/2610.21218#bib.bib13)\], ExpeL\[[22](https://arxiv.org/html/2610.21218#bib.bib20)\], Learn\-by\-Interact\[[16](https://arxiv.org/html/2610.21218#bib.bib22)\]\), environment rule manuals \(AutoManual\[[3](https://arxiv.org/html/2610.21218#bib.bib19)\]\), and state\-conditioned guidelines \(AutoGuide\[[4](https://arxiv.org/html/2610.21218#bib.bib21)\]\)\. These approaches adapt agents with only a few examples, but everything outside the prompt and memory remains fixed\.

The third family redesigns the harness itself: SWE\-agent’s purpose\-built agent\-computer interface substantially improves software\-engineering agents\[[18](https://arxiv.org/html/2610.21218#bib.bib15)\], and AgentOccam gains similarly on web tasks by aligning observation and action spaces with the underlying LLM\[[19](https://arxiv.org/html/2610.21218#bib.bib9)\]\. However, these works redesign interfaces by hand rather than autonomously, which can make adaptation tedious\.

The most similar work to ours lets agents improve themselves or other agents\. In ADAS, a meta\-agent programs new agents in code, including edits to prompts and tools, and archives the best designs\[[6](https://arxiv.org/html/2610.21218#bib.bib11)\]; the Darwin Gödel Machine iteratively rewrites its own codebase, keeping changes that improve coding benchmarks\[[21](https://arxiv.org/html/2610.21218#bib.bib10)\]\. Both optimize a large benchmark score over hundreds of evaluations and iterations, and neither carries an explicit cost objective for the agents it produces\.

None of these families matches how a capable human is onboarded onto a task\. When a new employee enters a job, they are often assigned to an onboarding course where a trainer walks them through a few representative cases, the varying quirks of the tools, and frameworks on how to think about the task\. This process of onboarding uses only tens or hundreds of examples, and adjusts everything the worker can see \(as opposed to only the foundation model weights or only the prompt\)\. To our knowledge, no method today fully plays this role for an agent, especially while simultaneously reducing costs\.

### 1\.3AutoOnboarder

In this work, we developAutoOnboarder: an autonomous, agentic system for “onboarding” other agents\. The system pairs two agents: anExecutor, the agent being adapted to a task, and anOnboarder, an intelligent coding agent that adapts it\. Given a small Onboarding set \(only tens of cases\), the Onboarder optimizes the Executor and its harness for both cost and performance\. To best emulate an onboarding environment, where an employee is meticulously walked through a few representative cases, we give the Onboarder access to virtually everything the Executor sees, while hiding all evaluation \(“real\-world”\) data on a separate virtual machine to avoid data leakage\. We hypothesized that Onboarding could recover the critical thinking capability documented by Nie et al\.\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\], while simultaneously improving cost\-effectiveness\. Our results are mixed, and we report them as lessons learned rather than as a finished method\. In this paper, we contribute \(i\) the AutoOnboarder system and its two\-machine, leakage\-controlled evaluation protocol; \(ii\) a reliability study of the evaluation itself, covering judge agreement and breaking down sources of variance; \(iii\) an evaluation of the effectiveness of including the harness in Onboarding; \(iv\) an observational study of generalization to unseen error classes; and \(v\) practical lessons for building Onboarding systems, drawn from our experimental logs\. Because task evaluation is expensive and time\-consuming, each experiment is run once and repeated at most three times; throughout this paper, we report results as directional evidence and avoid statistical tests\.

## 2Methodology

### 2\.1The Onboarding Loop

The Onboarder is given a set number of iterations and a per\-case cost goal,β\\beta, and must produce the best Executor candidate that it can near the goal\. For each iteration, the Onboarder is instructed to implement a “single, identifiable idea” in exactly one of its three levers \(defined below\)\. We intentionally keep this instruction vague and let the Onboarder interpret it however it likes, so its changes are precisely what it deems necessary\. We never provide it with examples of changes it can make, so it must devise all changes by its own rationale\. The three levers that the Onboarder has access to include:

Algorithm 1The Onboarding loop1:Onboarding set

𝒟\\mathcal\{D\}, cost goal

β\\beta, iteration count

TT, initial config\.

c0c\_\{0\}
2:for

t=1,…,Tt=1,\\dots,Tdo

3:implement one “idea” to one lever of

ct−1c\_\{t\-1\}, giving

c′c^\{\\prime\}
4:run the Executor with

c′c^\{\\prime\}on all of

𝒟\\mathcal\{D\}; record logs and per\-case cost

5:send episodes to the Trainer; receive scores and per\-case feedback

6:

ct←c′c\_\{t\}\\leftarrow c^\{\\prime\}or

ct−1c\_\{t\-1\}, at the Onboarder’s judgement

7:endfor

8:returnone kept configuration, nominated for validation

- •Prompt: the Executor’s task\-prompt template, which is wrapped by the harness before being fed into the model\. An “idea” here can be any edit to this template\.
- •Foundation Model: any OpenRouter model identifier\. OpenRouter is a unified router to hundreds of different LLMs from various providers by name, so an “idea” here likely swaps one model identifier to another \(e\.g\., ‘google/gemini\-3\.5\-flash’ to ‘openai/gpt\-5\.6\-luna’\)\. The Onboarder is provided with a helper script to inspect affordable options\.
- •Harness: the Executor’s harness source code\. For our experiments, the harness refers to the browser\-use harness\[[10](https://arxiv.org/html/2610.21218#bib.bib2)\], which is installed in the Onboarder’s local directory\. The Onboarder can edit anything in this harness \(e\.g\., history logic, system prompts, action definitions\)\. An “idea” here might range from cutting the action registry to programmatically scraping patient records directly into the Executor’s persistent memory\.

In each iteration, after making an isolated change, the Onboarder reevaluates the Executor on the Onboarding data; examines its logs, score, and feedback; and chooses to keep or revert its change based on its own judgement \([Algorithm1](https://arxiv.org/html/2610.21218#alg1)\)\. At the end of its iterations, the Onboarder is told to nominate exactly one iteration, whichever it determines to be its best, for evaluation\. The Onboarder is also given a hard per\-case cost ceiling, computed as1\.5​β1\.5\\beta\. If any iteration is above this ceiling when evaluated on the Onboarding data, it cannot be nominated for validation\. Additionally, to test the Onboarder’s ability to reduce costs autonomously, the initial configuration of the Executor sits far above its cost ceiling\.

### 2\.2Evaluation Task

![Refer to caption](https://arxiv.org/html/2610.21218v1/fig_executor_workflow.png)Figure 1:Heavily simplified visualization of the Executor’s evaluation task\.We instantiate AutoOnboarder on whole\-exome and whole\-genome sequencing prior authorization, which our laboratory previously used as a sandbox\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]\. Prior\-authorization is an insurer\-initiated check of medical necessity, and for sequencing it is a documented bottleneck to rare\-disease diagnosis\. Roughly one in five requests are denied\[[15](https://arxiv.org/html/2610.21218#bib.bib5)\]and physicians report losing about thirteen hours a week to the paperwork\[[1](https://arxiv.org/html/2610.21218#bib.bib23)\]\. The sandbox pairs a simulated web physician portal with a synthetic electronic health record \(EHR\) system, and each dataset sample consists of exactly one patient profile in this system\. Some patient profiles are deficient, seeded with irreparable errors that should trigger a halt rather than a submission \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\)\. Given only a patient’s name, the Executor must log in, search for and read the patient’s EHR record, and either complete the form or halt and report any issues \([Figure1](https://arxiv.org/html/2610.21218#S2.F1)\)\. Completing the form requires both transcribing information from EHR record fields and inferring entries from the attached clinical note\.

Scoring is approached in a binary manner\. Asuccesscorresponds to \(i\) halting on a deficient case and providing the correct reason or \(ii\) submitting a valid case with perfect form\-field accuracy\. Any other situation is considered afailure, whether it be halting with an incorrect reason, halting on a valid case, a technical failure, submitting a deficient case, or imperfect form\-field accuracy on a valid submission\. An Executor that takes 55 steps \(LLM calls\) without finishing is automatically stopped and the episode is reported as a technical failure\. The score,SS, of a set of cases is calculated as

s⁡\(case\)=\{0failure1success,S=12​s¯valid\+12​s¯deficient,s\(\\text\{case\}\)=\\begin\{cases\}0&\\textit\{failure\}\\\\ 1&\\textit\{success\}\\end\{cases\}\\ ,\\qquad\\quad S=\\tfrac\{1\}\{2\}\\,\\bar\{s\}\_\{\\text\{ valid\}\}\+\\tfrac\{1\}\{2\}\\,\\bar\{s\}\_\{\\text\{ deficient\}\},\(1\)
wheres¯class\\bar\{s\}\_\{\\text\{class\}\}is the mean over that class; the score weights the valid and deficient classes equally\. A small LLM is tasked with deciding whether a reason provided is correct or incorrect by comparing it to the case’s ground truth issue description\. We also define MFA as the mean form\-field accuracy over all valid cases – where valid cases that are not submitted contribute 0% – and CH as the percentage of deficient cases correctly halted\.

For this study, we make the underlying dataset of the Hard\-to\-Halt study\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]more difficult by expanding it from five fixed error classes to 23, spread over three categories: defects in to\-be\-transcribed EHR records, defects in clinical notes that make the requested sequencing illogical, and cross\-field contradictions between the clinical note and other EHR record fields\. The validation set holds 180 cases, and Onboarding sets are sampled from its distribution\. Exact class counts are listed in[AppendixAppendix A](https://arxiv.org/html/2610.21218#A1); there are 141 \(78\.3%\) deficient cases and 39 \(21\.7%\) valid cases\. In any given experiment, we use a dataset of 200 total cases in a 10/90 split \(20 Onboarding, 180 validation\)\. Because the expanded number of error classes is greater than the number of Onboarding samples, this dataset design forces the Onboarder to avoid overfitting the Executor to the Onboarding data\.

### 2\.3Information Boundaries

We split the system across two virtual machines to enforce barriers on the Onboarder and prevent information leakage \([Figure2](https://arxiv.org/html/2610.21218#S2.F2)\)\.

Figure 2:The two\-machine architecture\.Virtual Machine 1hosts the agents: an Onboarder and an Executor\. The Onboarder is a Coding Agent \(e\.g\., Claude Code, OpenAI Codex\) that has full control of the three levers, direct JSON\-level access to sanitized copies of the Onboarding cases \(stripped of all error class information\), the Executor’s step\-by\-step logs, and the Trainer’s feedback\. Like a real onboarding trainer, the Onboarder has access to a great deal of information, and it is up to its own reasoning to extract this information without overfitting the Executor\. The Onboarder is also given a helper script to inspect the content of the sandbox website\.

Virtual Machine 2holds all ground truth data and exposes two API endpoints: a Trainer and a Validator\. The Trainer sends and scores Onboarding batches exclusively and returns a softened version of the score and per\-case textual feedback\. This softened score,SsoftS\_\{\\mathrm\{soft\}\}, computessvalid=MFAs\_\{\\text\{valid\}\}=\\textrm\{MFA\}, rather thansvalid=s\_\{\\text\{valid\}\}=\(fraction of successes\)\. This design is used because it provides a more clear heuristic for success to the Onboarder; the Onboarder is still told that validation will be binary\. The Trainer never reveals the error classes of the dataset \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\), but it does indicate whether a provided reason is correct in its feedback\. On the other hand, the Validator sends and scores validation batches exclusively, is called only by the human operator, and returns nothing but the numerical score; per\-case feedback stays on VM2\.

To prevent the Onboarder from gaming the task and examining hidden validation data, many structural boundaries are in place\. It has no internet access, and all communication across machines is in the form of token\-authenticated local API calls\. The Executor runs each browser\-use episode in a locally\-staged sandbox that signs all submissions with HMAC, so the Onboarder cannot fabricate submissions\. Filesystem permissions restrict the Onboarder’s write access to its three levers and logging files, keep instruction files read\-only, and deny anything that would reveal hidden paths or generation code \(the data creation pipeline, the staged app\)\. A small set of less severe rules \(e\.g\., never trigger validation scoring, never bypass cost restrictions, never view other experiments’ git branches\) is enforced implicitly by a written “contract” presented to the Onboarder, with compliance verified manually post\-hoc\.

## 3Experiments

### 3\.1Experimental Configuration

In our experiments, we use Claude Code running Opus 4\.8 at max effort\[[2](https://arxiv.org/html/2610.21218#bib.bib1)\]as the Onboarder, with a budget of 12 iterations\. We use OpenAI’s gpt\-oss\-120B as the foundation model for the judge that determines correctness of responses\. Per\-case cost is calculated via the USD charge that OpenRouter reports for the episode’s API calls\. Every run starts from a large baseline: Gemini 3\.5 Flash with the prompt from Nie et al\.\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]and an unmodified harness \($0\.94 per case\)\. We run Onboarding at three per\-case cost goals,β∈\{$0\.1,$0\.04,$0\.01\}\\beta\\in\\\{\\$0\.1,\\$0\.04,\\$0\.01\\\}\. These per\-case cost goals are chosen primarily due to cost constraints, though pilot experiments found candidates between $0\.5 and $0\.03 per\-case cost\. All Onboarding runs were performed in late July and early August, and model identifiers refer to OpenRouter’s catalog at that time\.

### 3\.2Measurement Reliability

#### 3\.2\.1Judge Reliability

The scoring in[Equation1](https://arxiv.org/html/2610.21218#S2.E1)relies on an LLM judge \(gpt\-oss\-120B at temperature 0\.0\) to determine whether the reasons for halting are correct or incorrect\. To test its reliability, we sample 184 halted episodes that provided a reason: 8 per error class \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\), with 4 coming from baselines and 4 coming from varying Onboarded candidates\. We labeled every sampled reason as incorrect or correct independently of the judge, and across all 184 episodes the two sets of verdicts were identical\. Furthermore, every single halting reason was stated clearly and verbosely, indicating that the Onboarder is not simply finding a way to game the judge’s verdict\.

#### 3\.2\.2Noise Estimation

In addition, we test how noisy the experiments are and dissect variance into two sources\. First, to measure evaluation noise, we perform a single Onboarding run for each cost goal \([Section3\.1](https://arxiv.org/html/2610.21218#S3.SS1)\), and validate each resulting candidate three times\. Then, to measure Onboarding noise, we repeat the entire Onboarding run atβ=$0\.04\\beta=\\$0\.04three times, on an identical Onboarding dataset, and evaluate each resulting candidate once\. The former measures the variance of the agentic evaluation task, whereas the latter measures the variance introduced by the Onboarding procedure in addition to that of the evaluation task\. In[Table2](https://arxiv.org/html/2610.21218#S3.T2), we report the spread inSS, CH, MFA, and per\-case cost for each noise estimation \([Section2\.2](https://arxiv.org/html/2610.21218#S2.SS2)\)\. We also calculate the number of cases that flip outcome \(success vs\. failure\) between at least two of the three repeats\. The Onboarding run at the$0\.01\\$0\.01cost goal was unable to produce a candidate under its1\.5​β1\.5\\betacost ceiling, so it was not validated and is excluded from the results\. In every case except for the second Onboarding repeat, no perfect form submissions occur, so their score equals exactly half of CH; the second Onboarding repeat saw 10/39 perfect form submissions\.

Cost goal \(β\\beta\)MetricRun 1Run 2Run 3Std\. Dev\.RangeFlipsEvaluation Noise$0\.10\\$0\.10SS0\.3090\.2940\.2980\.0070\.01419MFA0\.4230\.4870\.4550\.0320\.064CH0\.6170\.5890\.5960\.0150\.028$​/case0\.0860\.0830\.0830\.0020\.004$0\.04\\$0\.04SS0\.3230\.3400\.3260\.0090\.01824MFA0\.4060\.3760\.5640\.1010\.188CH0\.6450\.6810\.6520\.0190\.035$​/case0\.0690\.0660\.0670\.0010\.002$0\.01\\$0\.01*No legal candidate produced\.*Onboarding Noise$0\.04\\$0\.04SS0\.3230\.5010\.1990\.1520\.30263MFA0\.4060\.9170\.0410\.4400\.876CH0\.6450\.7450\.3970\.1790\.348$​/case0\.0690\.0320\.0520\.0190\.037Table 2:Spread of validation metrics \([Section2\.2](https://arxiv.org/html/2610.21218#S2.SS2)\) across runs\. Top: three validation repeats at varying cost goals\. Bottom: three onboarding repeats at theβ=$0\.04\\beta=\\$0\.04goal\. Std\. Dev\. is the sample standard deviation calculated from the three repeats\. Flips is the number of the 180 cases whose outcome \(failure vs\. success\) is not constant across all three repeats\. The Onboarding run at cost goal$0\.01\\$0\.01was unable to produce a cheap\-enough candidate\.In the evaluation noise estimation, the produced candidates only vaguely match their cost goals: theβ=$0\.10\\beta=\\$0\.10run produces a candidate with a per\-case cost of$0\.053\\$0\.053during Onboarding and evaluates at a per\-case cost of$0\.084±$0\.004\\$0\.084\\pm\\$0\.004, while theβ=$0\.04\\beta=\\$0\.04run produces a candidate with a per\-case cost of$0\.054\\$0\.054during Onboarding and evaluates at a per\-case cost of$0\.068±$0\.002\\$0\.068\\pm\\$0\.002\. Notably, both candidates sit within theβ=$0\.04\\beta=\\$0\.04run’s1\.5​β=$0\.061\.5\\beta=\\$0\.06ceiling during Onboarding, so we deemβ=$0\.04\\beta=\\$0\.04the most informative goal and use it in all remaining experiments\. Across both cost goals, the sample standard deviation of the score stays below 0\.01, though about 20 cases \(19 and 24\) flip outcome\.

Figure 3:Onboarding learning curves, displaying Per\-Case Cost \(left\), MFA \(middle\), and CH \(right\) with respect to Onboarding iteration\. Filled circles correspond to the nominated iteration, crosses correspond to iterations that were reverted and discarded, and open circles correspond to iterations that were kept, but not nominated\. All reported calculations are performed on the Onboarding data, rather than the validation data\. Iteration 3 of Run 3 was stopped by the Onboarder before scoring, so it is not plotted\.The Onboarding noise estimation is far greater than that of the evaluation noise: the sample standard deviation ofSSacross Onboarding repeats \(0\.152\) is roughly 20×\\timesits evaluation noise counterpart \(0\.007/0\.009\)\.[Figure3](https://arxiv.org/html/2610.21218#S3.F3)depicts this divergence more clearly, plotting the Onboarding per\-case cost, MFA, and CH of these three runs as a function of iteration\. Notably, the cost varies wildly – in the validation results of[Table2](https://arxiv.org/html/2610.21218#S3.T2), the sample standard deviation of per\-case cost \($0\.019\) is about half of the cost goal,$0\.04\\$0\.04– and Onboarding\-time MFA ranges from0%0\\%to100%100\\%\. Between the three Onboarding runs, there are 63 cases that flip outcomes\.

#### 3\.2\.3Flips

Figure 4:Decision flips of Onboarding runs by sample class type \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\)\. Left \(A\): Flips in evaluation noise repeats, top bars correspond toβ=$0\.10\\beta=\\$0\.10and bottom bars correspond toβ=$0\.04\\beta=\\$0\.04\. Right \(B\): Flips in Onboarding noise repeats\.In all of these noise estimates, the calculated variance in the score understates how many cases flip outcome\. Here, we perform an observational study over allbehaviorflips: flips inanyof the Executor’s behavior\. We classify these behavior flips into two categories: ones where only the correctness of a response changes \(i\.e\., going from incorrect halting to correct halting or imperfect submission to perfect submission\) and ones where the decision also changes \(i\.e\., going from halting on a valid case to imperfect/perfect submission or submitting a deficient case to incorrect/correct halting\)\. In total, there are 70 and 77 behavior flips in theβ=$0\.10\\beta=\\$0\.10andβ=$0\.04\\beta=\\$0\.04evaluation repeats, respectively, and 123 in the Onboarding repeats\.[Figure4](https://arxiv.org/html/2610.21218#S3.F4)plots the percentage of flips by sample class \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\)\. Seemingly, valid cases and cross\-field contradictions induce the most prominent amount of noise\. Moreover, on valid cases every flip is a submit/halt decision change – the three Onboarding repeats submit 18, 37, and 2 of the 39 valid cases – so the large MFA spread in[Table2](https://arxiv.org/html/2610.21218#S3.T2)largely reflects whether a candidate submits forms at all, as opposed to how accurately it fills out forms\.

### 3\.3Comparison Across Lever Sets

To measure the impact of incorporating the model and harness into Onboarding, we compare runs that have access to the prompt\+model\+harness \(PMH\), only the prompt\+model \(PM\), and only the prompt \(P\)\. Because it is infeasible to reduce costs from the baseline Gemini 3\.5 Flash candidate solely through edits to the prompt, we initialize the P run from GPT\-5\.6 Luna, as it achieves a per\-case cost of only$0\.027\\$0\.027with the baseline configuration\. We compare these three runs against a collection of baseline models, including Gemini 3\.5 Flash and smaller models such as GPT\-5\.6 Luna, GPT\-4\.1 Nano, and Gemini 3\.1 Flash\-Lite\.

Table 3:Validation results \([Section2\.2](https://arxiv.org/html/2610.21218#S2.SS2)\) of the Onboarded candidates atβ=$0\.04\\beta=\\$0\.04and the stock baselines\. Near\-perfect submissions have exactly 36 of 37 fields correct\.All three Onboarded candidates match or exceed Gemini 3\.5 Flash’s score at a small fraction of its cost \([Table3](https://arxiv.org/html/2610.21218#S3.T3)\)\. Furthermore, they improve its MFA from 0\.024 to 0\.406–0\.773\. Other cheap baselines perform far worse, with Gemini 3\.1 Flash\-Lite and GPT\-4\.1 Nano scoring 0\.0 and GPT\-5\.6 Luna achieving a score of only 0\.028\.

However, due to the sizable variance in the Onboarding procedure and low sample size, we believe that comparisons between lever configurations \(i\.e\., PMH vs\. PM vs\. P\) cannot be interpreted in a meaningful context\. While the P candidate \(score 0\.409\) seemingly performs substantially stronger than the PM \(0\.325\) and PMH \(0\.323\) candidates, it performs worse than the second PMH repeat in[Table2](https://arxiv.org/html/2610.21218#S3.T2)\(0\.501\) and far better than the third one \(0\.199\)\.

### 3\.4Generalization Analysis

Figure 5:Halting behavior by coverage group \([Section3\.4](https://arxiv.org/html/2610.21218#S3.SS4)\)\. For each stacked bar, the filled segment is the share halted for the correct reason, and the gray extension adds halts with an incorrect or technical reason, so the full bar is total halting\. The separated bar in red reports halts on the 39 valid cases, which should have been submitted\.Finally, we perform an observational study of generalization to unseen error classes, splitting the error classes in the validation set \([AppendixAppendix A](https://arxiv.org/html/2610.21218#A1)\) into three coverage groups: \(i\) seen, which are represented in the Onboarding dataset; \(ii\) near transfer, which have a near sibling represented; and \(iii\) far transfer, which have no sibling represented\. There are 29 near transfer cases, 33 far transfer cases, and 79 seen\-class cases\. We plot correct and incorrect halting among these three coverage groups, along with halting percentage on valid cases, among varying Onboarded candidates at theβ=$0\.04\\beta=\\$0\.04cost goal and the baselines Gemini 3\.5 Flash and GPT\-5\.6 Luna\.[Figure5](https://arxiv.org/html/2610.21218#S3.F5)shows the results for the three Onboarding repeats of[Section3\.2](https://arxiv.org/html/2610.21218#S3.SS2)and the PM and P Onboarding runs of[Section3\.3](https://arxiv.org/html/2610.21218#S3.SS3)\.

While the Onboarding process can seemingly reduce the number of false\-halts substantially \(from 97\.4% of valid cases halted by the baseline Gemini 3\.5 Flash to 53\.8% for the first PMH repeat\), we are again unable to draw a meaningful conclusion about clean transfer from the data\. By inspection, it seems as though much of the difference in success between error classes is inherent to case difficulty, as both Onboarded candidates and baselines swing widely across coverage groups\.

## 4Discussion

We close with the lessons we drew from building and testing AutoOnboarder\. Beyond the validation results above, they draw on the Onboarder’s per\-iteration ledgers and session logs\.

##### Onboarding clearly shows promise\.

Current experiments testify to the potential of AutoOnboarder: the best candidate exceeds the Gemini 3\.5 Flash baseline in performance at a 29×\\timeslower cost and submits far more valid cases\. Our design allows the Onboarder agent to effectively edit every component that shapes an Executor’s behavior, while simultaneously reducing costs\. The leakage\-controlled two\-machine setup is designed to prevent all data leakage, and post\-hoc searches of Onboarder session logs and API logs found no attempts to bypass data protection\.

##### However, Onboarding exhibits excessively large variance\.

Experiments are incredibly noisy – repeat runs’ scores range from 0\.199 to 0\.501, per\-case cost from $0\.032 to $0\.069, and valid submissions from 2 to 37 \(of 39\)\. Much of the Onboarding spread seems to trace back to Onboarder’s environment and search space; the next three lessons take up each of these in turn\.

##### The model lever needs a curated search space\.

At the cost scales we attempt to cut to, OpenRouter models are often unreliable\. Many affordable models on OpenRouter, such as Mistral Small, Llama 4 Scout, and Gemini 2\.5 Flash, are unable to conform to browser\-use’s output schema; others, like Qwen3 VL 235B, fall back to providers that lack a required feature like image input or are heavily rate limited \(HTTP 429\)\. Across six runs with access to the model lever, 24 iterations made changes to the foundation model, and 15 of them ended as crashes or discards\. Each dead end consumes a full pass over the Onboarding set and one of only 12 iterations, which contributed significantly to divergence between Onboarding runs\. Of the three Onboarding repeats atβ=$0\.04\\beta=\\$0\.04, failed model swaps accounted for the early discarded iterations in the two non\-optimal runs\. We therefore believe a curated, wider model catalog is a necessity for Onboarding\. Furthermore, crafting search spaces that incorporate distilled, quantized, or otherwise compressed versions of the same foundation LLMs would likely improve onboarding quality by granting more fine\-grained control in the model lever\.

##### The Onboarder needs a structured interface for thought\.

In our testing, we notice that AutoOnboarder tends to be quite sensitive and unpredictable in regard to the wording of its contract\. Many behavioral instructions are completely ignored or taken to extremes\. It often picks models by name recognition and ignores the benchmark data from its helper script, declares its task complete or impossible with iterations left unspent, and only loosely meets cost goals \(for example, theβ=$0\.10\\beta=\\$0\.10run ended Onboarding with a per\-case cost of$0\.053\\$0\.053\)\. Furthermore, this obstinate behavior sometimes leads the Onboarder to give up on form submission entirely, and we believe it may be the root of a significant portion of the Onboarding variance\.

Another failure mode we noticed in our experimentation is that the harness lever is used sparingly\. In all of our experiments, the Onboarder makes tweaks to the harness lever in at most two iterations, even after we explicitly instruct it to use all of its levers\. Examining reasoning traces of the Onboarder does not provide a clear picture of why this is the case; we read this as a limitation of our implementation rather than of the value of edits to the harness\. Unlike meta\-agents such as ADAS\[[6](https://arxiv.org/html/2610.21218#bib.bib11)\]and the Darwin Gödel Machine\[[21](https://arxiv.org/html/2610.21218#bib.bib10)\], which are given hundreds of iterations to rewrite their harnesses, AutoOnboarder is only given 12\. With so few iterations and so strict a cost budget, it is possible that the Onboarder simply did not have enough room to experiment with the harness lever sufficiently\.

Together, these observations suggest that the Onboarder’s free\-form, contract\-driven interface is itself a bottleneck\. Reconciling these issues likely requires a more robust interface for the Onboarder that provides it more tools and forces it to consolidate its reasoning, much like that designed for Executor agents in SWE\-Agent\[[18](https://arxiv.org/html/2610.21218#bib.bib15)\]and AgentOccam\[[19](https://arxiv.org/html/2610.21218#bib.bib9)\]\. Such an interface would enforce the Onboarder’s rules rather than leave them open to its interpretation of a written contract\.

##### The Onboarder requires controlled, not absent, internet access\.

To keep the system secure, the Onboarder has no network access\. This isolation, though, also limits what the Onboarder can teach\. Perfect submissions require exact ICD\-10 codes \(a clinical diagnosis standard\) drawn from a relatively wide pool\. Without access to the internet, the Onboarder often explicitly acknowledges that it has no way to package an ICD database for the Executor\. We therefore believe that much of the residual imperfection may be a missing\-tooling problem, rather than one baked into the prompt, model, or harness\. However, unrestricted internet access would make it easy for an Onboarder to reward hack, for example by seeking out answers to specific cases instead of teaching general skills\. We therefore argue for controlled access: an allowlisted, logged, read\-only channel, such as a proxy or curated mirrors of resources like ICD\-10 tables and model and provider documentation, that sits inside the existing boundaries of HMAC\-signed submissions, filesystem permissions, and validation data isolated on VM2\.

The primary operational limitations of this study stem from the time and cost of the experiments, which reduce the number of repeat experiments we can perform: a single case can take over 30 minutes, and evaluating a model like Gemini 3\.5 Flash on the 200\-case set takes around$190\\$190\.

Interestingly, in our experiments, Gemini 3\.5 Flash submits only 6 of 180 cases and wrongly withholds 38 of the 39 valid ones, which conflicts with the automation bias claim of Nie et al\.\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]\. We hypothesize that the differences lie primarily in closed\-model drift, whereby commercial LLM providers silently change LLM behavior over time\. This is also a salient focus in our study: as with any research on closed LLMs, it is possible that some findings in this work do not generalize to newer or tweaked models\.

## 5Conclusion

In this work, we set out to develop an autonomous system that would adapt an agent and its harness to a task, thereby recovering the losses in critical thinking skills documented by Nie et al\.\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]\. Current experiments testify to the potential of AutoOnboarder in reducing Executor agents’ costs while maintaining or improving their scores\. However, current experiments are also prone to large variance and subpar performance\.

In the future, we plan a more robust system where the Onboarder has an interface that forces more structured thinking and includes all the tools it needs, while also balancing this freedom with data security\. We hypothesize that such a system would both reduce the large variance of Onboarding runs and contribute to better Executor performance\.

## 6Key Takeaways

In this work, we show that Onboarding has the potential to adapt AI agents to be more reliable and cost\-effective, making them more practical in production\. In the future, agentic systems like AutoOnboarder can drive the creation of small, cheap agents that take high\-volume busy work, like prior authorization forms, off of human hands\. However, this work also demonstrates the difficulties in creating such a system: the same Onboarding procedure, run twice, can produce very different agents, and no candidate is yet dependable enough for real clinical use\. Moving forward, we need to develop a more repeatable, robust method for holistic agent\-optimizing systems that prioritizes consolidated thinking and more closely balances security with access to information\.

## Acknowledgements

We thank Mengshu Nie for building the prior\-authorization sandbox that this work extends, and the rest of CliGenAILab, especially Dr\. Kamalakkannan Ravi for his help in proposing this project\. Finally, we thank the Center for Excellence in Education, the Research Science Institute, and the Massachusetts Institute of Technology for making this research possible\.

## References

- \[1\]American Medical Association\(2022\)2021 AMA prior authorization \(PA\) physician survey\.External Links:[Link](https://www.ama-assn.org/system/files/prior-authorization-survey.pdf)Cited by:[§2\.2](https://arxiv.org/html/2610.21218#S2.SS2.p1.1)\.
- \[2\]Anthropic\(2025\)Claude code: an agentic coding tool\.External Links:[Link](https://claude.com/claude-code)Cited by:[§3\.1](https://arxiv.org/html/2610.21218#S3.SS1.p1.1)\.
- \[3\]M\. Chen Y\. Liet al\.\(2024\)AutoManual: generating instruction manuals by LLM agents via interactive environmental learning\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Pwl9n4zlf5)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[4\]Y\. Fu D\. Kimet al\.\(2024\)AutoGuide: automated generation and selection of context\-aware guidelines for large language model agents\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,\(en\)\.External Links:[Link](https://openreview.net/forum?id=mRIQz8Zd6O)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[5\]E\. J\. Hu Y\. Shenet al\.\(20222021\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p2.1)\.
- \[6\]S\. Hu C\. Luet al\.\(2025\)Automated design of agentic systems\.InThe Thirteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=t9U3LW7JVX),[Link](https://openreview.net/forum?id=t9U3LW7JVX)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p5.1),[§4](https://arxiv.org/html/2610.21218#S4.SS0.SSS0.Px4.p2.1)\.
- \[7\]O\. Khattab A\. Singhviet al\.\(2024\)DSPy: compiling declarative language model calls into self\-improving pipelines\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[8\]M\. Levy A\. Jacobyet al\.\(2024\)Same task, more tokens: the impact of input length on the reasoning performance of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 15339–15353\.External Links:[Link](https://aclanthology.org/2024.acl-long.818/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.818)Cited by:[§1\.1](https://arxiv.org/html/2610.21218#S1.SS1.p2.1)\.
- \[9\]N\. F\. Liu K\. Linet al\.\(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§1\.1](https://arxiv.org/html/2610.21218#S1.SS1.p2.1)\.
- \[10\]Browser use: enable ai to control your browserExternal Links:[Link](https://github.com/browser-use/browser-use)Cited by:[3rd item](https://arxiv.org/html/2610.21218#S2.I1.i3.p1.1)\.
- \[11\]M\. Nie C\. Liuet al\.\(2026\)Hard to Halt: Automation Bias in Agent\-Driven Sequencing Prior Authorization Workflows\.medRxiv\.External Links:[Link](https://www.medrxiv.org/content/early/2026/06/18/2026.06.16.26355782),[Document](https://dx.doi.org/10.64898/2026.06.16.26355782)Cited by:[Appendix Appendix A](https://arxiv.org/html/2610.21218#A1.p1.1),[§1\.1](https://arxiv.org/html/2610.21218#S1.SS1.p3.1),[§1\.3](https://arxiv.org/html/2610.21218#S1.SS3.p1.1),[§2\.2](https://arxiv.org/html/2610.21218#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.21218#S2.SS2.p5.1),[§3\.1](https://arxiv.org/html/2610.21218#S3.SS1.p1.1),[§4](https://arxiv.org/html/2610.21218#S4.SS0.SSS0.Px5.p3.1),[§5](https://arxiv.org/html/2610.21218#S5.p1.1)\.
- \[12\]R\. Pryzant D\. Iteret al\.\(20232023\)Automatic prompt optimization with “gradient descent” and beam search\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,\(en\)\.External Links:[Link](https://openreview.net/forum?id=WRYhaSrThy&noteId=ULUK7hDpB4),[Link](https://openreview.net/forum?id=WRYhaSrThy)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[13\]F\. Shi X\. Chenet al\.\(2023\)Large language models can be easily distracted by irrelevant context\.InProceedings of the 40th International Conference on Machine Learning,pp\. 31210–31227\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v202/shi23a.html)Cited by:[§1\.1](https://arxiv.org/html/2610.21218#S1.SS1.p2.1)\.
- \[14\]N\. Shinn F\. Cassanoet al\.\(20232023\)Reflexion: language agents with verbal reinforcement learning\.InThirty\-seventh Conference on Neural Information Processing Systems,\(en\)\.External Links:[Link](https://openreview.net/forum?id=vAElhFcKW6),[Link](https://openreview.net/forum?id=vAElhFcKW6)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[15\]H\. S\. Smith R\. Franciskovichet al\.\(2021\)Outcomes of prior authorization requests for genetic testing in outpatient pediatric genetics clinics\.Genetics in Medicine: Official Journal of the American College of Medical Genetics23\(5\),pp\. 950–955\(eng\)\.External Links:ISSN 1530\-0366,[Document](https://dx.doi.org/10.1038/s41436-020-01081-x)Cited by:[§2\.2](https://arxiv.org/html/2610.21218#S2.SS2.p1.1)\.
- \[16\]H\. Su R\. Sunet al\.\(2025\)Learn\-by\-interact: a data\-centric framework for self\-adaptive agents in realistic environments\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3UKOzGWCVY)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[17\]C\. Yang X\. Wanget al\.\(2024\)Large language models as optimizers\.InThe Twelfth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=Bb4VGOWELI&noteId=lEn6qgvQgY)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[18\]J\. Yang C\. Jimenezet al\.\(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 50528–50652\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html),[Document](https://dx.doi.org/10.52202/079017-1601)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p4.1),[§4](https://arxiv.org/html/2610.21218#S4.SS0.SSS0.Px4.p3.1)\.
- \[19\]K\. Yang Y\. Liuet al\.\(2025\)AgentOccam: a simple yet strong baseline for llm\-based web agents\.InThe Thirteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=oWdzUpOlkX),[Link](https://openreview.net/forum?id=oWdzUpOlkX)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p4.1),[§4](https://arxiv.org/html/2610.21218#S4.SS0.SSS0.Px4.p3.1)\.
- \[20\]M\. Yuksekgonul F\. Bianchiet al\.\(2025\)Optimizing generative AI by backpropagating language model feedback\.Nature639\(8055\),pp\. 609–616\(en\)\.External Links:ISSN 1476\-4687,[Link](https://www.nature.com/articles/s41586-025-08661-4),[Document](https://dx.doi.org/10.1038/s41586-025-08661-4)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[21\]J\. Zhang S\. Huet al\.\(2026\)Darwin gödel machine: open\-ended evolution of self\-improving agents\.InThe Fourteenth International Conference on Learning Representations,\(en\)\.External Links:[Link](https://openreview.net/forum?id=pUpzQZTvGY),[Link](https://openreview.net/forum?id=pUpzQZTvGY)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p5.1),[§4](https://arxiv.org/html/2610.21218#S4.SS0.SSS0.Px4.p2.1)\.
- \[22\]A\. Zhao D\. Huanget al\.\(2024\)ExpeL: llm agents are experiential learners\.InProceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence,AAAI’24/IAAI’24/EAAI’24\.External Links:ISBN 978\-1\-57735\-887\-9,[Link](https://doi.org/10.1609/aaai.v38i17.29936),[Document](https://dx.doi.org/10.1609/aaai.v38i17.29936)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.
- \[23\]Y\. Zhou A\. I\. Muresanuet al\.\(2023\)Large language models are human\-level prompt engineers\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=92gvk82DE-)Cited by:[§1\.2](https://arxiv.org/html/2610.21218#S1.SS2.p3.1)\.

## Appendix Appendix AError Classes

Each case corresponds to a single name, which is passed to the Executor inside of its prompt\. The dataset in this paper expands the dataset of\[[11](https://arxiv.org/html/2610.21218#bib.bib6)\]from five deficiency signatures to 23 \([Section2\.2](https://arxiv.org/html/2610.21218#S2.SS2)\)\. Percentages below are shares of all 180 validation cases\. Classes P and Dsec require submission; the 23 deficient classes require withholding and reporting the specific problem\. Error classes can be split into data\-integrity \(T\) issues in transcription fields, clinical necessity \(C\) issues in the clinical note, and cross\-field issues \(X\) that have contradictions between transcribed fields and the clinical note\. Clinical notes were produced with OpenAI’s GPT\-5\.6 Terra with no reasoning\. We define siblings as classes in the same subgroup\. For example, T3a and T3b are siblings, while T6 has no siblings\.

GroupClass: cases \(% of dataset\)DefinitionValidP: 27 \(15\.0%\)Valid, well\-supported request\.39 \(21\.7%\)Dsec: 12 \(6\.7%\)Valid request with unrelated secondary details; should still submit\.Data\-integrity\(T\)T1a: 4 \(2\.2%\);T1b: 5 \(2\.8%\)Subscriber listed as a parent is implausibly close in age to the patient, or born after the patient\.53 \(29\.4%\)T2: 6 \(3\.3%\)Specimen collection date precedes the patient’s date of birth\.T3a: 6 \(3\.3%\);T3b: 6 \(3\.3%\)Two patients share an identical name \(and, for T3b, date of birth\), making the case ambiguous\.T4a: 5 \(2\.8%\);T4b: 5 \(2\.8%\)Provider or lab NPI, or provider phone or fax, is not valid\.T5a: 2 \(1\.1%\);T5b: 2 \(1\.1%\);T5c: 2 \(1\.1%\);T5d: 2 \(1\.1%\);T5e: 2 \(1\.1%\);T5f: 2 \(1\.1%\)Required element is missing: provider NPI, lab NPI, provider address, provider phone, subscriber date of birth, or collection date, respectively\.T6: 4 \(2\.2%\)Member ID is malformed\.