Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs

# Action-Level Reliability of Clinical LLM Agents Under Repeated Runs
Source: [https://arxiv.org/html/2609.13582](https://arxiv.org/html/2609.13582)
\\mlhtrack

proceedings\\jmlrauthorsBellibatlu, Singh, Han and Zhang

## Same Patient, Different Order: Action\-Level Reliability of Clinical LLM Agents Under Repeated Runs

Manpreet Singh[https://orcid.org/0000-0003-2368-2377](https://orcid.org/0000-0003-2368-2377)2Zhoutian Han3and Wenbin Zhang[https://orcid.org/0000-0003-3024-5415](https://orcid.org/0000-0003-3024-5415)1 1Florida International University, Miami, FL, USA 2Boston University, Boston, MA, USA 3Stevens Institute of Technology, Hoboken, NJ, USA \*Corresponding author:[rohithreddybc@gmail\.com](mailto:[email protected])

###### Abstract

A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run\. Such agents order tests, request medications and place referrals, yet benchmarks typically score one run per task and rarely ask whether identical inputs produce identical actions; MedAgentBench, the benchmark we use, scores a single attempt and says so\. To measure this gap we introduce*same\-input rerun*, which replays a task with every input held fixed and compares the orders rather than the score, with six reliability metrics, and apply it to 1000 MedAgentBench runs across 50 tasks from its five write\-capable families, two open\-weight models below ten billion parameters quantised to four bits, and two temperatures\. The study establishes that action\-level divergence exists and can pass unrecorded by the score, not that any rate generalises\. Under the 8B model at temperature 0\.7, all 43 ordering groups emit a different set of orders across five identical runs, 26 emit the order on some runs and not others, and 28 record a different coded value, dose or analyte\. In 22 of those 43 the benchmark reports the same failing verdict for materially different behaviour, as it does for all 10 divergent groups of the 4B model at 0\.7\. Orders also reach different endpoints across runs, one of which the record server rejects while the agent is told it succeeded\. These findings motivate repeated\-run evaluation, action\-level stability reporting and execution\-faithful environment feedback in clinical\-agent benchmarks\.

††proceedings:: Preprint###### keywords

clinical agents, evaluation reliability, benchmark design, run\-to\-run variability, electronic health records, reproducibility

### Data and Code Availability\.

The harness, the analysis code, the environment reproduction script and all 1250 run records, the grid and the eight\-bit arm alike, are at[https://github\.com/rohithreddybc/clinical\-agent\-action\-reliability](https://github.com/rohithreddybc/clinical-agent-action-reliability)\. An anonymised read\-only mirror of the same commit is served at[https://anonymous\.4open\.science/r/clinical\-agent\-action\-reliability](https://anonymous.4open.science/r/clinical-agent-action-reliability); it carries the identical file tree, so either address reaches the artefact if the other becomes unavailable\. The records include full transcripts, so every number here is recomputable by the supplied scripts\. The patient records are MedAgentBench’s own, described by its authors as real cases de\-identified at source, timestamps jittered and identifiers regenerated\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)\. We republish neither the record store nor any patient data beyond what the environment returned into a transcript, which a run record must keep if the agent’s behaviour is to be audited at all\.

### Institutional Review Board \(IRB\)\.

We collected no data\. The records are those distributed with MedAgentBench, which its authors report as de\-identified at source, timestamps jittered and identifiers regenerated\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)\. We received them in that form, attempted no re\-identification and computed locally, so the work is secondary analysis of an existing de\-identified dataset rather than human subjects research, and no IRB review was sought\.

### Use of AI Tools\.

The research question, the study design, the six metric definitions of Section[4](https://arxiv.org/html/2609.13582#S4)and every claim made here are the authors’, who take full responsibility for the content\. Working to those specifications, an AI assistant implemented much of the harness and the analysis code\. That work is verified rather than trusted: every number is regenerated from the run records by the released scripts, an automated gate re\-derives each table and fails if the manuscript disagrees, and the authors re\-derived the metric definitions against the FHIR payloads and confirmed each discordance count andppby hand\.

one task, one patient, inputs held fixedrun 3run 2run 1run 4run 5∙\\bullet∙\\bullet∘\\circ∙\\bullet–∙\\bulletorder to the typed endpoint∘\\circsame order, other endpoint–no order at allcompare the five runs against each otherorders: same payload, same endpoint, emitted at all?trajectory: at which turn do the runs first differ?verdict: does the benchmark’s pass/fail change?Figure 1:The rerun protocol, with an illustrative outcome\. Inputs are held fixed, so any difference between runs belongs to the agent, and the three comparison layers come apart in practice\.
## 1Introduction

Clinical language\-model agents no longer only answer questions\. They retrieve patient data, order tests, draft prescriptions, and issue write requests against the record system\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1);[Ferber and others, 2026](https://arxiv.org/html/2609.13582#bib.bib3);[Liu and others, 2026](https://arxiv.org/html/2609.13582#bib.bib4)\), and benchmarks have followed them into interactive environments\([Schmidgall and others, 2024](https://arxiv.org/html/2609.13582#bib.bib2)\)\. The reporting convention has not followed: these benchmarks report accuracy, and mostly from a single run per task\. For question answering the cost is precision, and it is documented, since repeated evaluation of the same model on the same items produces different scores and rankings built from single runs invert under replication\([Atil and others, 2024](https://arxiv.org/html/2609.13582#bib.bib5);[Madaan and others, 2024](https://arxiv.org/html/2609.13582#bib.bib6);[Alvarado Gonzalez and others, 2025](https://arxiv.org/html/2609.13582#bib.bib8)\)\.

For an agent that writes orders, something else goes missing\. Accuracy records whether the agent was right, not whether it was consistent, and only one of those is reported\. The same objection has been raised for static clinical decision support, cleared on one aggregate figure that says nothing about reliability under shift\([Bellibatlu et al\., 2026b](https://arxiv.org/html/2609.13582#bib.bib15)\)\. Agents inherit that problem and add one of their own, because what they produce is an action\.

This paper measures consistency\. We take MedAgentBench\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\), a clinical agent benchmark built on a FHIR\-compliant record environment, and replay each task five times with every input held fixed\. Instead of asking whether the verdict changed, we compare what the agent did: which resource it posted, where, and whether it posted at all\. What we propose follows directly: that clinical agent benchmarks rerun each task and report how often the action changes, that they report an action\-level stability measure beside accuracy, and that their environments answer as the record server would rather than confirming whatever parses\.

Doing this is harder than rerunning a benchmark more times, and three difficulties shape everything that follows\.\(i\) Deciding when two orders are the same order\.Runs that differ textually can be clinically identical, since timestamps, a workflow status field and the order of query parameters all vary without the order varying\. Comparing raw transcripts inflates divergence and comparing only the verdict erases it, so what is needed is a rule for when two requests are the same order, and every choice inside that rule moves the number\.\(ii\) Judging an action the environment never carries out\.MedAgentBench parses the agent’s write request, replies that it succeeded, and does not write, so nothing inside a run separates an order the record server would have created from one it would have refused; settling that means leaving the benchmark loop and putting the requests to the server itself\.\(iii\) Reporting a rate that still means something outside the study that produced it\.A rate measured over five reruns is not the rate over three, groups that never order have to stay out of the denominator, and the groups that remain are not independent\. Sections[4](https://arxiv.org/html/2609.13582#S4),[6](https://arxiv.org/html/2609.13582#S6)and[7](https://arxiv.org/html/2609.13582#S7)take these in turn\.

Three findings follow\. Action\-level divergence appears in every arm we measured, and order\-set divergence and omission both rise with decoding temperature in the two models we tested\. The mechanism is often mechanical: an order is routed to an endpoint that would reject it, or is lost after a malformed payload and a broken retry\. Not always, though\. In the sampling arm of the larger model 28 of 43 ordering groups record a different coded value, dose or analyte across identical runs, which no routing or parsing account explains\. And among the tasks that order, the verdict changes less often than the orders do in every arm\.

Taken together these amount to a way of designing and reporting clinical agent benchmarks, not only the code that implements it here\.A protocol\.Same\-input rerun makes the unit of evaluation a task group of identical reruns rather than one attempt, with the ordering groups as the denominator any action\-level rate is quoted over \(Section[4](https://arxiv.org/html/2609.13582#S4)\)\.A measurement layer beneath the verdict\.Six reliability metrics, four reading the order itself and two the run that produced it, each computable from transcripts a benchmark already keeps, and one reported in a form that does not depend on how many reruns were bought\.Evidence to match\.Interval estimates and exact paired tests on matched groups, a cluster bootstrap over families \(Section[7](https://arxiv.org/html/2609.13582#S7)\), and one metric checked against the record server rather than against our own transcripts \(Section[6](https://arxiv.org/html/2609.13582#S6)\)\.

## 2Background

### Variance in language model evaluation\.

[Atil and others \(2024\)](https://arxiv.org/html/2609.13582#bib.bib5)report accuracy variation across runs of models configured to be deterministic, which[Yuan and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib7)trace partly to floating\-point non\-associativity under changing batch sizes\.[Madaan and others \(2024\)](https://arxiv.org/html/2609.13582#bib.bib6)say when a difference is meaningful,[Alvarado Gonzalez and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib8)find single\-run leaderboards invert pairwise rankings often enough to justify repetition, and[Potamitis and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib9)characterise the variance structure of reasoning systems, while[Mustahsan and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib10)apply intraclass correlation to agentic evaluation on general\-purpose benchmarks\.

### Repeatability in medicine\.

Clinical work on this question is more developed than the agent literature\. Repeatability has been measured on board\-style examinations\([Krishna et al\., 2024](https://arxiv.org/html/2609.13582#bib.bib18)\), on unstructured clinical notes\([Shah, 2024](https://arxiv.org/html/2609.13582#bib.bib20)\), and on licensing and rare\-disease items, where[Shyr and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib11)separate repeatability from reproducibility and[Gürses and others \(2026\)](https://arxiv.org/html/2609.13582#bib.bib12)report that single\-trial accuracy hides both volatility and stable systematic error\. Closest to a decision with consequences,[Franc et al\. \(2024\)](https://arxiv.org/html/2609.13582#bib.bib19)measure both for a commercial model assigning emergency triage levels\.

### What is missing\.

Every one of those medical studies scores a single\-turn output: an examination answer, a triage level, a summary of a note\. The gap is not merely that none reruns an agent benchmark; it is that agreement rates and intraclass correlations describe a scalar or a string, not an order that went to the wrong endpoint or one that was never issued at all\. Several lines of work come closer\.passk\\mathrm\{pass\}^\{k\}, the probability that allkktrials of a task succeed\([Yao et al\., 2024](https://arxiv.org/html/2609.13582#bib.bib23)\), scores end states;[Gupta \(2026\)](https://arxiv.org/html/2609.13582#bib.bib25)pairs it with fault injection, and[Liu and others \(2026\)](https://arxiv.org/html/2609.13582#bib.bib4)applies that verdict\-level idea to a clinical agent\.[Yagubyan \(2026\)](https://arxiv.org/html/2609.13582#bib.bib24)does reach the calls, asking whether an agent selects the same tools with the same arguments across identical invocations, but outside medicine\. Closest of all,[Mokssit et al\. \(2026\)](https://arxiv.org/html/2609.13582#bib.bib26)rerun a clinical FHIR sandbox three times per task and execute its writes against a resettable server, but their run\-to\-run variance is over task success: the payloads and field values are never compared\. What none compares is the order itself across identical runs: its payload, its destination, and the verdict returned for it\. Section[4](https://arxiv.org/html/2609.13582#S4)defines four action\-level quantities that do, and Section[7](https://arxiv.org/html/2609.13582#S7)measures them\.

The same shift, from grading what an agent says to grading what it does, has been argued for fairness\([Morla et al\., 2026](https://arxiv.org/html/2609.13582#bib.bib16)\); reliability has not had the equivalent treatment\. Automated judges also move under rephrased prompts\([Bellibatlu et al\., 2026a](https://arxiv.org/html/2609.13582#bib.bib17)\), though our grader is a fixed reference implementation rather than a model, and adjacent work finds clinical agents acting without noticing patient\-identity errors\([Klang and others, 2026](https://arxiv.org/html/2609.13582#bib.bib22)\)and benchmark scores diverging from deployment behaviour\([Agrawal and others, 2025](https://arxiv.org/html/2609.13582#bib.bib13);[Mehandru and others, 2024](https://arxiv.org/html/2609.13582#bib.bib14)\)\.

## 3Environment

### MedAgentBench\.

MedAgentBench\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)provides 300 clinically derived tasks in ten families, a FHIR\-compliant record server holding records its authors describe as covering about one hundred patients, and a text protocol in which the agent issuesGETandPOSTrequests and ends withFINISH\. Five families can involve a write: one records an observation, one places a service request, and three are conditional on a laboratory value or its age, one of those three pairing a medication with a service request\. In the conditional families ordering nothing is sometimes correct, so a group there can pass without writing, though inputs are fixed within a group and the right answer is the same across its five runs\. We use all five, since only there does an action exist to diverge\.

A second version\([Chen and others, 2026](https://arxiv.org/html/2609.13582#bib.bib21)\)revises the agent scaffold and leaves the environment alone: a new system prompt, few\-shot examples, memory, and tools that let the agent act without constructing HTTP requests by hand, reaching 91% success with GPT\-4\.1\. Our measurements use the original scaffold, and Section[9](https://arxiv.org/html/2609.13582#S9)states what that does and does not imply\. The record server ships as a container image; lacking a container runtime we pulled the layers from the registry and ran the extracted HAPI FHIR server locally, releasing the script and the store’s resource counts \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\.

### How the benchmark handles writes\.

The benchmark never forwards an agent’s write to its record server, and says so: its authors “decide to only send GET requests to the environment” because it “takes around 90 seconds to start”, and on aPOSTthey “conduct a simple sanity check to make sure the payload data is JSON\-loadable, and indicate success of execution to the agent system”\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)\. The grader then scores from the transcript\. No state reset is needed between runs, which we confirmed: agents emitted 189ServiceRequestpayloads across the 1000 runs, and the store still held zero afterwards\. Two departures from the reference implementation remain: agent turns keep their message roles, andGETresponses are truncated at 6000 characters\. Neither touches the task set, the prompt or the grader \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\.

Table 1:Divergence across five identical reruns of each task\. Action quantities are computed over the task groups that emitted at least one order, since a task that never orders cannot diverge in its orders; trajectory and verdict quantities are over all groups, apart from the last row, which repeats verdict instability over the ordering groups; the mean pass rate is over the arm’s 250 runs\.*The columns are not a ranking\.*Each arm carries its own denominator, and we ran no cross\-model test\.

## 4Protocol and metrics

The request is what stays comparable across runs, and comparing requests is the whole of the protocol, which we call*same\-input rerun*\. Figure[1](https://arxiv.org/html/2609.13582#S0.F1)shows its shape: for a fixed task, model and decoding configuration the benchmark’s agent loop runs five times with every input identical, and the five runs are compared at three layers\. The task is its own control, so a difference cannot be attributed to the case, the prompt or the environment, none of which changed\.

Six quantities are reported\.

A task that never orders cannot diverge in its orders, so the four action\-level quantities are computed only over groups that emit at least one order in at least one run, the*ordering groups*\. The runs determine that set, so it differs by arm: Llama orders in 21 groups at 0\.05 and 43 under sampling\. Every action\-level rate carries its own arm’s denominator, and the paired tests below handle the problem differently\.

### Order\-set divergence\.

For each task group we count the distinct canonical order sets across its five runs, the canonical form being the destination endpoint together with the whole payload, compared as a set\. Nothing is discarded, the clock and workflow fields included, and no group in any arm diverges on those alone \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. A run that emitted nothing contributes the empty set, so an omitted order counts as a difference here, where endpoint and clinical\-content divergence exclude it\. A group is divergent if that count exceeds one\. Section[6](https://arxiv.org/html/2609.13582#S6)shows why the endpoint is part of the action rather than a detail of how it was written\.

### Omission split\.

A group is split if at least one run emitted an order and at least one emitted none; we report the proportion of ordering groups that are\. This is the sharpest of the six: the failure it names is silent, nothing in the transcript announcing a missing order\. In the conditional families a split can equally be an order placed where none was called for; we do not separate the two\.

### Endpoint divergence\.

For each run that emitted we take the set of endpoints its orders reached, the endpoint being the final path segment of the request URL, and a group diverges if two such runs disagree\. Runs that emitted nothing are excluded rather than entered as empty sets, so dropping an order cannot by itself make a group divergent and this measure is not determined by the omission split, though a group may do both and omission censors it, since a group left with one emitting run can never be counted divergent \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. It is reported separately because a destination error and a payload error need different fixes\.

### Trajectory first\-divergence\.

The index of the first agent turn at which runs stop agreeing, reported as a distribution over diverging groups and, in Table[1](https://arxiv.org/html/2609.13582#S3.T1), as the proportion of groups that diverge at all\. Query parameters are sorted first, since?patient=X&code=Yand?code=Y&patient=Xare the same request; that correction changes at most one group per arm \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\.

### Clinical\-content divergence\.

Two runs may emit the same number of orders to the same endpoint and still order different things\. For every pair of runs that both emitted we compare the fields carrying clinical content: the resource code and the resource’s owntextelement, the recorded value, the medication code and its text, and the dose value, unit, rate and route\. A group diverges if any of those differs, and we report the proportion of ordering groups that do\. Timestamps and workflow status are excluded, since they record when a payload was written and in what state rather than what was ordered, as are runs that emitted nothing, so the omission split cannot account for this measure\. The human\-readable label a coding carries alongside its code is also excluded, since it names the code rather than changing it\. That exclusion is load\-bearing, and Appendix[B](https://arxiv.org/html/2609.13582#A2)gives its cost: counting the label would put this measure above zero in three arms rather than one\. The comparison is otherwise syntactic, so two clinically equivalent codings would count as divergent, which can only inflate this rate\.

### Verdict instability\.

A group is unstable if the benchmark’s own pass/fail judgement is not identical across its five runs; we report the proportion over all groups and, separately, over ordering groups, so it can be compared with the action\-level rates on a matched denominator\. This is what a leaderboard reports, included so it can be measured against the other five rather than assumed to stand for them\. It is also, with trajectory, a quantity a group can move without ordering anything, by changing the answer it reports\.

### A rate that survives a change of budget\.

Each of the four action\-level quantities asks whether a group diverged at least once in five runs, and that question is monotone in the rerun budget: run the same agent ten times and the rate can only rise, so a figure measured at five runs cannot be set beside one measured at three\. We therefore report order\-set divergence a second way, as*pairwise disagreement*: the share of run pairs within a group whose order sets differ, averaged over the ordering groups\. For exchangeable runs it estimates the same quantity at any budget of two or more, though which groups enter the average still depends on that budget\. The at\-least\-once form says whether a task ever went wrong; the pairwise form is the one another study can be read against\.

Proportions carry Wilson 95% intervals, which remain defined at these denominators and at zero or one\. Between the two temperatures of a model we use an exact paired McNemar test and state the discordant counts, since at these sizes a non\-significant result is not evidence of similarity\. The tests run over all 50 task groups rather than the ordering groups, because a group that orders in one arm and not the other is informative about the temperature effect and dropping it would condition on an outcome\. A group that never orders in either arm is concordant by construction on order\-set divergence, though not on trajectory or verdict, where it can still move \(Section[7](https://arxiv.org/html/2609.13582#S7)\)\. Appendix[A](https://arxiv.org/html/2609.13582#A1)gives each test’s discordant counts and denominator\. The corrected family is one quantity per comparison layer per model, six in all; the other three action rows decompose order\-set divergence rather than standing beside it, so they are read descriptively and carry no correctedpp\. Task groups cluster into families rather than being independent, so we bootstrap over the families that contain ordering groups: five for Llama under sampling, three at its near\-greedy setting, and two in each Qwen arm\. With two or three clusters the interval is little more than the range of the family rates, and should be read as a robustness check rather than an estimate\.

## 5Experimental setup

The grid is small and fully crossed: two models, two decoding temperatures, and the same 50 tasks throughout, so that a difference between the two temperatures of a model cannot come from the task sample\. Differences between models are a separate matter, and Table[1](https://arxiv.org/html/2609.13582#S3.T1)says why we do not read them\.

### Models\.

Two open\-weight instruction\-tuned models, Qwen3\-4B and Llama\-3\.1\-8B, served locally and quantised to four bits \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. The panel is restricted to models an institution could run on its own hardware, the deployment mode available where patient data cannot be sent to a hosted API\. It is not a frontier panel \(Section[9](https://arxiv.org/html/2609.13582#S9)\)\.

### Decoding\.

MedAgentBench evaluates at temperature 0 and scores a single attempt per task, stating that it adoptspass@1because a single incorrect action can have significant consequences\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)\. We run each model at 0\.05, a near\-greedy setting, and at 0\.7 with nucleus sampling\. Neither is the benchmark’s own value, and the choice is deliberate: the noise floor below shows greedy decoding on this stack to be exactly reproducible, so a study run at 0 would measure nothing here\.

That reproducibility belongs to the stack, and it does not survive a shared server\. We serve one request at a time on one local GPU; a deployment serving many clinicians batches them, and batch composition changes which floating\-point reductions run\([Atil and others, 2024](https://arxiv.org/html/2609.13582#bib.bib5);[Yuan and others, 2025](https://arxiv.org/html/2609.13582#bib.bib7)\)\. Reissuing the noise floor with other traffic in flight breaks determinism at temperature zero in both models: four of twelve greedy cells return more than one completion of 20, where every matched serial cell returned one \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. What moves is completion text rather than an order, but the benchmark’s own setting guarantees no run\-to\-run identity where a clinical agent would run, and a fixed seed does not close the gap, since seeding makes five runs one draw repeated\.

### Tasks\.

Each family holds 30 tasks; we take the first ten of each in the benchmark’s own identifier order, a deterministic prefix rather than a random sample, and run each five times, giving 250 runs per arm and 1000 in total\. The round limit of eight is the reference implementation’s, and the identifiers are released with the harness\.

### Controls\.

Nothing is trained here, so there is no baseline and no data split; the controls are on the instrument instead, the record\-server check of Section[6](https://arxiv.org/html/2609.13582#S6)and a noise floor\. Before attributing anything to the agent we measured what the serving stack contributes when nothing should vary: twenty identical calls per cell over eighteen cells \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. Greedy decoding here is exactly reproducible, seeded or not, so downstream variation belongs to the decoding configuration and not to infrastructure\. This replicates[Atil and others \(2024\)](https://arxiv.org/html/2609.13582#bib.bib5), who report the same determinism running a model on their own hardware without optimisations\.

## 6Is endpoint divergence cosmetic?

Endpoint divergence counts as an action difference only if the two destinations behave differently, and it is reasonable to ask whether posting to/fhir/withresourceTypein the body is equivalent to posting to/fhir/ServiceRequest, since some servers resolve the type from the payload\. The question is decidable against the same HAPI FHIR server the benchmark ships, so we decided it\. An identicalServiceRequestpayload sent to the typed endpoint returned201 Createdand the resource was created, which we then deleted; sent to the collection root it returned400 Bad RequestwithHAPI\-0450: Failed to parse request body as JSON resource\.

The two are therefore not interchangeable: one places the order and the other does not\. The benchmark’s paper tells its human readers that writes are not executed, but the agent reads only the runtime reply, which reports success for any payload the sanity check parses, whatever the destination\. The gap is not the absent write, which is disclosed and deliberate, but that the surface the agent acts on carries no trace of it\.

## 7Results

Order\-set divergence and the omission split rise with temperature in both models, and clinical\-content divergence appears in one arm only\. Endpoint divergence is the exception: for Llama it falls as a rate because the base grows faster than the count\. Figure[2](https://arxiv.org/html/2609.13582#S7.F2)plots three of those rows as the move from near\-greedy to sampling\.

Figure 2:Three of the four action\-level rows of Table[1](https://arxiv.org/html/2609.13582#S3.T1); clinical\-content divergence is zero in every arm but one\. Drawn as the movement fromT=0\.05T\{=\}0\.05\(open mark\) toT=0\.7T\{=\}0\.7\(solid\), with Wilson intervals\. Each arm’s denominator is its own ordering groups: 20 and 20 for Qwen, 21 and 43 for Llama\. Only Llama’s grows under sampling, which is why its endpoint arrow points left as the count rises from 8 to 12\.A group whose orders differ has always taken a different path to them, so trajectory divergence is the weaker condition: over the ordering groups it exceeds order\-set divergence in both Qwen arms and coincides with it in both Llama arms \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. Divergence is immediate rather than cumulative, the median first\-divergence index being zero in three arms of four and never exceeding four \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\.

### What the tests do and do not establish\.

Paired exact McNemar tests over all 50 task groups find the rise with temperature significant for both models on trajectory, order\-set and verdict divergence, with 96 of the 100 discordant pairs running toward more divergence and Holm leaving all six significant at 0\.047\. Two of the six need care\. Llama’s ordering base grows from 21 groups to 43, so most of the 33 groups divergent only at 0\.7 began ordering at all rather than became inconsistent; restricted to the 20 that order at both temperatures the effect holds, 10 discordant to none the other way \(p=0\.002p=0\.002\)\. And all seven discordant groups in Qwen’s verdict test never order in either arm, so what moves there is the answer the agent reports, not an order\. The tests also treat task groups as independent, which Section[4](https://arxiv.org/html/2609.13582#S4)says they are not\. Taking each family as one signed observation instead, no family in any test moves back, but only one to four move at all, and an exact sign test on that many cannot fall belowp=0\.125p=0\.125\. The direction is consistent at both units; the significance is not, and we rest nothing on it\. The pairwise row of Table[1](https://arxiv.org/html/2609.13582#S3.T1)carries the same rise in the form that does not grow with the rerun budget, 0\.08 to 0\.29 for Qwen and 0\.33 to 0\.78 for Llama\.

### A number belongs to a model and a configuration\.

Llama\-3\.1\-8B shows more order\-set divergence and more omission than Qwen3\-4B at both temperatures, which we do not read as a model difference: the rates overlap once clustered, and we ran no cross\-model test \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\.

### Two mechanisms are mechanical, and a third is not\.

Orders are misrouted, reaching the typed endpoint on some runs and the collection root on others, which Section[6](https://arxiv.org/html/2609.13582#S6)shows is the difference between a created order and a rejected one\. Orders are also lost: in three service\-request groups under Qwen sampling the referral is placed on four runs of five, while on the fifth the payload fails to parse and the retry repeats the same malformed construction, a protocol failure rather than a change of mind\. Nor is the loss the agent running out of turns: of 72 non\-emitting runs in Llama’s 26 split groups, 61 ended with the agent declaring itself finished, 7 on a malformed action and 4 at the round limit\.

The third is not\. In the Llama sampling arm 28 of the 43 ordering groups post a different value across identical runs: 23 differ on the resource code, 18 on a recorded value, 5 on the medication and 3 on the dose, categories that overlap\. The count covers only runs that emitted, so omission cannot explain it, and only fields carrying clinical content\. No other arm of the grid shows any, and Appendix[B](https://arxiv.org/html/2609.13582#A2)gives two worked instances\.

This measure is the one our design can least well attribute, appearing in one arm only, where four\-bit quantisation and sampling compound\. The eight\-bit Qwen rerun complicates a simple quantisation account, since content divergence appears there at eight bits and not at four, though in Qwen rather than the Llama arm that shows it here\. What we stand behind is that divergence reaches the content of an order, not only its envelope\.

### Orders move without the verdict moving\.

Restricting both to the ordering groups, what matters is the gap: groups whose orders differ across identical runs while the verdict holds\. In the Llama sampling arm 22 of the 43 divergent groups are of that kind, spanning all five write families; at the near\-greedy setting it is 3 of 11, and for Qwen under sampling 10 of 10\. That verdict is almost always a failure: of the 38 such groups across the four arms, 36 fail on all five runs, and the 2 that pass differ only in the label attached to an identical procedure code\. What the score conceals is therefore not success but that the agent failed differently each time; none of the 251 passing runs across the grid posted to a destination the server would refuse\.

The converse, that no verdict moves while the orders hold, is a property of the grader: in three of the five families it decides from the emitted request alone, so a verdict cannot move there unless the order set does, and 25 of the 29 ordering\-group verdict flips fall there\. We draw no conclusion in that direction\.

A natural objection is that this only measures agents too weak to do the task\. Llama at the near\-greedy setting diverges in all 10 ordering groups it ever passes against 1 of the 11 it never passes, but eight of those ten pass only on some runs, and in a family whose grader reads only the request a change of verdict and a change of order are one event rather than two pieces of evidence\. Two groups pass on all five runs and diverge anyway, but in the label rather than the code, so what reaches a task the agent completes every time is instability of form\. No group in this study both passes every run and changes what it ordered\. The objection therefore stands to this extent: we show that agents fail inconsistently, and showing that they succeed inconsistently would need a stronger panel\.

Order\-set divergence is not independent of the rows below it in Table[1](https://arxiv.org/html/2609.13582#S3.T1): a group counts as divergent if it omitted the order, sent it elsewhere or changed the payload, so the 43 of 43 is their union rather than a third finding, and Appendix[B](https://arxiv.org/html/2609.13582#A2)gives the decomposition\.

### The effect is not spread evenly, and thins under clustering\.

Table[3](https://arxiv.org/html/2609.13582#A3.T3)breaks the ordering groups down by family\. Qwen’s entire action\-level temperature effect sits in one family, its service requests going from 3 of 10 divergent to 10 of 10, while its observation tasks never diverge and its other three never order at all; Llama under sampling diverges in every family that orders, so concentration belongs to the quieter arms rather than to the measure\. Resampling families rather than groups then widens the intervals sharply: two arms collapse to\[0\.00,1\.00\]\[0\.00,1\.00\], Qwen at 0\.7 and Llama at 0\.05, Qwen at 0\.05 reaches\[0\.00,0\.30\]\[0\.00,0\.30\], and only the Llama sampling arm holds, at\[1\.00,1\.00\]\[1\.00,1\.00\], every one of its five ordering families diverging\. The study establishes that action divergence occurs and can pass unrecorded by the score, not that any rate generalises: a benchmark can report a stable number while the behaviour under it moves\.

## 8What this suggests for benchmark design

Three changes would make clinical agent benchmarks report what readers take them to report\. None is a research programme; each can be adopted on its own\.

### R1\. Rerun, and say so\.

MedAgentBench states its protocol: a single attempt per task, reasoning that one wrong action can matter clinically\([Jiang and others, 2025](https://arxiv.org/html/2609.13582#bib.bib1)\)\. That argues for reruns, since one attempt cannot show whether the wrong action recurs, and guidance for single\-turn evaluation already suggests at least two\([Alvarado Gonzalez and others, 2025](https://arxiv.org/html/2609.13582#bib.bib8)\)\.

### R2\. Report an action\-level stability measure\.

No clinical agent benchmark we read in detail reports one:[Jiang and others \(2025\)](https://arxiv.org/html/2609.13582#bib.bib1)scores a single attempt, while[Liu and others \(2026\)](https://arxiv.org/html/2609.13582#bib.bib4)and[Mokssit et al\. \(2026\)](https://arxiv.org/html/2609.13582#bib.bib26)rerun but score the outcome rather than the order\. Order\-set divergence and the omission split can be computed from transcripts a benchmark already stores, and we would report the first in its pairwise form, so that studies bought at different rerun budgets stay comparable\.

### R3\. Let the environment answer as the real system would\.

A request the FHIR server would reject is reported to the agent as executed, removing the signal it would need to recover\. The stated reason for not executing writes is the cost of re\-initialising the environment, which covers an accepted write; a refused one leaves nothing to reset, and the refusal would make misrouting visible to the agent and the score\. One clinical sandbox already executes writes against a resettable server\([Mokssit et al\., 2026](https://arxiv.org/html/2609.13582#bib.bib26)\)\.

## 9Limitations

### Emitted, not executed\.

We do not claim clinical agents are unsafe to deploy: the benchmark forwards no writes, so every statement here concerns the order issued, not the order received\.

### Scaffold, model panel and scope\.

The benchmark’s second version removes hand\-constructed HTTP requests as error\-prone\([Chen and others, 2026](https://arxiv.org/html/2609.13582#bib.bib21)\), the mechanism behind many failures here, so our rates describe the original scaffold, though a higher success rate need not imply action stability\. The panel is two open\-weight models below ten billion parameters, quantised to four bits, on one benchmark and one serving stack, and a prefix of each family rather than a random sample, which under\-represents harder tasks if difficulty rises with identifier order\. An eight\-bit rerun of the Qwen sampling arm separates part of this: the ten groups carrying its divergence diverge at eight bits too, and only the mode of failure changes \(Appendix[B](https://arxiv.org/html/2609.13582#A2)\)\. A frontier model might show none of it, though reaching one means a hosted API, while this panel is the deployment where records stay inside the institution\. The protocol transfers; the rates do not\.

## 10Conclusion

One run says whether the agent was right, not whether it is consistent\. A benchmark for clinical actions must rerun, measure the order as well as the verdict, and answer as the record system would\.

## References

- Agrawalet al\.\(2025\)M\. Agrawalet al\.The evaluation illusion of large language models in medicine\.npj Digital Medicine\.External Links:[Document](https://dx.doi.org/10.1038/s41746-025-01963-x)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p2.1)\.
- Alvarado Gonzalezet al\.\(2025\)M\. A\. Alvarado Gonzalezet al\.Do repetitions matter? strengthening reliability in LLM evaluations\.arXiv preprint arXiv:2509\.24086\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2509.24086)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1),[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px1.p1.1)\.
- Atilet al\.\(2024\)B\. Atilet al\.Non\-determinism of “deterministic” LLM settings\.arXiv preprint arXiv:2408\.04667\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2408.04667)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1),[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.13582#S5.SS0.SSS0.Px2.p2.1),[§5](https://arxiv.org/html/2609.13582#S5.SS0.SSS0.Px4.p1.1)\.
- Bellibatluet al\.\(2026a\)R\. R\. Bellibatlu, E\. Raff, and W\. ZhangJudgeSense: a benchmark for prompt sensitivity in LLM\-as\-a\-Judge systems\.arXiv preprint arXiv:2604\.23478\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.23478)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p2.1)\.
- Bellibatluet al\.\(2026b\)R\. R\. Bellibatlu, M\. Singh, Y\. Jajoo, S\. Lakhanpal, and A\. IsraniRISED: a pre\-deployment evaluation framework for high\-stakes AI decision\-support systems, with application to healthcare\.arXiv preprint arXiv:2605\.12895\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.12895)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p2.1)\.
- Chenet al\.\(2026\)E\. Chenet al\.MedAgentBench v2: improving medical LLM agent design\.InPacific Symposium on Biocomputing,Vol\.31\.External Links:[Document](https://dx.doi.org/10.1142/9789819824755%5F0025)Cited by:[§3](https://arxiv.org/html/2609.13582#S3.SS0.SSS0.Px1.p2.1),[§9](https://arxiv.org/html/2609.13582#S9.SS0.SSS0.Px2.p1.1)\.
- Ferberet al\.\(2026\)D\. Ferberet al\.Towards autonomous medical artificial intelligence agents\.Nature\.External Links:[Document](https://dx.doi.org/10.1038/s41586-026-10675-5)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1)\.
- Francet al\.\(2024\)J\. M\. Franc, L\. Cheng, A\. Hart, R\. Hata, and A\. HertelendyRepeatability, reproducibility, and diagnostic accuracy of a commercial large language model \(ChatGPT\) to perform emergency department triage using the Canadian triage and acuity scale\.Canadian Journal of Emergency Medicine26\(1\)\.External Links:[Document](https://dx.doi.org/10.1007/s43678-023-00616-w)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px2.p1.1)\.
- Gupta \(2026\)A\. GuptaReliabilityBench: evaluating LLM agent reliability under production\-like stress conditions\.arXiv preprint arXiv:2601\.06112\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.06112)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p1.1)\.
- Gürseset al\.\(2026\)Ö\. A\. Gürseset al\.Consistency over accuracy: run\-to\-run stability of contemporary large language models on turkish curriculum\-aligned theoretical anatomy multiple\-choice questions\.BMC Medical Education\.External Links:[Document](https://dx.doi.org/10.1186/s12909-026-08656-3)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px2.p1.1)\.
- Jianget al\.\(2025\)Y\. Jianget al\.MedAgentBench: a realistic virtual EHR environment to benchmark medical LLM agents\.arXiv preprint arXiv:2501\.14654\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2501.14654)Cited by:[Data and Code Availability\.](https://arxiv.org/html/2609.13582#S0.SS0.SSS0.Px1.p1.1),[Institutional Review Board \(IRB\)\.](https://arxiv.org/html/2609.13582#S0.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2609.13582#S1.p1.1),[§1](https://arxiv.org/html/2609.13582#S1.p3.1),[§3](https://arxiv.org/html/2609.13582#S3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2609.13582#S3.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.13582#S5.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px2.p1.1)\.
- Klanget al\.\(2026\)E\. Klanget al\.Clinical agents fail silently on patient identity\.International Journal of Medical Informatics218,pp\. 106514\.External Links:[Document](https://dx.doi.org/10.1016/j.ijmedinf.2026.106514)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p2.1)\.
- Krishnaet al\.\(2024\)S\. Krishna, N\. Bhambra, R\. Bleakney, and R\. BhayanaEvaluation of reliability, repeatability, robustness, and confidence of GPT\-3\.5 and GPT\-4 on a radiology board\-style examination\.Radiology311\(2\)\.External Links:[Document](https://dx.doi.org/10.1148/radiol.232715)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)R\. Liuet al\.PhysicianBench: evaluating LLM agents in real\-world EHR environments\.arXiv preprint arXiv:2605\.02240\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2605.02240)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1),[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px2.p1.1)\.
- Madaanet al\.\(2024\)L\. Madaanet al\.Quantifying variance in evaluation benchmarks\.arXiv preprint arXiv:2406\.10229\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2406.10229)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1),[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1)\.
- Mehandruet al\.\(2024\)N\. Mehandruet al\.Evaluating large language models as agents in the clinic\.npj Digital Medicine\.External Links:[Document](https://dx.doi.org/10.1038/s41746-024-01083-y)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p2.1)\.
- Mokssitet al\.\(2026\)Y\. Mokssit, K\. Ravi, M\. Nie, J\. Kim, and C\. LiuFHIR\-AgentEval: a modular sandbox for benchmarking clinical LLM agents with an evaluation of memory\-augmented configurations\.Research Square\.Note:PreprintExternal Links:[Document](https://dx.doi.org/10.21203/rs.3.rs-8746188/v1)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2609.13582#S8.SS0.SSS0.Px3.p1.1)\.
- Morlaet al\.\(2026\)T\. Morla, R\. R\. Bellibatlu, M\. Singh, and M\. S\. KapoorAgentFairBench: do LLM agents discriminate when they act?\.arXiv preprint arXiv:2606\.16723\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2606.16723)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p2.1)\.
- Mustahsanet al\.\(2025\)Z\. Mustahsanet al\.Stochasticity in agentic evaluations: quantifying inconsistency with intraclass correlation\.arXiv preprint arXiv:2512\.06710\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2512.06710)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1)\.
- Potamitiset al\.\(2025\)N\. Potamitiset al\.ReasonBENCH: benchmarking the \(in\)stability of LLM reasoning\.arXiv preprint arXiv:2512\.07795\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2512.07795)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1)\.
- Schmidgallet al\.\(2024\)S\. Schmidgallet al\.AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments\.arXiv preprint arXiv:2405\.07960\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2405.07960)Cited by:[§1](https://arxiv.org/html/2609.13582#S1.p1.1)\.
- Shah \(2024\)S\. V\. ShahAccuracy, consistency, and hallucination of large language models when analyzing unstructured clinical notes in electronic medical records\.JAMA Network Open7\(8\)\.External Links:[Document](https://dx.doi.org/10.1001/jamanetworkopen.2024.25953)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px2.p1.1)\.
- Shyret al\.\(2025\)C\. Shyret al\.A statistical framework for evaluating the repeatability and reproducibility of large language models\.medRxiv\.External Links:[Document](https://dx.doi.org/10.1101/2025.08.06.25333170)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px2.p1.1)\.
- Yagubyan \(2026\)A\. YagubyanHow consistent are LLM agents? measuring behavioral reproducibility in multi\-step tool\-calling pipelines\.arXiv preprint arXiv:2605\.28840\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.28840)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p1.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhanτ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2406.12045)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px3.p1.1)\.
- Yuanet al\.\(2025\)J\. Yuanet al\.Understanding and mitigating numerical sources of nondeterminism in LLM inference\.InAdvances in Neural Information Processing Systems,External Links:[Document](https://dx.doi.org/10.52202/085713-5653)Cited by:[§2](https://arxiv.org/html/2609.13582#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.13582#S5.SS0.SSS0.Px2.p2.1)\.

## Appendix APaired\-test discordance tables

Table 2:The six paired tests, with the discordance behind each\.bbcounts groups showing the measure only atT=0\.7T\{=\}0\.7andcconly atT=0\.05T\{=\}0\.05; the exact two\-sided McNemarppusesbbandccalone\. All six run over the 50 task groups\. A group that never orders in either arm is concordant by construction on order\-set divergence, but not on the other two\.
## Appendix BImplementation details

### Serving stack\.

Both models are served through Ollama at four\-bit quantisation \(q4\_K\_M\) on a single consumer GPU with an 8192\-token context window\. The exact tags areqwen3:4b\-instruct\-2507\-q4\_K\_Mandllama3\.1:8b\-instruct\-q4\_K\_M; the quantisation suffix is part of the tag, and pulling either name without it gives a different build\. Sampling runs use nucleusp=0\.9p=0\.9\. The round limit of eight turns is that of the reference implementation\.

### Why the harness departs from the reference implementation\.

Agent turns are sent as multi\-turn messages with roles preserved\. An earlier version flattened the dialogue into a single prompt, under which nearly every task failed on output formatting before reaching the reasoning it was meant to test, which would have measured the harness rather than the agent\.GETresponses are truncated to 6000 characters because unbounded FHIR bundles exhaust the context window of a locally served model; truncation fired on 18 and 11 of 250 runs in the Qwen arms and 26 and 10 of 250 in the Llama arms, and is recorded in every run record\. Truncation bounds rather than explains the omissions of Section[7](https://arxiv.org/html/2609.13582#S7): of the 72 non\-emitting runs in the 26 omission\-split groups under Llama sampling, 7 saw a truncated response, against 2 of the 58 runs in those groups that did emit\. Truncation is thus about three times as frequent where the order went missing, and could at most account for 7 of the 72; the remaining 65 non\-emitting runs saw the response in full\. Those seven are among the runs that finished, not the seven that ended on a malformed action\.

### What order\-set divergence is made of\.

In the Llama sampling arm 26 of the 43 divergent groups diverge through omission and 12 through misrouting, overlapping on 5, so those two cover 33; the remaining 10 differ in payload alone, every one on an observation code or a recorded value\. The five in the overlap drop the order on some run and split the destination across the runs that remain\.

### The coding label\.

Clinical\-content divergence compares the code a resource carries, not the display string beside it: two runs that both post SNOMED 306181000000106, one labelling it*Orthopaedic surgical referral*and the other*Orthopedic surgery referral*, have ordered the same procedure\. Counting the label as content would raise the measure from none to 10 of 21 in the Llama near\-greedy arm, from 28 to 33 of 43 under sampling, and from none to 1 of 20 for Qwen at 0\.7\. The finding that content divergence is confined to one arm is therefore a finding about what was ordered, not about how it was labelled, and the two groups of Section[7](https://arxiv.org/html/2609.13582#S7)that pass every run while their orders differ are label differences of exactly this kind\.

### The eight\-bit arm\.

To separate the agent’s contribution from the four\-bit quantisation, the Qwen sampling arm was rerun asqwen3:4b\-instruct\-2507\-q8\_0with everything else held fixed: the same 50 task groups, the same temperature and nucleus setting, the same 8192\-token context and the same five reruns\. Its records are released alongside the grid\. Across all 50 task groups the eight\-bit arm orders in 22 against the four\-bit arm’s 20 and diverges in 12 of those against 10, while endpoint divergence falls from 8 of 20 to none of 22 and omission rises from 4 of 20 to 12 of 22\. Over the ten groups that carry the four\-bit arm’s divergence, all ten diverge at both precisions, endpoint divergence falls from 8 of 10 to none, omission rises from 4 of 10 to all ten, and clinical\-content divergence appears in 3 of 10 where four bits shows none\. Higher precision therefore decides which failure occurs rather than whether one occurs\.

### Endpoint divergence and single\-emitter groups\.

Endpoint divergence compares the endpoint sets of runs that emitted, so a group left with one emitting run cannot be counted divergent\. Seven of the 43 ordering groups in the Llama sampling arm are in that position, one of 21 at the near\-greedy setting, and none in either Qwen arm\. Table[1](https://arxiv.org/html/2609.13582#S3.T1)keeps the ordering groups as the denominator so that all four action rows are read against the same population; over the groups that could show it, the Llama sampling rates are 12 of 36 for endpoint divergence and 28 of 36 for clinical content, and 8 of 20 and 0 of 20 at the near\-greedy setting\.

### Noise floor cells\.

Twenty identical calls were issued per cell across two models, three decoding configurations and three clinical prompts, giving 18 cells\. The prompts ask for a treatment choice, for the most urgent action in a deteriorating patient, and for one laboratory value read out of a note; the first two are referred to below as the treatment and triage prompts\. All twelve greedy cells returned twenty identical completions, with and without a fixed seed\. Under sampling the two open\-ended prompts produced between 14 and 20 distinct completions of 20, while the prompt asking only for a numeric value extracted from a note produced 1 for one model and 3 for the other: sampling does not perturb an output that has essentially one form\.

### The noise floor under batching\.

The cells above serve one request at a time\. A deployment does not, so we reissued them with other traffic in flight: each trial sends the target request together with one to three filler requests of differing length, so the batch it joins differs from trial to trial, and every other input is held fixed\. The server is configured to decode four sequences at once, and four concurrent requests complete in 8\.3 seconds against 21\.9 if run one after another, so the requests share batches rather than queueing\. Over 240 such trials, four of the twelve greedy cells stop being reproducible, two in each model, against one distinct completion in all twelve matched serial cells\. Which cell moves depends on the model: the 4B model diverges on the open\-ended triage prompt, three completions of 20 both with the seed fixed and without it, while the 8B model diverges on the treatment prompt, two of 20 seeded and three unseeded\. The prompt asking only for a number extracted from a note never moves in either\. Divergence here is textual: the three triage completions name the same action and differ in wording and in the shock they attribute it to\. This is a property of the serving stack rather than of the agent, and it is why a benchmark evaluating at temperature zero cannot infer from that setting alone that its runs repeat in deployment\.

### Record store\.

The store holds 695Patientresources, of which the 300 benchmark tasks name 98, so most of the cohort is background a query has to search, plus 563,426 observations and 124,969 procedures\.

### Clinical\-content divergence, two instances\.

In one observation task under Llama sampling, the same blood pressure is recorded under LOINC 8480\-6 with value‘118/77’on one run, under 8462\-4 with value 118 on a second, and under 8462\-4 with value 77 on a third\. In a magnesium task, one run posts aMedicationRequestfor one gram and another posts anObservation\. Writing the systolic value under a diastolic code is not a difference of clinical judgement, though what produces it is what Section[7](https://arxiv.org/html/2609.13582#S7)says we cannot settle\.

### The two models side by side\.

Llama diverges in 11 of its 21 ordering groups at the near\-greedy setting against Qwen’s 3 of 20, but that Llama rate carries a family\-level interval of\[0\.00,1\.00\]\[0\.00,1\.00\]and overlaps almost entirely with Qwen’s at 0\.7, 10 of 20\. We ran no cross\-model test and read no model difference from the pair\.

### Trajectory against orders\.

Restricted to each arm’s ordering groups, Qwen diverges in trajectory in 8 of 20 groups against 3 in their orders at 0\.05, and 19 against 10 under sampling, while in both Llama arms the two coincide exactly, 11 of 21 and 43 of 43\.

### First\-divergence index\.

Of the groups that diverge at all, 12 of 28, 40 of 49, 27 of 35 and 50 of 50 part company on the very first agent turn, in the column order of Table[1](https://arxiv.org/html/2609.13582#S3.T1)\. The median index is 1, 0, 0 and 0 in that same order, the largest anywhere being four, in the Qwen near\-greedy arm\. Runs separate immediately far more often than they drift apart\.

### Clock and workflow fields\.

Order\-set divergence compares whole payloads, so a differingauthoredOn,effectiveDateTimeorstatuswould count\. In practice the benchmark hands the agent a fixed clock: those fields vary within 11 of the 43 divergent groups in the Llama sampling arm and in none elsewhere, and no group in any arm is divergent on them alone, so the reported counts do not rest on a timestamp\.

### Query\-parameter normalisation\.

Across the four arms, raw comparison finds 29, 49, 36 and 50 divergent groups of 50, against 28, 49, 35 and 50 after sorting query parameters, so at most one group per arm diverges only in the order of its query string\.

## Appendix CDivergence by task family

The pooled rates in Table[1](https://arxiv.org/html/2609.13582#S3.T1)average over five task families that behave very differently: F3 records an observation and F8 places a service request, both unconditionally, while F5 orders magnesium replacement if the last level is recent and low, F9 orders potassium and a follow\-up draw if the last level is low, and F10 orders a repeat test if the last result is over a year old\. In the latter three, a run that orders nothing can be correct, and none of the three ever orders in either Qwen arm: F9’s pass rates of 0\.46 and 0\.40 there are runs that correctly withheld the order\. This breakdown is what that averaging hides, and it is the evidence for the claim in Section[7](https://arxiv.org/html/2609.13582#S7)that the pooled figures summarise this task mixture rather than a general rate\.

Table 3:Divergent order sets out of ordering groups, with each family’s mean pass rate in parentheses\. Ten task groups per family per arm; a dash marks a family that never ordered in that arm\.