使基于大语言模型的医疗器械文档评估的每个阶段都具有确定性

# Making every stage of medical device document assessment with large language models deterministic
Source: [https://arxiv.org/html/2610.21300](https://arxiv.org/html/2610.21300)
###### Abstract

To show that a medical device complies with regulation, experts judge whether documents and records meet each requirement and record the judgment for others to verify\. For each assessment item that experts define, large language models could prepare a preliminary assessment: a label such as Satisfied or Not satisfied, a reason and quoted evidence\. We sent fourteen commercial deployments of such models \(seven models from three providers, through six routes\) the same request repeatedly, with the deterministic sampling settings each accepted; every deployment returned differing assessments for at least one request, and in twelve the label changed\. Reducing reasoning where the interface allows it did not reduce the variation, and prompts of retrieved passages removed it only for the smallest models\. The same open\-weight model was reproducible or not depending only on how it was served\. A pipeline we built to be deterministic at every stage—text extraction and optical character recognition \(OCR\), chunking, embedding, indexing, retrieval and inference—returned byte\-identical output at each stage across reruns, interruptions, edits, machines of one graphics processing unit \(GPU\) model and days, for blood glucose meter and infusion pump documents\. OCR, embedding and approximate retrieval varied when their own serving was not made invariant, and outputs changed with the GPU model and with the inference engine’s scheduling settings, which therefore belong in the recorded configuration\. End\-to\-end determinism is achievable and gives a testable basis for audit and change control\.

###### keywords

large language models, reproducibility, determinism, medical devices, regulatory science, assessment science, quality management, validation

## Introduction

Demonstrating that a medical device complies with regulation is expert work that must leave a record\. Experts interpret the applicable requirements, judge whether the documents and records—technical files, test reports and data held in electronic quality management systems—meet them, and record each judgment with its evidence\. The work recurs with every submission, design change and audit, on the manufacturer’s side and on the side of notified bodies and regulators, and the records are kept for at least ten years\([1](https://arxiv.org/html/2610.21300#bib.bib1)\); a notified body must document its conformity assessment so that its conclusions “can represent objective evidence of such compliance to persons that are not themselves involved in the assessment”\([1](https://arxiv.org/html/2610.21300#bib.bib1)\), and the US Food and Drug Administration \(FDA\) reviews the substance of a 510\(k\) \(premarket notification\) submission only after a completeness screening\([2](https://arxiv.org/html/2610.21300#bib.bib2),[3](https://arxiv.org/html/2610.21300#bib.bib3)\)\.

Large language models \(LLMs\) could help with this work\([4](https://arxiv.org/html/2610.21300#bib.bib4),[5](https://arxiv.org/html/2610.21300#bib.bib5),[6](https://arxiv.org/html/2610.21300#bib.bib6)\)\. We take a limited view of their role\. Deciding which regulations, standards and guidance apply, and stating what must be shown as assessment items, remains the experts’ work\. For each item that experts define, an LLM can prepare a*preliminary assessment*—a label such as Satisfied or Not satisfied, a short reason and the passages it relies on, quoted verbatim—for a qualified expert to check against the source; the judgment, and the responsibility for it, stay with the expert, whose decision, not the preliminary assessment, enters the quality record\([7](https://arxiv.org/html/2610.21300#bib.bib7)\)\. Regulators’ guidance recommends exactly this form of evidence, quoted sentences that can be checked\([6](https://arxiv.org/html/2610.21300#bib.bib6)\), and warns against the documented tendency of even expert users to over\-rely on an automated suggestion\([8](https://arxiv.org/html/2610.21300#bib.bib8),[9](https://arxiv.org/html/2610.21300#bib.bib9),[10](https://arxiv.org/html/2610.21300#bib.bib10)\)\.

The recorded judgment rests on what the expert was shown\. A system that prepares such assessments should therefore return exactly the same assessment whenever the documents, the assessment items and its own configuration are the same\. Software used in a quality management system must be validated for its intended use, including when it changes \(ISO 13485:2016, subclause 4\.1\.6, incorporated by 21 CFR 820\.10\(a\);[11](https://arxiv.org/html/2610.21300#bib.bib11),[12](https://arxiv.org/html/2610.21300#bib.bib12),[13](https://arxiv.org/html/2610.21300#bib.bib13),[14](https://arxiv.org/html/2610.21300#bib.bib14)\), and electronic\-record systems for “accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records”\([15](https://arxiv.org/html/2610.21300#bib.bib15), §11\.10\(a\)\); with a system whose output varies from run to run, a single difference observed after a change—to a document, an item, a model or the hardware—cannot be told from ordinary variation without repeated runs, and the individual assessments the change affects cannot be identified\. An LLM pipeline does not give the same output for the same input by construction: floating\-point reductions depend on how requests are batched\([16](https://arxiv.org/html/2610.21300#bib.bib16)\), embeddings and retrieval drift with precision and with insertions\([17](https://arxiv.org/html/2610.21300#bib.bib17)\), a commercial deployment runs on hardware and software the user cannot see\([18](https://arxiv.org/html/2610.21300#bib.bib18)\), and in a recent preprint covering five API providers, repeated outputs at temperature zero were character\-identical 22% of the time on average for the two models evaluated on every task\([19](https://arxiv.org/html/2610.21300#bib.bib19)\)\. Reproducibility is a separate axis from accuracy, and this paper concerns only reproducibility\.

Deterministic inference is now available: batch\-invariant kernels and, more recently, verified and scheduling\-based engines make a model’s output independent of how requests are batched\([16](https://arxiv.org/html/2610.21300#bib.bib16),[20](https://arxiv.org/html/2610.21300#bib.bib20),[21](https://arxiv.org/html/2610.21300#bib.bib21),[22](https://arxiv.org/html/2610.21300#bib.bib22),[23](https://arxiv.org/html/2610.21300#bib.bib23)\)\. What has not been tested is whether a complete assessment pipeline, in which inference is one stage among several, can be made reproducible, and what a user of a commercial deployment actually observes in the fields of an assessment record\. We asked whether preliminary assessments of public FDA documents can be produced again, byte for byte, and found three things\. Commercial LLM deployments did not reproduce their assessments at the deterministic sampling settings they accepted: the quoted evidence changed more often than the label, one model varied differently through different routes, another changed labels only between days, and reducing reasoning did not help\. For the open\-weight models we served, reproducibility was a property of how the model was served\. A pipeline deterministic from text extraction to inference reproduced every intermediate result and output across repetition, reordering, concurrency, interruption, corpus edits, machines and days for two device types, provided the GPU model and the inference engine’s configuration are held fixed and recorded; left to their defaults, OCR, embedding and approximate retrieval were themselves sources of variation\. We argue that, where judgments recorded in a quality system rest on LLM\-prepared preliminary assessments, byte\-level reproducibility of the entire pipeline is a strong and testable engineering condition for auditing those judgments and controlling change\. It is not a regulatory requirement, and it is not sufficient for validation\.

## Results

Each preliminary assessment has four outputs \(Table[1](https://arxiv.org/html/2610.21300#Sx2.T1)\): a label \(here Satisfied, Not satisfied or Cannot determine\), a comment giving the reason, the passages quoted as evidence, and the files they come from\. We compared repeated runs of the same prompt on six fields in two layers\. The*substance*layer is what an expert relies on: the label, the cited files and the cited passages, where two runs cite the same passage when their quotations overlap at the same place in the same document\. The*text*layer is what the assessment record stores: the exact quotation strings, the comment and the full output, compared character for character\. Table[1](https://arxiv.org/html/2610.21300#Sx2.T1)compares two of the five same\-day runs of one prompt field by field; it illustrates how runs are compared, not which label is correct, since the study measures reproducibility, not accuracy\.

Table 1:One prompt of Experiment 1: a preliminary assessment and how two of its runs are compared\. Two of five runs of one prompt on the same day \(claude\-opus\-5\-5, day 1, runs 3 and 4; device K153561; texts shortened\)\. The assessment item: “At least two meters should be included in this study and at least 10 measurements should be taken per glucose level, per meter\.” Both runs cite the same document but different studies in it, and reach opposite labels; the other three runs answered*Cannot determine*\.### Commercial LLM deployments did not reproduce their assessments

No commercial deployment returned the same output for every prompt, and twelve of fourteen changed at least one label \(Table[2](https://arxiv.org/html/2610.21300#Sx2.T2); Supplementary Table[S13](https://arxiv.org/html/2610.21300#Sx5.T13)\)\. A*deployment*is a model reached through a particular route: two OpenAI models through the OpenAI API and Azure OpenAI, three Claude models through the Claude API and Amazon Bedrock, and two Gemini models through the Gemini API and Google’s Agent Platform \(Experiment 1; Figure[1](https://arxiv.org/html/2610.21300#Sx2.F1)\)\. Each received 70 prompts built from public FDA documents for blood glucose meters, in three tiers of length \(12 to 500 thousand characters\), five times on each of two consecutive days, as byte\-identical requests asking for temperature 0, seed 0 and schema\-constrained output wherever accepted and reasoning left at the provider’s default \(Methods; Supplementary Table[S1](https://arxiv.org/html/2610.21300#Sx5.T1)gives every metric by tier\)\.

Figure 1:Experiment 1\. Each of 70 prompts, built from the public FDA documents of one device and one assessment item, is sent as a byte\-identical request to each of 14 commercial deployments \(a model reached through a route\), five times on each of two days\. Only responses to the same prompt are compared, on six fields in two layers; one pair of runs is shown \(Table[1](https://arxiv.org/html/2610.21300#Sx2.T1)gives it in full\)\. The experiment measures whether the same request gives the same assessment, not whether the assessment is correct\.Within a single day, ten of the fourteen deployments changed the label of at least one prompt\. The number of such prompts ranged from 0 to 18 of 70 by deployment, the largest beinggpt\-6\-lunathrough Azure OpenAI \(Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\)\. Two further deployments changed labels only between the two days, bringing the total to twelve \(Supplementary Table[S13](https://arxiv.org/html/2610.21300#Sx5.T13)\)\. The sampling setting does not account for this\. Eight deployments rejected temperature 0 and ran at their default sampling, the four Google deployments accept the field but, by Google’s documentation, ignore it\([24](https://arxiv.org/html/2610.21300#bib.bib24)\), and the one model that ran at temperature 0 without reasoning tokens through both routes,claude\-haiku\-4\-5, still changed its label on 4 of 70 prompts through the Claude API\. Nor does the provider’s default reasoning: the eight deployments whose interface allows reasoning to be reduced were re\-measured at the lowest documented setting, which left reasoning tokens on almost no call, and the number of prompts on which the label changed rose in seven of the eight \(forgpt\-6\-luna, from 14 to 42 of 70 through the OpenAI API and from 18 to 43 through Azure\), whilegpt\-6\-lunaalso reached the output limit without a parsable answer on 53 of its 1,400 calls \(Supplementary Table[S12](https://arxiv.org/html/2610.21300#Sx5.T12)\)\. Nor does the length of the prompt: when the fourteen deployments received the much shorter prompts that the pipeline described below sends to its own judgment model \(six retrieved chunks, 7,200–8,700 characters, 49 prompts\), nine of them still changed their full output on 48–49 of the 49 prompts and two on 1 and 5, and only three—the two Gemini models through the Gemini API andclaude\-haiku\-4\-5through Bedrock—returned every prompt byte\-identically within a day \(Supplementary Table[S11](https://arxiv.org/html/2610.21300#Sx5.T11)\)\.

Table 2:Experiment 1, within one day and with the same number of runs \(each deployment’s first measurement day, 5 runs per prompt, 70 prompts; dates in the footnote\)\. Columns 3–8: prompts on which the field differed in at least one run; columns 3–5 are the substance layer and 6–8 the text layer \(Table[1](https://arxiv.org/html/2610.21300#Sx2.T1)\)\. Last column: probability that two runs of the same prompt differ in label \(mean over prompts, 95% bootstrap interval\)\. “Ours” is the judgment stage of our deterministic pipeline \(Gemma 4 31B on one NVIDIA RTX PRO 6000\) receiving the same prompts\. Rows are grouped by provider, with the two routes of a model adjacent; within a provider, models are in increasing order of the last column\. Tables[S13](https://arxiv.org/html/2610.21300#Sx5.T13), S1 and S3 and Figure[2](https://arxiv.org/html/2610.21300#Sx2.F2)use the same order\. For rows in which no prompt changed its label the bootstrap interval is degenerate and is omitted \(—\); the exact one\-sided 95% upper limit on the share of prompts whose label changes is then 4\.2% \(0 of 70\)\.RouteModelLabelFilesPassagesQuotesCommentOutputPP\(label differs\)Ours \(self\-hosted\)gemma\-4\-31b0000000\.000 —Gemini APIgemini\-3\.8\-flash000014140\.000 —Agent Platformgemini\-3\.8\-flash000011110\.000 —Gemini API§gemini\-3\.6\-flash013752520\.000 —Agent Platform§gemini\-3\.6\-flash113661610\.009 \[0\.000, 0\.026\]Claude APIclaude\-haiku\-4\-545111330310\.034 \[0\.009, 0\.069\]Amazon Bedrockclaude\-haiku\-4\-50011220\.000 —Claude APIclaude\-opus\-5\-51111233770700\.081 \[0\.039, 0\.130\]Amazon Bedrock‡claude\-opus\-5\-51013283870700\.080 \[0\.034, 0\.131\]Claude API‡claude\-haiku\-5\-51332424470700\.091 \[0\.049, 0\.140\]Amazon Bedrock‡claude\-haiku\-5\-51436424670700\.104 \[0\.056, 0\.156\]OpenAI APIgpt\-6\-astra65111970700\.046 \[0\.014, 0\.086\]Azure OpenAI†gpt\-6\-astra107152170700\.077 \[0\.034, 0\.129\]OpenAI APIgpt\-6\-luna1412202270700\.103 \[0\.057, 0\.160\]Azure OpenAI†gpt\-6\-luna1817253070700\.137 \[0\.086, 0\.197\]

†Measured on 7 October;‡on 12 October;§on 13 October; the other commercial routes on 4 October; Ours on 5 October\.

001010202030304040505060607070Ours \(self\-hosted\) / gemma\-4\-31bGemini API / gemini\-3\.8\-flashAgent Platform / gemini\-3\.8\-flashGemini API / gemini\-3\.6\-flashAgent Platform / gemini\-3\.6\-flashClaude API / claude\-haiku\-4\-5Amazon Bedrock / claude\-haiku\-4\-5Claude API / claude\-opus\-5\-5Amazon Bedrock / claude\-opus\-5\-5Claude API / claude\-haiku\-5\-5Amazon Bedrock / claude\-haiku\-5\-5OpenAI API / gpt\-6\-astraAzure OpenAI / gpt\-6\-astraOpenAI API / gpt\-6\-lunaAzure OpenAI / gpt\-6\-lunaPrompts, of 70, on which the field differed in at least one of 5 runsFull outputCited passagesLabelFigure 2:How often each field differed between runs of the same prompt \(Experiment 1, day 1: 5 runs, 70 prompts\): the number of prompts on which the field was not identical in all five runs, by deployment\. Each row is one deployment; the three marks on a row are the label \(circle\), the cited passages \(square\) and the full output \(triangle\), joined by a line\. Label and cited passages are the substance layer, full output the text layer\. Counts and order are those of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2), which also gives the probability that two runs differ, with intervals\.##### A stable label is not a stable record\.

Quoted evidence varied more than labels, in substance and not only in text\. In every deployment the number of prompts on which the cited passages differed was at least the number on which the label differed, and in twelve of the fourteen it was larger, by up to a factor of three \(Table[2](https://arxiv.org/html/2610.21300#Sx2.T2); Figure[2](https://arxiv.org/html/2610.21300#Sx2.F2)\)\.claude\-opus\-5\-5through the Claude API shows the pattern: its label changed on 11 of 70 prompts, but it cited a different passage—a different place in the documents, as in Table[1](https://arxiv.org/html/2610.21300#Sx2.T1)—on 23\. Even on prompts whose label was the same in all five runs, the cited passages differed on between 0 and 30 prompts by deployment \(Supplementary Table[S6](https://arxiv.org/html/2610.21300#Sx5.T6)\)\. Formatting accounts for little of this: normalising white space, punctuation and case reduced the count of prompts with differing quotations by at most 6 in any deployment\. The evidence, not the label alone, is what an expert reads and an auditor checks\.

##### The same model could behave differently through different routes\.

Each of the seven models was measured through two routes, and for one of them the routes differed\.claude\-haiku\-4\-5, with the same version and request settings, changed its label on 4 of 70 prompts, its cited passages on 11 and its full output on 31 through the Claude API, but on 0, 1 and 2 through Amazon Bedrock \(exact McNemar test over the prompts:p=0\.125p=0\.125for the label,0\.0060\.006for the passages and<10−6<10^\{\-6\}for the output\)\. For the other six models no route difference was detected at this resolution: the Claude 5\.5 and OpenAI models varied in every output through both routes, and the two Gemini models differed between routes in output but not in label \(Supplementary Table[S4](https://arxiv.org/html/2610.21300#Sx5.T4)\)\.claude\-haiku\-4\-5is also the one model that ran at temperature 0 without reasoning, so whether a route difference is visible depends on the sampling regime as well as on the model; either way, a validation record that names only the model does not identify what was validated\.

##### Some label changes appeared only between days\.

For thirteen of the fourteen deployments, label differences between the two days were no larger than within a day \(Supplementary Table[S13](https://arxiv.org/html/2610.21300#Sx5.T13)\)\. The exception wasgemini\-3\.8\-flash: through both Google routes it gave the same label in all five runs of each day on every prompt, yet on one and two prompts the label of the second day differed from that of the first, under an unchanged model identifier, and the probability that two of its outputs differed rose from 0\.01–0\.09 within a day to 0\.18–0\.52 across days\. A one\-day validation of its labels would have called them stable\. Prompt caching did not explain the variation either: through the OpenAI routes the probability that two runs differed was the same whether neither, one or both calls reported a cached prefix, and only through the Gemini API was it associated with cache state \(Supplementary Table[S5](https://arxiv.org/html/2610.21300#Sx5.T5)\)\.

### Reproducibility depended on how the model was served

The same open\-weight model was reproducible or not depending only on how it was served\. As a positive control, we sent the 70 prompts, on the same schedule, to the judgment stage of the deterministic pipeline described below \(“Ours” in Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\): Gemma 4 31B \(FP8\) served by vLLM with batch\-invariant inference\([16](https://arxiv.org/html/2610.21300#bib.bib16),[20](https://arxiv.org/html/2610.21300#bib.bib20)\)on one NVIDIA RTX PRO 6000 GPU; all 700 outputs over two days were byte\-identical, including prompts of about 130,000 tokens\. A dedicated server sent one request at a time might be stable without any special measure, so we also sent the prompts from eight concurrent senders with batch invariance enabled and disabled, all else equal\. Enabled, all 350 outputs were byte\-identical to the sequential outputs; disabled, the full output varied on 58 of 70 prompts and the label on 5, and Qwen3\-32B showed the same pattern, while Qwen3\.8\-27B could not be served in batch\-invariant mode at all \(Supplementary Table[S7](https://arxiv.org/html/2610.21300#Sx5.T7)\)\. Batch invariance lengthened the concurrent run by 18%\. This changes one thing on one server, and is a direct form of the attribution that Rover et al\. reached indirectly: variation comes from how the serving infrastructure executes a request, not from cloud hosting as such\([19](https://arxiv.org/html/2610.21300#bib.bib19)\)\.

### A complete assessment pipeline can be reproduced end to end

Every intermediate result and output of a complete assessment pipeline was byte\-identical across every operation we tried, for two medical device types, on one GPU model and one engine configuration \(Table[3](https://arxiv.org/html/2610.21300#Sx2.T3)\); we writeΘ\\Thetafor the configuration, everything other than the inputs that can change an output \(Methods\)\. The pipeline \(Figure[3](https://arxiv.org/html/2610.21300#Sx2.F3)\) extracts text from each document, reading scanned pages by OCR, splits it into chunks, embeds each chunk, and stores chunks and vectors under hashes of their content; for each device–item pair it retrieves the six chunks that best match the item among that device’s documents, scoring every chunk in scope exactly, builds the prompt, and has the judgment model \(Gemma 4 31B, as above\) write the preliminary assessment, whose quotations are then checked against the source by rule\. One*run*does this for every pair from an empty database and records a hash at every stage \(Experiment 2\)\.

Figure 3:Experiment 2\. The pipeline extracts text, chunks and embeds it, retrieves the six best chunks among the device’s documents, has the judgment model write the preliminary assessment, verifies each quotation against its source and records the result; the output of every stage is hashed\. A reference run is compared, stage by stage, with reruns under operations that should not change the result and with runs under a changed configuration \(another GPU model; another prefill chunk budget\); the first stage that differs locates the cause\.∗An item is paired with a device only when the device’s decision summary has a section on the item’s topic, so that every pair has a section in which to look for evidence \(not all83×11=91383\\times 11=913combinations; Methods\)\. The second document set \(85 infusion\-pump PDFs, 12 items, 516 pairs\) was identical across four runs\.We ran the pipeline on the 337 public FDA PDFs for blood glucose meters \(3,150 pages, 100 of them read by OCR\) and judged 776 device–item pairs over 83 devices\. A reference run did this once, and we then repeated the whole process under operations that should not change the result—from fresh databases; interrupted and resumed during ingestion, including during OCR, and during judgment, with and without a restart of the inference server; with documents in reverse and in hash order; with 4 or 16 documents ingested and 1, 8 or 16 items judged concurrently; with a document added, assessed and deleted; on another machine with the same GPU model; and on a later day\. Every per\-stage hash matched the reference run for all 337 documents and all 776 assessments, and deleting the added document restored the database snapshot byte for byte\. Determinism is not insensitivity to input: 19 of the 20 questions assessed for more than one device received different labels for different devices, and 746 of 776 comments are distinct\. On infusion pumps—85 PDFs and 12 assessment items chosen by a rule fixed before the documents were fetched, 516 pairs—four runs in different orders, concurrencies and days agreed on every per\-stage hash\.

Table 3:Experiment 2\. Agreement with the reference run, by stage:✓\\checkmark= all 337 documents \(ingestion: file, provenance, version and text hashes; embedding: chunk and vector hashes\) or all 776 items \(judgment: retrieval, prompt, raw output, label, comment, evidence, downgrade\) identical; otherwise the count that matched\. Operations in the first group should not change the result; the second changes the configurationΘ\\Theta, so differences are expected; the third removes one of the six properties\.OperationIngestionEmbeddingJudgment*Fixed configuration: operations that should not change the result*T1 Repeat; repeat on another day✓\\checkmark✓\\checkmark✓\\checkmarkT2 Interrupt and resume; inference server restarteda✓\\checkmark✓\\checkmark✓\\checkmarkT3 Reverse order; hash order✓\\checkmark✓\\checkmark✓\\checkmarkT4 Concurrency 1, 4, 16 documents; 1, 8, 16 items✓\\checkmark✓\\checkmark✓\\checkmarkT5 Add a document, assess, delete itsnapshot identical✓\\checkmarkT6 Same GPU model, another machine✓\\checkmark✓\\checkmark✓\\checkmarkT7′Second H100 machine vs\. the first H100 run✓\\checkmark✓\\checkmark✓\\checkmarkT8/T9 Budget 16,384; prefix caching disabled——✓\\checkmarkInfusion\-pump corpus, four runsb✓\\checkmark✓\\checkmark✓\\checkmark*Changed configurationΘ\\Theta: differences expected*T7 H100 80GB instead of RTX PRO 6000c327/3370/337labels 742; raw 117/776T8 Budget 4,096, 8 concurrent items, two runsd——raw 662, 654; labels 770, 768/776T8 Budget 4,096, 16 concurrent items——raw 609; labels 768/776T8 Budget 4,096, 1 concurrent item——✓\\checkmark*One property removed \(ablation on the reference GPU\)*OCR without batch invariance, 16 concurrent docse323/336——Embedding without batch invariance \(vLLM pooling\)f—0/337—control: pipeline embedding server, two concurrent runs—✓\\checkmark—
T8 and T9 rerun judgment on the reference database \(budget 16,600 tokens, prefix caching enabled\)\.aIngestion killed after 131 files and, separately, during OCR; judgment killed; the inference server killed 90 s into judgment and restarted\.bTwo sorted\-order runs, one hash\-order run with 16 concurrent documents and items, one on the next day\.cThe 294 text\-layer documents matched and the 10 differing documents all contain OCR pages; labels agreed for 620 of the 650 items with identical prompts; an A100 80GB could not run the configuration \(Methods\)\.dThe two runs agreed with each other on 633 of 776 raw outputs\.eStopped by a two\-hour limit after 336 documents; OCR documents 29/42, text\-layer documents 294/294\.fThe two concurrent runs agreed with each other on 3 of 337 documents\. The hash\-order run with 16 concurrent documents is the rerun with the corrected extractor \(Supplementary Note 2\)\.

##### Each GPU stage needs its own invariance\.

Removing batch invariance from one stage at a time, on the reference GPU, moved that stage’s output and no other \(Table[3](https://arxiv.org/html/2610.21300#Sx2.T3)\)\. With the OCR servers run without batch invariance and sixteen documents ingested concurrently, the extracted text changed for 13 of 42 OCR documents and for none of the 294 text\-layer documents\. With embedding done by the inference engine’s pooling runner without batch invariance, the vectors differed between a run made alone and two runs made concurrently for 332–336 of 337 documents, whereas the pipeline’s embedding server, which computes one text at a time with deterministic kernels, gave identical vectors in two concurrent runs\. Approximate retrieval indexes built from the same vectors varied with how they were built: hnswlib at its default search depth was reproducible only single\-threaded in a fixed order, Elasticsearch’s default vector index changed 3 of 33 queries on an identical rebuild, and its BM25 ranking changed only after a document was added and deleted \(because deleted documents are retained as soft deletes\) or the shard count changed \(Supplementary Table[S14](https://arxiv.org/html/2610.21300#Sx5.T14)\); the pipeline’s exact, scoped retrieval matched in every case\.

##### The hardware class and the engine’s scheduling belong in the configuration\.

One run on an H100 instead of the RTX PRO 6000, with the same software and settings, did not reproduce the reference run, and the per\-stage hashes located every difference: all 294 text\-layer documents matched, OCR output differed for 10 of 43 OCR documents, every embedding set differed, the retrieved passages and prompts differed for 126 of 776 items, and labels matched for 742 of 776 items \(95\.6%\)—even among the 650 items whose prompts were identical, 30 labels changed, because greedy decoding is sensitive to small numerical differences\([25](https://arxiv.org/html/2610.21300#bib.bib25)\), as the vendors of GPU libraries document\([26](https://arxiv.org/html/2610.21300#bib.bib26),[27](https://arxiv.org/html/2610.21300#bib.bib27),[28](https://arxiv.org/html/2610.21300#bib.bib28),[29](https://arxiv.org/html/2610.21300#bib.bib29)\)\. This is a difference between GPU models, not variation on the H100: a second run on another H100 machine, eleven days later, reproduced the first H100 run exactly, every document, vector and judgment\. The engine’s prefill chunk budget behaved the same way, with a condition: with the budget lowered from 16,600 to 4,096 tokens, judgment at one item at a time was byte\-identical to the reference, but with eight concurrent items the raw output differed on 122 and 114 items in two runs and on 143 items between them, and with sixteen on 167, while at 16,384 tokens, and with prefix caching disabled, every item was byte\-identical \(Table[3](https://arxiv.org/html/2610.21300#Sx2.T3); Vosti reported the same sensitivity of this batch\-invariant mode to prefill chunking\([21](https://arxiv.org/html/2610.21300#bib.bib21)\)\)\. The budget by itself changes nothing; when several prompts share a budget too small for them, how the scheduler divides their prefills across engine steps depends on which requests are present, and the batch\-invariant kernels do not make the result independent of that division\. The list of what changed and at which stage is the change\-impact analysis a validation team needs when hardware or engine settings change, and both the GPU model and the engine’s scheduling settings are therefore part ofΘ\\Theta\.

## Discussion

We propose that, where LLMs prepare assessments for regulated work such as that of the medical device industry, byte\-level reproducibility of the whole pipeline be a testable engineering condition: the same documents, assessment items and configuration should yield the same assessment, at every stage\. The fourteen commercial deployments we measured did not provide this at the deterministic sampling settings they accepted, with reasoning at its default or at its minimum, and where the label held, the evidence behind it often changed; the open\-weight models we served were reproducible or not depending on how they were served; and a pipeline built to be deterministic at every stage reproduced every intermediate result and output for two device types, on one hardware class and engine configuration\. We measured the commercial deployments from the outside, as any customer would, so we do not attribute their variation to causes, and our results do not rank the models\. They do show that a deployment’s reproducibility cannot be established once and then assumed: it differed between models, for one model between routes, and for another between days, so that a test confined to one session would have missed it\.

End\-to\-end determinism makes three things possible that agreement between runs alone does not\. A stored assessment can be regenerated as well as kept: producing it again from the stored documents, items and configuration and obtaining the same bytes shows that the record is what the recorded configuration still produces for these inputs, and lets an auditor see the passages and prompt behind a judgment exactly as the expert saw them\. The effect of a change can be listed: when a document, an item, a prompt template, a model or the hardware is changed and the assessments are run again, every difference is caused by that change, and the per\-stage hashes show which assessments changed and at which stage, as the H100 run did, so that the judgments that relied on them can be looked at again—although a listed difference is not thereby a meaningful one \(a change of GPU model alone moved 30 of 650 labels on identical prompts\), nor is an unchanged assessment thereby unaffected\. And when documents change, only the assessments whose retrieved evidence changed need the model again, since an unchanged input to the model has an unchanged output; retrieval is cheap and inference is not \(Supplementary Table[S3](https://arxiv.org/html/2610.21300#Sx5.T3)\)\.

What determinism does not give is correctness: a system that reproduces a wrong assessment exactly has reproduced a wrong assessment, and we have not measured how often the assessments are right\. Two alternatives are weaker\. Storing every output with an audit trail preserves what was produced, and is what remains once a hardware class is retired, but cannot show that the validated system still produces it, nor which later changes would alter it\. Statistical consistency suffices for population\-level questions\([4](https://arxiv.org/html/2610.21300#bib.bib4),[30](https://arxiv.org/html/2610.21300#bib.bib30)\), and disagreement between runs is itself used as a signal of uncertainty\([31](https://arxiv.org/html/2610.21300#bib.bib31),[32](https://arxiv.org/html/2610.21300#bib.bib32),[33](https://arxiv.org/html/2610.21300#bib.bib33),[34](https://arxiv.org/html/2610.21300#bib.bib34)\); a recorded judgment, however, is an individual decision made on an individual assessment, and a deterministic pipeline can still estimate uncertainty by varying the configuration deliberately, as the H100 and chunk\-budget runs did\. For manufacturers choosing such tools today, the results suggest testing reproducibility directly, across days and on the quoted evidence, and recording the route, the engine settings and the hardware class, because the same model name covered different behaviour; without determinism, a vendor’s silent change can be detected only statistically\([35](https://arxiv.org/html/2610.21300#bib.bib35),[36](https://arxiv.org/html/2610.21300#bib.bib36),[37](https://arxiv.org/html/2610.21300#bib.bib37)\), and served behaviour has changed under unchanged model identifiers\([38](https://arxiv.org/html/2610.21300#bib.bib38),[39](https://arxiv.org/html/2610.21300#bib.bib39),[40](https://arxiv.org/html/2610.21300#bib.bib40)\)\. For auditors, whether the preliminary assessment behind a recorded judgment can be produced again is a concrete question, to be scaled to risk as Computer Software Assurance recommends\([14](https://arxiv.org/html/2610.21300#bib.bib14)\)\.

Determinism holds within a hardware class, so records outlive the hardware that produced them only as stored outputs\. Technical documentation must be kept for at least 10 years, and 15 for implantable devices\([1](https://arxiv.org/html/2610.21300#bib.bib1),[41](https://arxiv.org/html/2610.21300#bib.bib41)\), longer than a GPU model may be available and far longer than a hosted model version, which providers retire on months’ notice\([42](https://arxiv.org/html/2610.21300#bib.bib42),[43](https://arxiv.org/html/2610.21300#bib.bib43),[44](https://arxiv.org/html/2610.21300#bib.bib44)\)\. Container images, model weights and drivers should therefore be archived with the records, and reference outputs re\-established on new hardware by listing the changed assessments\. This favours models an organisation serves itself, which today means open\-weight models whose architecture the serving software supports deterministically; batch invariance cost 18% in wall time in our test, and newer engines narrow the gap in some regimes\([22](https://arxiv.org/html/2610.21300#bib.bib22),[21](https://arxiv.org/html/2610.21300#bib.bib21)\)\.

Regulations and regulators’ guidance do not require byte\-level reproducibility, but they repeatedly ask that a recorded result can be checked, and industry guidance frames validation by risk\([45](https://arxiv.org/html/2610.21300#bib.bib45),[46](https://arxiv.org/html/2610.21300#bib.bib46)\)\. Part 11 requires audit trails and reliable, consistent performance of electronic\-record systems, applies to records that predicate rules require, here the expert’s recorded judgment, and has been enforced with discretion since 2003\([15](https://arxiv.org/html/2610.21300#bib.bib15),[47](https://arxiv.org/html/2610.21300#bib.bib47)\); the EU device regulations require the technical documentation to locate the evidence of conformity\([1](https://arxiv.org/html/2610.21300#bib.bib1),[41](https://arxiv.org/html/2610.21300#bib.bib41)\); the FDA–EMA principles for AI in drug development ask for “detailed, traceable, and verifiable” documentation\([48](https://arxiv.org/html/2610.21300#bib.bib48), Principle 6\); EU GMP Annex 11 requires life\-cycle validation\([49](https://arxiv.org/html/2610.21300#bib.bib49)\); and the EU AI Act requires high\-risk systems, from 2028 for devices under notified\-body assessment, to log events and to perform “consistently in those respects throughout their lifecycle”\([50](https://arxiv.org/html/2610.21300#bib.bib50),[51](https://arxiv.org/html/2610.21300#bib.bib51)\), which we cite by analogy\. Only a consultation draft of EU GMP Annex 22 names identical outputs, limiting its scope to “models with a deterministic output” and excluding LLMs from critical use\([52](https://arxiv.org/html/2610.21300#bib.bib52)\), and its drafting group has since discussed admitting generative models under a risk\-based strategy\([53](https://arxiv.org/html/2610.21300#bib.bib53),[54](https://arxiv.org/html/2610.21300#bib.bib54)\)\. FDA’s 2019 discussion paper defines a “locked” algorithm as one that gives the same result for the same input\([55](https://arxiv.org/html/2610.21300#bib.bib55)\), and FDA’s and PMDA’s 2026 papers on generative\-AI devices list reproducibility across repeated runs among the properties to evaluate\([56](https://arxiv.org/html/2610.21300#bib.bib56),[57](https://arxiv.org/html/2610.21300#bib.bib57)\)\.

Previous work has made individual stages deterministic or measured variation, but not tested a complete assessment pipeline\. Batch\-invariant and tree\-based kernels, verified replay and fixed execution shapes make inference independent of batching\([16](https://arxiv.org/html/2610.21300#bib.bib16),[20](https://arxiv.org/html/2610.21300#bib.bib20),[58](https://arxiv.org/html/2610.21300#bib.bib58),[23](https://arxiv.org/html/2610.21300#bib.bib23),[22](https://arxiv.org/html/2610.21300#bib.bib22),[21](https://arxiv.org/html/2610.21300#bib.bib21)\); ReproRAG finds embeddings and FAISS retrieval reproducible run to run under fixed seeds but displaced by insertions\([17](https://arxiv.org/html/2610.21300#bib.bib17)\), and deterministic\-first retrieval, retrieval\-aware ingestion and a finance\-oriented harness address other stages without extraction or OCR\([59](https://arxiv.org/html/2610.21300#bib.bib59),[60](https://arxiv.org/html/2610.21300#bib.bib60),[61](https://arxiv.org/html/2610.21300#bib.bib61),[62](https://arxiv.org/html/2610.21300#bib.bib62)\)\. Variation of hosted LLMs at temperature 0 has been measured in research settings\([63](https://arxiv.org/html/2610.21300#bib.bib63),[64](https://arxiv.org/html/2610.21300#bib.bib64),[19](https://arxiv.org/html/2610.21300#bib.bib19)\), the same model has been found to differ between routes\([40](https://arxiv.org/html/2610.21300#bib.bib40),[36](https://arxiv.org/html/2610.21300#bib.bib36),[65](https://arxiv.org/html/2610.21300#bib.bib65)\), and providers describe seeds as “best effort” and temperature 0 as not “fully deterministic”\([66](https://arxiv.org/html/2610.21300#bib.bib66),[67](https://arxiv.org/html/2610.21300#bib.bib67)\)\. LLMs have been validated for extraction from FDA device documents without repeated runs\([4](https://arxiv.org/html/2610.21300#bib.bib4)\)and proposed for source\-linked safety analysis\([7](https://arxiv.org/html/2610.21300#bib.bib7)\)\. To our knowledge, no prior work tests byte\-level reproducibility of a complete assessment pipeline from text extraction to quoted evidence, relates it to medical\-device quality work, or measures, field by field on an assessment record, how commercial deployments vary across repeated requests, routes and days\.

The study has limitations\. It concerns reproducibility only; the assessment items are guidance sentences chosen by rule, and public 510\(k\) documents for two device types stand in for manufacturers’ records\. The pipeline used one judgment model family on a single GPU, relies on the batch\-invariant mode of one inference engine, and its six properties were sufficient under the operations tried, which is an empirical finding, not a proof; no one outside the author’s company has yet reproduced it\. The commercial measurements cover two days per deployment, one client region and pay\-as\-you\-go tiers, with eight deployments sampled at provider defaults; the prompts were whole documents, and the 70 prompts come from 49 device–item pairs resampled as independent units; the positive control, the concurrency test, the between\-day count and the two addenda were added during the study \(Supplementary Note 2\)\. Usefulness is a property of assessments in the hands of experts, and this study establishes reproducibility, not usefulness: how often the assessments are correct, whether the quoted evidence is what an expert would cite, and whether review helps or biases the expert\([9](https://arxiv.org/html/2610.21300#bib.bib9),[68](https://arxiv.org/html/2610.21300#bib.bib68)\)are questions we cannot answer alone, and we publish this preprint in part to find the regulatory and quality\-management experts with whom to answer them\. With a deterministic pipeline, what the experts evaluate is fixed, and any later difference can be attributed to a change\.

## Methods

##### Definitions\.

An*assessment item*is a requirement to be assessed, defined by the assessment’s administrator, who also defines the label set\. A*preliminary assessment*is, for one item and a set of documents, a label from the label set, a comment giving the reason, and quoted evidence—passages copied verbatim from the documents, each with its cited file; the model’s internal reasoning is not part of it\. The*assessment record*is the stored preliminary assessment with the hashes that identify its inputs and configuration; the*quality record*is the expert’s recorded judgment, which is what quality\-system requirements and Part 11 concern, and we assume that an assessment record is retained with the judgment that relied on it\. A*route*is the way a model is reached \(a provider’s API, a cloud platform or a self\-hosted server\) and a*deployment*is a model reached through a particular route\. The*configuration*Θ\\Thetais everything other than the inputs that can change an output: the extraction, chunking and embedding components, the retrieval parameters, the prompt template and label set, the model weights, the inference engine’s version and settings, including its prefill chunk budget and concurrency limit, and the hardware class \(GPU model with its driver and CUDA versions\); the inputs are the document bytes and the assessment items, and a*manifest*is the recorded listing ofΘ\\Theta\.*Byte\-level reproducibility*means that, for a fixedΘ\\Theta, every intermediate result and output is identical byte for byte across runs that differ only by operations that should not matter: repeating, restarting, interrupting and resuming, reordering, changing concurrency, and adding and deleting a document\. We use*reproducibility*in the computational sense; in the terms of FDA’s draft guidance on AI\-enabled device software\([69](https://arxiv.org/html/2610.21300#bib.bib69)\), repeating a request under the same conditions tests repeatability and agreement across days, orders, concurrency and machines tests reproducibility under pre\-specified changed conditions\.

##### Documents and assessment items\.

The main corpus is the public FDA documents for over\-the\-counter blood glucose test systems \(product code NBW; in vitro diagnostic devices for self\-testing in the EU\) with a decision date from 2015 to 2025 in openFDA and a published 510\(k\) summary: 133 510\(k\) summaries, 132 decision summaries and 72 predicate summaries, 337 PDFs\. The assessment items are the 11 sentences of FDA’s 2020 guidance for these systems that state a numeric minimum of a counted item \(“at least” or “a minimum of” followed by a numeral and a count of meters, strips, vials, lots, days, measurements or concentrations\)\. An item is paired with a device when the device’s decision summary has a section whose heading matches the item’s topic, so that every pair has a section in which to look for evidence; 83 devices had such a section, giving 776 pairs, of which the reference run labelled 356*Cannot determine*\(324 by the model and 32 by the verification rule\), 380*Satisfied*and 40*Not satisfied*\. We restricted the pairs in this way because the hard case for reproducibility is a judgment that has to find and quote evidence\. The second corpus is the 43 FDA 510\(k\) summaries for infusion pumps \(product code FRN\) cleared in 2015–2025 and the 42 predicate summaries they cite, 85 PDFs; its 12 items are every 8\.9th of the 107 sentences containing “should” or “recommend” in Section 5 of FDA’s infusion pump guidance \(Supplementary Table[S9](https://arxiv.org/html/2610.21300#Sx5.T9)\), each assessed for every device \(516 pairs\)\. Both pairing rules were fixed before any assessment\.

##### Experiment 1: commercial deployments\.

The 70 prompts were built from the 776 pairs, ordered by 510\(k\) number and item, by a fixed rule without randomness: every 38th pair for the 20*short*prompts \(the 510\(k\) summary; 12–47 thousand characters\), every 19th for the 40*mid*prompts \(the device’s document set; 31–119 thousand\) and every 77th for the 10*long*prompts \(the set plus other devices’ summaries; 483–502 thousand\), skipping pairs whose summary had under 1,000 characters of text; the series overlap, so the 70 prompts come from 49 distinct pairs\. Each prompt was sent five times on each of two consecutive calendar days \(JST\), one request at a time, from a virtual machine in Google Cloud us\-central1; the days differ by deployment \(Supplementary Table[S2](https://arxiv.org/html/2610.21300#Sx5.T2)\), and the same\-day route comparisons re\-sent the OpenAI API’s short prompts andclaude\-opus\-5\-5through the Claude API on the days of their counterparts\. The size of the experiment was set by cost; five identical runs exclude only a per\-run change probability above 45% at the one\-sided 95% level, so a zero in Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)means that no change was observed, not that the deployment is stable\. Requests were sent by Python’s standard HTTP client without vendor SDKs, asking for temperature 0, seed 0 and JSON\-schema output wherever accepted, with no reasoning parameter and an output limit of 16,384 tokens; settings a provider rejected were removed and recorded\. Both OpenAI models, andclaude\-opus\-5\-5andclaude\-haiku\-5\-5, rejected temperature; the Gemini deployments accept the field but ignore it\([24](https://arxiv.org/html/2610.21300#bib.bib24)\)\. Reasoning tokens were reported on every call of the Gemini deployments and ofgpt\-6\-luna, on most calls ofgpt\-6\-astraand the Claude 5\.5 models, and on none ofclaude\-haiku\-4\-5\. Of the calls reported, 78 to the Claude API failed when a credit balance ran out and were resent, 12 to Azure were retried after a rate\-limit response, and 39 ofgpt\-6\-lunaandgpt\-6\-astrareached the output limit without parsable JSON and count as differing in every field\. The system prompt, the user prompt format and the JSON schema are reproduced in Supplementary Note 1; the 70 prompts, their hashes and every response with its headers, timestamps and usage fields are in the supplementary material, and Supplementary Table[S8](https://arxiv.org/html/2610.21300#Sx5.T8)gives every deployment, date, start and end time and estimated cost \($1,291 in all, from provider\-reported tokens at list prices\)\.

##### Experiment 1 addenda\.

*Reduced reasoning*: on 14 and 15 October the 70 prompts were sent again togpt\-6\-lunathrough both routes withreasoning\_effort: "none", toclaude\-haiku\-5\-5through both routes withthinking: \{type: "disabled"\}, and to the four Gemini deployments with the lowest documented thinking level \(LOWforgemini\-3\.8\-flash,MINIMALforgemini\-3\.6\-flash\);gpt\-6\-astraandclaude\-opus\-5\-5reject these settings\. No setting was rejected\.*Retrieved\-passage prompts*: for the 49 pairs, the chat messages that the pipeline’s reference run sent to its judgment model were rebuilt from the chunk export and the recorded retrieval results, verified against the recorded prompt hash of every item, and sent five times on each of the same two days to all fourteen deployments at the settings above; two calls ofclaude\-opus\-5\-5through the Claude API failed and were omitted\.

##### Concurrency test\.

The 70 prompts were split into eight groups sent concurrently, five runs each, to one vLLM 0\.29\.0 server on one RTX PRO 6000 with batch\-invariant inference enabled or disabled and all else equal, for Gemma 4 31B \(FP8\), Qwen3\-32B \(BF16; 57 prompts fit its context\) and Qwen3\.8\-27B \(FP8\), the Qwen models with thinking disabled and the attention kernel fixed \(TRITON\_ATTN\)\.

##### Experiment 2: pipeline\.

Six properties fix what each stage does\.*Content addressing*: every document version, extraction, chunk set, embedding set, retrieval result, prompt and raw output is identified by the SHA\-256 hash of its content and of the producing component’s manifest, which lists that component’s models, libraries and parameters, not the pipeline’s source code\.*Deterministic extraction*: the same file yields the same text, including pages read by OCR with a self\-hosted model under batch\-invariant inference and greedy decoding\.*Deterministic embedding*: for a fixed configuration the same text yields the same chunks and the same vectors bit for bit whenever they are computed; within one database, vectors are stored under their content hash and reused\.*Exact, scoped retrieval*: the result depends only on the query and the documents in scope\.*Serving\-invariant judgment*: for a fixed engine configuration, the model’s output depends only on the prompt, not on which other requests are served with it, their order, or whether cached prefixes are reused; implemented with batch\-invariant inference, greedy decoding and grammar\-constrained output\.*Deterministic records*: quotations are verified against the source by rule, unverifiable ones removed and the label downgraded by rule, and every run is written to a hash\-chained ledger\. The implementation tested is AskDona Assessment, a product of the author’s company \(Competing interests\)\. Pages whose text layer has fewer than 200 characters are rendered at 150 dpi and read by olmOCR\-2, with GLM\-OCR used when olmOCR\-2’s output is truncated; chunks of at most 1,400 characters are embedded one at a time with Qwen3\-Embedding\-8B; retrieval returns the top six of the device’s chunks \(4,977 in the main corpus\) by BM25 and cosine similarity combined; judgment uses Gemma 4 31B \(FP8\) on vLLM 0\.29\.0 with batch invariance; and a quotation that is not found in its cited passage is removed, the label being downgraded to*Cannot determine*only when no quotation remains \(41 quotations and 32 labels in the reference run\)\. Every component, version, parameter, rule and hash is listed in Supplementary Table[S15](https://arxiv.org/html/2610.21300#Sx5.T15)\. Judging the 776 pairs took 20\.3 minutes with eight concurrent items\.

##### Experiment 2: operations\.

Each run starts from an empty database, re\-reads every OCR page and re\-embeds every chunk, and records a hash at every stage; judgment\-only reruns reuse the reference database’s chunks and vectors\. The operations were: T1, a repeat from fresh databases, also on a later day; T2, ingestion killed after 131 files and, separately, after 3 files during OCR, then resumed, judgment killed and rerun, and the inference server killed 90 seconds into judgment and restarted before the rerun; T3, sorted, reverse and hash order; T4, 1, 4 and 16 concurrent documents and 1, 8 and 16 concurrent items; T5, a document added, assessed with the retrieval scope set to the whole corpus, and deleted; T6, the same GPU model on another machine; T7, an H100 80GB \(an A100 could not run the configuration\), run on 4 October and again on 14 October on a second H100 machine created from a disk image of the reference machine, with the GPU memory share raised to give the same absolute memory; T8, judgment rerun with the prefill chunk budget \(max\-num\-batched\-tokens\) changed from 16,600 to 4,096 \(twice with 8 concurrent items, once each with 1 and 16\) and to 16,384 tokens; T9, judgment rerun with prefix caching disabled\. Two ablations removed batch invariance from one stage: the OCR servers started withVLLM\_BATCH\_INVARIANT=0and the corpus ingested with 16 concurrent documents \(stopped by a two\-hour limit after 336 of 337 documents, which are the documents compared\), and the corpus embedded by vLLM’s pooling runner for the same model without batch invariance, once alone and twice concurrently, with the pipeline’s embedding server run twice concurrently as the control\.

##### Baselines\.

From the same 4,977 chunks, vectors and 33 query embeddings we built hnswlib 0\.8\.0 indexes \(M=16M=16, ef\_construction=200=200, default search depth ef=10=10\) and Elasticsearch 9\.5\.4 indexes \(default mapping, one shard\) under the operations of Supplementary Table[S14](https://arxiv.org/html/2610.21300#Sx5.T14), plus a sweep of the hnswlib search depth over 10, 24, 50, 100, 200 and 500 \(at ef≥100\\geq 100every rebuild matched the first on all 33 queries, with recall 1\.0 against exact search\) and a soft\-delete test in which settingindex\.soft\_deletes\.retention\_lease\.periodto zero and force\-merging restored all 33 BM25 scores\.

##### Measures and statistics\.

For each prompt we determine whether two runs differ in each of six fields: the label; the set of cited files; the cited passages, where each quotation is located in its source document after normalising white space and two runs agree if every quotation of each overlaps, by at least one character, a quotation of the other in the same document \(a lenient, non\-transitive rule, so the Passages counts are lower bounds\); the exact quotation strings; the comment; and the output bytes\. Files, passages and quotations are compared as sets; a call that returned no parsable JSON differs from every other run in every field; between 88% and 100% of quotations were located in the source, by deployment \(Supplementary Table[S6](https://arxiv.org/html/2610.21300#Sx5.T6)\); Experiment 1 compares raw model outputs without the pipeline’s verification rule\. A prompt counts as differing in a field when any pair of its runs differs; such counts cannot fall as runs are added\. The probability that two runs differ is computed per prompt over all pairs of runs and averaged over prompts, with 95% intervals from 2,000 percentile bootstrap resamples of prompts \(seed 0\)\. The normalised comparison removes white space, punctuation and case before comparing\. Between days, a prompt counts as changed when its label was the same in all runs of each day but differed between the days\. Route pairs are compared with exact McNemar tests on the paired prompt\-level indicators\. Experiment 2 compares, for every document and item, the per\-stage hashes of each run with those of the reference run\.

##### Software and data\.

Experiment 2 ran on Ubuntu 22\.04\.5 with NVIDIA driver 580\.178\.04 \(CUDA 13\.0\), vLLM 0\.29\.0 withVLLM\_BATCH\_INVARIANT=1\(a beta feature in this version\),RedHatAI/gemma\-4\-31B\-it\-FP8\-dynamic\(FP8 weights and KV cache, enforce\-eager, chunked prefill, xgrammar structured output\),allenai/olmOCR\-2\-7B\-1025,zai\-org/GLM\-OCRandQwen/Qwen3\-Embedding\-8Bin PyTorch with deterministic algorithms; revisions and full environment records for every run, the document lists with hashes \(documents\.csv,frn\_documents\.csv; all files were downloaded again from FDA and were identical\), and the per\-stage hashes are in the supplementary material\. The measures, operations and conditions were fixed before the runs and every change was recorded when it was made; Supplementary Note 2 lists them all\.

#### Data availability

All inputs, prompts, outputs and per\-stage hashes, with the document manifests \(documents\.csv,frn\_documents\.csv\) and the analysis scripts, will be deposited at Zenodo under the reserved DOI 10\.5281/zenodo\.23282335 on publication of this preprint; the documents themselves are public at FDA\.

#### Code availability

The implementation tested, AskDona Assessment \(commit 7adab97 with the extractor change described in Methods\), is proprietary; it will be made available to editors and reviewers on journal submission, and a minimal reference implementation of the six properties, with a script that checks the deposited per\-stage hashes on an NVIDIA RTX PRO 6000, will be released with the journal version under an open licence\. The analysis scripts that produce every number in this paper from the deposited records are in the supplementary material\.

#### Acknowledgements

This study was funded by GFLOPS Co\., Ltd\., which employs the author\. Two colleagues at GFLOPS reviewed drafts of the manuscript and checked its facts\. Large language model tools \(Anthropic’s Claude, through Claude Code\) were used under the author’s direction to write and check analysis scripts and to draft and edit text; the author verified every number and statement and is responsible for the content\.

#### Competing interests

The author is an employee of GFLOPS Co\., Ltd\., which develops and sells AskDona Assessment, the implementation evaluated here, and is the inventor on a provisional patent application covering the pipeline described\.

#### Author contributions

R\.S\. designed the study, built the pipeline, ran the experiments, analysed the data and wrote the manuscript\.

## References

- \(1\)European Parliament and Council of the European Union\.Regulation \(EU\) 2017/745 on medical devices\.Official Journal of the European Union, L 117, 5 May 2017 \(2017\)\.URL[https://eur\-lex\.europa\.eu/legal\-content/EN/TXT/?uri=CELEX:32017R0745](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745)\.Article 10\(8\); Annex II, Section 4; Annex VII, Section 4\.6\.
- \(2\)U\.S\. Food and Drug Administration\.Electronic submission template for medical device 510\(k\) submissions: guidance for industry and FDA staff \(2023\)\.URL[https://www\.fda\.gov/media/152429/download](https://www.fda.gov/media/152429/download)\.Issued 2 October 2023; docket FDA\-2021\-D\-0872\.
- \(3\)U\.S\. Food and Drug Administration\.Refuse to accept policy for 510\(k\)s: Guidance for industry and food and drug administration staff \(2022\)\.URL[https://www\.fda\.gov/media/83888/download](https://www.fda.gov/media/83888/download)\.
- \(4\)Li, H\.*et al\.*Scaling medical device regulatory science using large language models\.*npj Digital Medicine*9, 221 \(2026\)\.
- \(5\)U\.S\. Food and Drug Administration\.FDA launches agency\-wide AI tool to optimize performance for the american people\.Press announcement, 2 June 2025 \(2025\)\.URL[https://www\.fda\.gov/news\-events/press\-announcements/fda\-launches\-agency\-wide\-ai\-tool\-optimize\-performance\-american\-people](https://www.fda.gov/news-events/press-announcements/fda-launches-agency-wide-ai-tool-optimize-performance-american-people)\.
- \(6\)European Medicines Agency and Heads of Medicines Agencies\.Guiding principles on the use of large language models in regulatory science and for medicines regulatory activities \(2024\)\.URL[https://www\.ema\.europa\.eu/en/documents/other/guiding\-principles\-use\-large\-language\-models\-regulatory\-science\-medicines\-regulatory\-activities\_en\.pdf](https://www.ema.europa.eu/en/documents/other/guiding-principles-use-large-language-models-regulatory-science-medicines-regulatory-activities_en.pdf)\.29 August 2024; Section 5\.1, recommendations 4 and 7\.
- \(7\)Gangopadhyay, T\., Adler, R\., Liggesmeyer, P\. & Reich, J\.From safety documentation to safety knowledge support: An evidence\-grounded LLM framework for medical devices \(2026\)\.ArXiv:2608\.12025v1 \[cs\.SE\]\.
- \(8\)Parasuraman, R\. & Manzey, D\. H\.Complacency and bias in human use of automation: an attentional integration\.*Human Factors*52, 381–410 \(2010\)\.
- \(9\)Goddard, K\., Roudsari, A\. & Wyatt, J\. C\.Automation bias: a systematic review of frequency, effect mediators, and mitigators\.*Journal of the American Medical Informatics Association*19, 121–127 \(2012\)\.
- \(10\)Dratsch, T\.*et al\.*Automation bias in mammography: the impact of artificial intelligence BI\-RADS suggestions on reader performance\.*Radiology*307, e222176 \(2023\)\.
- \(11\)International Organization for Standardization\.ISO 13485:2016 medical devices — quality management systems — requirements for regulatory purposes \(2016\)\.URL[https://www\.iso\.org/standard/59752\.html](https://www.iso.org/standard/59752.html)\.Third edition, March 2016; subclause 4\.1\.6\.
- \(12\)U\.S\. Food and Drug Administration\.21 CFR § 820\.10 requirements for a quality management system\.Electronic Code of Federal Regulations \(2026\)\.URL[https://www\.ecfr\.gov/current/title\-21/part\-820/section\-820\.10](https://www.ecfr.gov/current/title-21/part-820/section-820.10)\.Current through 9 October 2026; ISO 13485:2016 incorporated by reference at § 820\.7\. Accessed 2026\-10\-14\.
- \(13\)U\.S\. Food and Drug Administration\.Medical devices; quality system regulation amendments \(final rule\)\.Federal Register 89 FR 7496, 2 February 2024; effective 2 February 2026 \(2024\)\.URL[https://www\.federalregister\.gov/documents/2024/02/02/2024\-01709/medical\-devices\-quality\-system\-regulation\-amendments](https://www.federalregister.gov/documents/2024/02/02/2024-01709/medical-devices-quality-system-regulation-amendments)\.
- \(14\)U\.S\. Food and Drug Administration\.Computer software assurance for production and quality management system software: Guidance for industry and food and drug administration staff \(2026\)\.URL[https://www\.fda\.gov/media/188844/download](https://www.fda.gov/media/188844/download)\.Issued February 3, 2026\. Docket FDA\-2022\-D\-0795\.
- \(15\)U\.S\. Food and Drug Administration\.21 CFR part 11: Electronic records; electronic signatures, § 11\.10 controls for closed systems\.Electronic Code of Federal Regulations \(2026\)\.URL[https://www\.ecfr\.gov/current/title\-21/part\-11/section\-11\.10](https://www.ecfr.gov/current/title-21/part-11/section-11.10)\.Current through 2026\-09\-30; accessed 2026\-10\-14\.
- \(16\)He, H\. & Thinking Machines Lab\.Defeating nondeterminism in LLM inference\.*Thinking Machines Lab: Connectionism*\(2025\)\.URL[https://thinkingmachines\.ai/blog/defeating\-nondeterminism\-in\-llm\-inference/](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/)\.
- \(17\)Wang, B\., Zhao, D\., Tallent, N\. R\. & Guo, L\.On the reproducibility limitations of RAG systems \(2025\)\.ArXiv:2509\.18869v1 \[cs\.DC\]\.
- \(18\)Ong, J\. C\. L\.*et al\.*Innovating global regulatory frameworks for generative AI in medical devices is an urgent priority\.*npj Digital Medicine*9, 364 \(2026\)\.
- \(19\)Rover, L\., Siqueira, H\. V\., Bacalhau, E\. T\., de Azevedo, A\. T\. & Tadano, Y\. d\. S\.Same prompt, different answer: Exposing the reproducibility illusion in large language model APIs \(2026\)\.Preprint at Research Square, 13 March 2026\.
- \(20\)vLLM Project\.Batch invariance\.vLLM documentation \(2026\)\.URL[https://docs\.vllm\.ai/en/latest/features/batch\_invariance/](https://docs.vllm.ai/en/latest/features/batch_invariance/)\.Beta feature; accessed 2026\-10\-04\.
- \(21\)Qin, J\., Du, A\., Zhang, D\., Lentz, M\. & Zhuo, D\.Vosti: Specifying, implementing, and verifying deterministic LLM inference \(2026\)\.ArXiv:2609\.38981v1 \[cs\.DC\]\.
- \(22\)Zhao, S\.*et al\.*CoRun: Padding is simple and efficient for deterministic LLM inference \(2026\)\.ArXiv:2608\.14376v1 \[cs\.OS\]\.
- \(23\)Gond, R\., Kamath, A\. K\., Ramjee, R\. & Panwar, A\.LLM\-42: Enabling determinism in LLM inference with verified speculation \(2026\)\.Proceedings of the ACM SIGOPS 32nd Symposium on Operating Systems Principles \(SOSP 2026\), pp\. 916–931, doi 10\.1145/3830418\.3843901; arXiv:2601\.17768v2 \[cs\.LG\]\.
- \(24\)Google\.What’s new in Gemini 3\.6 Flash and 3\.5 Flash\-Lite: API changes and parameter updates\.Gemini API documentation\.URL[https://ai\.google\.dev/gemini\-api/docs/whats\-new\-gemini\-3\.6](https://ai.google.dev/gemini-api/docs/whats-new-gemini-3.6)\.States that temperature, top\_p and top\_k are deprecated and ignored from Gemini 3\.6 Flash onward\. Accessed 2026\-10\-14\.
- \(25\)Yuan, J\.*et al\.*Understanding and mitigating numerical sources of nondeterminism in LLM inference \(2025\)\.Advances in Neural Information Processing Systems 38 \(NeurIPS 2025\), pp\. 188045–188077, doi 10\.52202/085713\-5653; arXiv:2506\.09501v2\.
- \(26\)NVIDIA\.cuBLAS 13\.4 documentation, section 2\.1\.4: Results reproducibility\.NVIDIA CUDA documentation \(2026\)\.URL[https://docs\.nvidia\.com/cuda/cublas/index\.html](https://docs.nvidia.com/cuda/cublas/index.html)\.Accessed 2026\-10\-14\.
- \(27\)NVIDIA\.cuDNN backend developer guide: Reproducibility \(determinism\)\.NVIDIA cuDNN documentation \(2026\)\.URL[https://docs\.nvidia\.com/deeplearning/cudnn/backend/latest/developer/misc\.html](https://docs.nvidia.com/deeplearning/cudnn/backend/latest/developer/misc.html)\.Page dated 2026\-10\-07\. Accessed 2026\-10\-14\.
- \(28\)PyTorch\.Reproducibility\.PyTorch 2\.14 documentation \(2026\)\.URL[https://docs\.pytorch\.org/docs/2\.14/notes/randomness\.html](https://docs.pytorch.org/docs/2.14/notes/randomness.html)\.Accessed 2026\-10\-14\.
- \(29\)vLLM Project\.Reproducibility\.vLLM documentation \(2026\)\.URL[https://docs\.vllm\.ai/en/latest/usage/reproducibility/](https://docs.vllm.ai/en/latest/usage/reproducibility/)\.Accessed 2026\-10\-14\.
- \(30\)Shyr, C\.*et al\.*A statistical framework for evaluating the repeatability and reproducibility of large language models \(2026\)\.Preprint at medRxiv, version 3, 25 March 2026\.
- \(31\)Wang, X\.*et al\.*Self\-consistency improves chain of thought reasoning in language models \(2023\)\.ArXiv:2203\.11171v4 \[cs\.CL\]; ICLR 2023\.
- \(32\)Kuhn, L\., Gal, Y\. & Farquhar, S\.Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation \(2023\)\.ArXiv:2302\.09664v3 \[cs\.CL\]; ICLR 2023\.
- \(33\)Farquhar, S\., Kossen, J\., Kuhn, L\. & Gal, Y\.Detecting hallucinations in large language models using semantic entropy\.*Nature*630, 625–630 \(2024\)\.
- \(34\)Joshi, T\., Aggarwal, S\., Saha, A\.*et al\.*Stochastic CHAOS: Why deterministic inference kills, and distributional variability is the heartbeat of artifical cognition \(2026\)\.ArXiv:2601\.07239v2 \[cs\.AI\]\.
- \(35\)Chen, L\., Zaharia, M\. & Zou, J\.How is ChatGPT’s behavior changing over time?*Harvard Data Science Review*6\(2024\)\.
- \(36\)Gao, I\., Liang, P\. & Guestrin, C\.Model equality testing: Which model is this API serving? \(2025\)\.ArXiv:2410\.20247v2 \[cs\.LG\]; ICLR 2025\.
- \(37\)Chauvin, T\., Le Merrer, E\., Taïani, F\. & Tredan, G\.Log probability tracking of LLM APIs \(2026\)\.ArXiv:2512\.03816v2 \[cs\.LG\]; ICLR 2026\.
- \(38\)Anthropic\.Model IDs and versioning\.Claude Developer Platform documentation \(2026\)\.URL[https://platform\.claude\.com/docs/en/about\-claude/models/model\-ids\-and\-versions](https://platform.claude.com/docs/en/about-claude/models/model-ids-and-versions)\.Accessed 2026\-10\-14\.
- \(39\)Anthropic\.A postmortem of three recent issues\.Anthropic Engineering blog \(2025\)\.URL[https://www\.anthropic\.com/engineering/a\-postmortem\-of\-three\-recent\-issues](https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues)\.17 September 2025\. Accessed 2026\-10\-14\.
- \(40\)Blackwell, R\. E\., Barry, J\. & Cohn, A\. G\.Towards reproducible LLM evaluation: Quantifying uncertainty in LLM benchmark scores \(2024\)\.ArXiv:2410\.03492v2 \[cs\.CL\]\.
- \(41\)European Parliament and Council of the European Union\.Regulation \(EU\) 2017/746 on in vitro diagnostic medical devices\.Official Journal of the European Union, L 117, 5 May 2017, p\. 176 \(2017\)\.URL[https://eur\-lex\.europa\.eu/eli/reg/2017/746/oj](https://eur-lex.europa.eu/eli/reg/2017/746/oj)\.Article 10\(7\); Annex II, Section 4; Annex VII, Section 4\.6; Annex VIII, Rule 4\. Accessed 2026\-10\-14\.
- \(42\)Microsoft\.Foundry models lifecycle and support policy\.Microsoft Learn \(2026\)\.URL[https://learn\.microsoft\.com/en\-us/azure/foundry/openai/concepts/model\-retirements](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-retirements)\.Updated 2026\-07\-24\. Accessed 2026\-10\-14\.
- \(43\)OpenAI\.Deprecations\.OpenAI API documentation \(2026\)\.URL[https://developers\.openai\.com/api/docs/deprecations](https://developers.openai.com/api/docs/deprecations)\.Accessed 2026\-10\-14\.
- \(44\)Anthropic\.Model deprecations\.Claude Developer Platform documentation \(2026\)\.URL[https://platform\.claude\.com/docs/en/about\-claude/model\-deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations)\.Accessed 2026\-10\-14\.
- \(45\)International Society for Pharmaceutical Engineering\.ISPE GAMP 5: A risk\-based approach to compliant GxP computerized systems, second edition \(2022\)\.URL[https://ispe\.org/publications/guidance\-documents/gamp\-5\-guide\-2nd\-edition](https://ispe.org/publications/guidance-documents/gamp-5-guide-2nd-edition)\.July 2022\. Accessed 2026\-10\-14\.
- \(46\)International Society for Pharmaceutical Engineering\.ISPE GAMP guide: Artificial intelligence \(2025\)\.URL[https://ispe\.org/publications/guidance\-documents/gamp\-guide\-artificial\-intelligence](https://ispe.org/publications/guidance-documents/gamp-guide-artificial-intelligence)\.July 2025\. Accessed 2026\-10\-14\.
- \(47\)U\.S\. Food and Drug Administration\.Part 11, electronic records; electronic signatures — scope and application: Guidance for industry \(2003\)\.URL[https://www\.fda\.gov/media/75414/download](https://www.fda.gov/media/75414/download)\.Final guidance, August 2003; docket FDA\-2003\-D\-0143\. Accessed 2026\-10\-14\.
- \(48\)European Medicines Agency & U\.S\. Food and Drug Administration\.Guiding principles of good AI practice in drug development \(2026\)\.URL[https://www\.ema\.europa\.eu/en/documents/other/guiding\-principles\-good\-ai\-practice\-drug\-development\_en\.pdf](https://www.ema.europa.eu/en/documents/other/guiding-principles-good-ai-practice-drug-development_en.pdf)\.Released 14 January 2026\.
- \(49\)European Commission\.Eudralex volume 4, Annex 11: Computerised systems \(2011\)\.URL[https://health\.ec\.europa\.eu/document/download/8d305550\-dd22\-4dad\-8463\-2ddb4a1345f1\_en?filename=annex11\_01\-2011\_en\.pdf](https://health.ec.europa.eu/document/download/8d305550-dd22-4dad-8463-2ddb4a1345f1_en?filename=annex11_01-2011_en.pdf)\.Revision 1, in operation from 30 June 2011\.
- \(50\)European Parliament and Council of the European Union\.Regulation \(EU\) 2024/1689 laying down harmonised rules on artificial intelligence \(Artificial Intelligence Act\)\.Official Journal of the European Union, L series, 12 July 2024 \(2024\)\.URL[https://eur\-lex\.europa\.eu/eli/reg/2024/1689/oj](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)\.As amended by Regulation \(EU\) 2026/1744, OJ L, 24 July 2026\.
- \(51\)European Parliament and Council of the European Union\.Regulation \(EU\) 2026/1744 of 8 july 2026 amending regulations \(EU\) 2024/1689, \(EU\) 2018/1139 and \(EU\) 2023/1230 \(digital omnibus on AI\)\.Official Journal of the European Union, L series, 24 July 2026 \(2026\)\.URL[https://eur\-lex\.europa\.eu/eli/reg/2026/1744/oj](https://eur-lex.europa.eu/eli/reg/2026/1744/oj)\.Article 1\(40\)\(b\), replacing Article 113, third paragraph, point \(c\), of Regulation \(EU\) 2024/1689; in force 27 July 2026\. Accessed 2026\-10\-14\.
- \(52\)European Commission\.Eudralex volume 4, Annex 22: Artificial intelligence \(draft for stakeholders’ consultation\) \(2025\)\.URL[https://health\.ec\.europa\.eu/document/download/5f38a92d\-bb8e\-4264\-8898\-ea076e926db6\_en?filename=mp\_vol4\_chap4\_annex22\_consultation\_guideline\_en\.pdf](https://health.ec.europa.eu/document/download/5f38a92d-bb8e-4264-8898-ea076e926db6_en?filename=mp_vol4_chap4_annex22_consultation_guideline_en.pdf)\.Consultation 7 July – 7 October 2025; not final as of October 2026\.
- \(53\)European Medicines Agency\.Good manufacturing practice: Multistakeholder workshop on expert contributions to artificial intelligence guidance development \(Annex 22\), 30 june 2026 — report \(2026\)\.URL[https://www\.ema\.europa\.eu/en/documents/report/report\-multistakeholder\-workshop\-expert\-contributions\-artificial\-intelligence\-guidance\-development\-annex\-22\_en\.pdf](https://www.ema.europa.eu/en/documents/report/report-multistakeholder-workshop-expert-contributions-artificial-intelligence-guidance-development-annex-22_en.pdf)\.EMA/156789/2026; published 2 October 2026; p\. 3\. Accessed 2026\-10\-14\.
- \(54\)European Medicines Agency\.3\-year work plan for the GMP/GDP inspectors working group, january 2026 – december 2028 \(2026\)\.URL[https://www\.ema\.europa\.eu/en/documents/other/3\-year\-work\-plan\-inspectors\-working\-group\_en\.pdf](https://www.ema.europa.eu/en/documents/other/3-year-work-plan-inspectors-working-group_en.pdf)\.Annex 22 target date Q4 2026\. Accessed 2026\-10\-14\.
- \(55\)U\.S\. Food and Drug Administration\.Proposed regulatory framework for modifications to artificial intelligence/machine learning \(AI/ML\)\-based software as a medical device \(SaMD\): discussion paper \(2019\)\.URL[https://www\.fda\.gov/media/122535/download](https://www.fda.gov/media/122535/download)\.Defines a “locked” algorithm as one that provides the same result each time the same input is applied\.
- \(56\)U\.S\. Food and Drug Administration\.Considerations for the regulation of generative AI\-enabled medical devices: discussion paper and request for feedback \(2026\)\.URL[https://www\.fda\.gov/media/194242/download](https://www.fda.gov/media/194242/download)\.Discussion paper; not draft or final guidance\. Accessed 2026\-10\-14\.
- \(57\)Pharmaceuticals and Medical Devices Agency\.Seisei AI wo katsuyo shita iryo kiki no shinsa pointo \(Draft\) \[review points for medical devices using generative AI \(draft\); in japanese\] \(2026\)\.URL[https://www\.pmda\.go\.jp/files/000282779\.pdf](https://www.pmda.go.jp/files/000282779.pdf)\.Draft dated 14 September 2026\. Accessed 2026\-10\-14\.
- \(58\)Zhang, Z\.*et al\.*Deterministic inference across tensor parallel sizes that eliminates training–inference mismatch\.*Proceedings of the 43rd International Conference on Machine Learning*306, 155026–155042 \(2026\)\.ArXiv:2511\.17826v2\.
- \(59\)Cogotti, N\.Noēsis: Deterministic\-first retrieval with two\-tier context hydration for factuality\-critical queries on small local models \(2026\)\.ArXiv:2609\.07663v1 \[cs\.IR\]\.
- \(60\)Bochenek, A\., Protasiewicz, J\. & Pedrycz, W\.RAGdeterm: Deterministic retrieval\-augmented generation for code generation\.*SoftwareX*34, 102636 \(2026\)\.
- \(61\)Allu, U\., Sivaprakash, A\., Singh, P\. & Manocha, A\.Document retrieval\-aware chunking \(D\-RAC\): Universal retrieval\-aware ingestion of enterprise documents via PDF normalization and multimodal markdown conversion \(2026\)\.ArXiv:2609\.24220v1 \[cs\.CV\]\.
- \(62\)Khatchadourian, R\. & Franco, R\.LLM output drift: Cross\-provider validation & mitigation for financial workflows \(2025\)\.ArXiv:2511\.07585v1 \[cs\.LG\]\.
- \(63\)Atil, B\.*et al\.*Non\-determinism of “deterministic” LLM system settings in hosted environments\.*Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems \(Eval4NLP\)*135–148 \(2025\)\.
- \(64\)Ouyang, S\., Zhang, J\. M\., Harman, M\. & Wang, M\.An empirical study of the non\-determinism of ChatGPT in code generation\.*ACM Transactions on Software Engineering and Methodology*34\(2025\)\.
- \(65\)Cai, W\., Shi, T\., Zhao, X\. & Song, D\.Are you getting what you pay for? auditing model substitution in LLM APIs \(2025\)\.ArXiv:2504\.04715v2 \[cs\.CL\]\.
- \(66\)Anadkat, S\.How to make your completions outputs consistent with the new seed parameter\.OpenAI Cookbook \(2023\)\.URL[https://developers\.openai\.com/cookbook/examples/reproducible\_outputs\_with\_the\_seed\_parameter](https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter)\.6 November 2023; marked archived\. Accessed 2026\-10\-04\.
- \(67\)Anthropic\.Glossary: Temperature\.Claude documentation\.URL[https://platform\.claude\.com/docs/en/about\-claude/glossary](https://platform.claude.com/docs/en/about-claude/glossary)\.Accessed 2026\-10\-04\.
- \(68\)Jabbour, S\.*et al\.*Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study\.*JAMA*330, 2275–2284 \(2023\)\.
- \(69\)U\.S\. Food and Drug Administration\.Artificial intelligence\-enabled device software functions: Lifecycle management and marketing submission recommendations\. draft guidance for industry and FDA staff \(2025\)\.URL[https://www\.fda\.gov/media/184856/download](https://www.fda.gov/media/184856/download)\.Draft, 7 January 2025; docket FDA\-2024\-D\-4488; Appendix C\. Accessed 2026\-10\-14\.
- \(70\)Amazon Web Services\.Amazon Bedrock model card: Anthropic Claude Opus 5\.5\.Amazon Bedrock User Guide\.URL[https://docs\.aws\.amazon\.com/bedrock/latest/userguide/model\-card\-anthropic\-claude\-opus\-5\-5\.html](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-5-5.html)\.Lists structured outputs as not supported\. Accessed 2026\-10\-14\.
- \(71\)Google\.Gemini API release notes: entry of 8 October 2026\.Gemini API documentation\.URL[https://ai\.google\.dev/gemini\-api/docs/changelog](https://ai.google.dev/gemini-api/docs/changelog)\.Deprecation of gemini\-3\.5\-flash with automatic routing to gemini\-3\.6\-flash\. Accessed 2026\-10\-14\.

## Supplementary Information

Supplementary Tables S1–S3 give the metrics over all runs of Experiment 1, the routes and accepted settings, and the tokens and estimated costs; Tables S4–S10 give the same\-day route comparisons, the comparison by cache status, the label\-stable prompts and located quotations, the concurrency test, the calls and costs by deployment and date, the infusion\-pump items and the label transitions of the H100 and chunk\-budget runs; Figure S1 gives the probabilities of Tables 2 and 3 with intervals; Tables S11 and S12 give the retrieved\-passage and reduced\-reasoning measurements of 14–15 October; Table S13 the across\-day results of Experiment 1; Table S14 the retrieval baselines; Table S15 every component and parameter of the pipeline; Supplementary Note 1 the prompts and schema; and Supplementary Note 2 the changes made during the study\. The supplementary files listed in Methods \(prompt index and prompts, document manifests, per\-stage hashes and analysis scripts\) will be deposited at Zenodo \(DOI 10\.5281/zenodo\.23282335\)\.

Table S1:Experiment 1 by route, model and tier, computed over all runs of both days \(10 runs per prompt\)\. Azure OpenAI was measured on 7–8 October, Bedrockclaude\-opus\-5\-5and bothclaude\-haiku\-5\-5deployments on 12–13 October, bothgemini\-3\.6\-flashdeployments on 13–14 October, the other commercial runs on 4–5 October\.PP: probability that two runs of the same prompt differ \(mean over prompts\)\. Any: prompts with at least one difference\. Latency in seconds \(median\)\. Rows are in the order of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2), with the tiers of each deployment together\.
Table S2:Experiment 1 routes\. “Settings” lists the determinism\-related settings the route accepted; no reasoning or thinking parameter was sent, so reasoning ran at each provider’s default on every route\.∗AWS documents structured outputs as unsupported for these models on Bedrock\([70](https://arxiv.org/html/2610.21300#bib.bib70)\); the request field was not rejected and every output parsed as JSON, but enforcement of the schema was not confirmed\.†Google documents temperature, top\-p and top\-k as ignored from Gemini 3\.6 Flash onward\([24](https://arxiv.org/html/2610.21300#bib.bib24)\)\.

Table S3:Experiment 1, day 1 \(70 prompts×\\times5 runs\): calls, tokens returned by the provider, median latency, and cost estimated from those tokens at list prices; cache\-read discounts and cache\-write charges are both excluded, and the figures are estimates, not billed amounts\. For our system, the cost is GPU time at the Spot price paid \($2\.25 per hour; about $9\.9 at the on\-demand rate of $5\.85\)\. Rows are in the order of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\.RouteModelCallsInput tokens \(M\)Output tokens \(k\)Median sCost \(USD\)Ours \(self\-hosted\)gemma\-4\-31b35010\.5458\.5≈\\approx3\.8Gemini APIgemini\-3\.8\-flash35010\.53693\.69\.26Agent Platformgemini\-3\.8\-flash35010\.63836\.110\.30Gemini API§gemini\-3\.6\-flash35010\.53565\.69\.21Agent Platform§gemini\-3\.6\-flash35010\.63415\.110\.12Claude APIclaude\-haiku\-4\-535011\.0723\.011\.37Amazon Bedrockclaude\-haiku\-4\-535011\.0683\.712\.49Claude APIclaude\-opus\-5\-535015\.61496\.271\.92Amazon Bedrock‡claude\-opus\-5\-535015\.614910\.071\.91Claude API‡claude\-haiku\-5\-535015\.61883\.36\.25Amazon Bedrock‡claude\-haiku\-5\-535015\.61874\.76\.25OpenAI APIgpt\-6\-astra3509\.7554\.099\.95Azure OpenAI†gpt\-6\-astra3509\.7594\.6110\.19OpenAI APIgpt\-6\-luna3509\.72183\.11\.08Azure OpenAI†gpt\-6\-luna3509\.73293\.81\.25

†Day 1 is 7 October; Data Zone Standard prices\.‡Day 1 is 12 October\.§Day 1 is 13 October\. Prices are the providers’ list prices read on 14 October 2026: the Gemini 3 Flash rows at the introductory rate valid to 31 December 2026, the Agent Platform rows at its non\-global \(US\) rate; the Claude API claude\-haiku\-4\-5 row at the global rate, since that model does not support US\-only inference; the Claude 5\.5 rows and the Bedrock claude\-haiku\-4\-5 row at the US regional rate\. Output tokens include reasoning tokens where the provider reports them \(for gpt\-6\-luna, 185 of 218 thousand through the OpenAI API and 296 of 329 thousand through Azure\); reasoning tokens were reported on every call of the Gemini deployments and of gpt\-6\-luna, on 435 of 700 calls of gpt\-6\-astra and 503 of 700 of claude\-opus\-5\-5 through their own APIs, on 237 of 350 of gpt\-6\-astra through Azure on day 1, on 481 of 700 of claude\-opus\-5\-5 through Bedrock, on 612 and 621 of 700 of claude\-haiku\-5\-5 through the Claude API and Bedrock, and on none of claude\-haiku\-4\-5\. A full run of the 776 pipeline assessments took 20\.3 minutes of one GPU, about $0\.76 at the Spot price\.

Table S4:Same\-day comparison of the two routes of each model \(Experiment 1\)\. For each field, the number of the common prompts on which the field differed in at least one run through route 1 / through route 2, and the two\-sidedppvalue of an exact McNemar test on the paired prompt\-level indicators\. “Day 1” uses the five runs of day 1; “both days” uses all ten runs\.claude\-opus\-5\-5is the 12–13 October measurement through both routes; the OpenAI models are the 20 short prompts of 7–8 October\. The test treats prompts as independent\.
Table S5:Experiment 1 by cache status \(deployments whose responses report cached input tokens\)\. Pairs of runs of the same prompt are grouped by whether neither call, one call or both calls reported a cache hit; for each group, the number of pairs and the share that differed in label / in output\. Through the OpenAI API and Azure OpenAI, 562 of 700 calls on each deployment reported cached tokens\. Agent Platform ran with implicit caching disabled and the Claude routes reported no cache reads, so they are not listed\. Within: pairs on the same day; across: pairs on different days\.Table S6:Experiment 1, day 1: prompts whose label was the same in all five runs \(“stable”\), and among them the prompts whose cited passages, or whose exact quotation strings, still differed; prompts whose label changed, and among them those on which one of the labels was*Cannot determine*; and, over both days, the quotations returned and the share located verbatim \(after normalising white space and case\) in the cited document\. Rows are in the order of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\.Table S7:Concurrency test: the 70 prompts sent five times each from eight concurrent senders to one vLLM server, with batch\-invariant inference \(BI\) enabled or disabled\. Columns: prompts on which the field differed in at least one of the five concurrent runs, and prompts on which any concurrent output differed from the output of the sequential runs of Tables[2](https://arxiv.org/html/2610.21300#Sx2.T2)and[S13](https://arxiv.org/html/2610.21300#Sx5.T13)\(Gemma only\)\. Qwen3\-32B: 57 of the 60 short and mid prompts fit its context of 40,960 tokens\. Qwen3\.8\-27B could not be served with BI\. Batch invariance lengthened the 350 concurrent Gemma calls from 55\.9 to 65\.9 minutes\.Table S8:Experiment 1: calls and estimated cost by deployment and measurement day \(list prices, from provider\-reported tokens; Supplementary Table[S3](https://arxiv.org/html/2610.21300#Sx5.T3)\), with the start of the first call and the end of the last call in Japan Standard Time\. “Calls” counts calls that returned status 200; failed calls \(78 on the Claude API on 4 October, 245 of the interrupted Bedrock attempt on 8 October\) are not counted\. Rows marked “not used” or “not reported” are in the deposited ledger but in no table; the self\-hosted runs are not listed\. The totals of the used rows are $577 through 7 October, $130 on 8 October, $313 on 12–13 October and $39 on 13–14 October\.Table S9:The 12 assessment items of the infusion\-pump corpus: of the 107 candidate sentences of Section 5 of FDA’s infusion pump guidance, the sentence at position⌊107​j/12⌋\\lfloor 107j/12\\rfloor,j=0,…,11j=0,\\ldots,11\(Methods\)\. Each item is sent as the sentence followed by “\(FDA guidance section: …\)” naming its subsection\. The reference run labelled 422 of the 516 pairs*Cannot determine*, 84*Satisfied*and 2*Not satisfied*, with 8 unparsable outputs identical across the four runs and 86 assessments citing at least one quotation\.Table S10:Label transitions from the reference run to the H100 run \(T7\) and to the chunk\-budget 4,096 run \(T8\), over the 776 items\. Rows: reference label; columns: label in the test run \(“none”: the output was cut off at the output limit and has no label\)\. Of the 34 changes on the H100, the model’s own label changed in 23, the verification rule downgraded the H100 label in 8, and the reference label had been downgraded while the H100 label was not in 3; of the 6 changes at the smaller budget, 4 and 2\.000\.10\.10\.20\.2Ours \(self\-hosted\) / gemma\-4\-31bGemini API / gemini\-3\.8\-flashAgent Platform / gemini\-3\.8\-flashGemini API / gemini\-3\.6\-flashAgent Platform / gemini\-3\.6\-flashClaude API / claude\-haiku\-4\-5Amazon Bedrock / claude\-haiku\-4\-5Claude API / claude\-opus\-5\-5Amazon Bedrock / claude\-opus\-5\-5Claude API / claude\-haiku\-5\-5Amazon Bedrock / claude\-haiku\-5\-5OpenAI API / gpt\-6\-astraAzure OpenAI / gpt\-6\-astraOpenAI API / gpt\-6\-lunaAzure OpenAI / gpt\-6\-lunaLabel000\.20\.20\.40\.4Cited passages000\.50\.511Full outputFigure S1:Experiment 1: probability that two runs of the same prompt differ, by field and deployment, within day 1 \(circles\), within day 2 \(squares\) and across days \(triangles\), with 95% percentile bootstrap intervals over the 70 prompts \(Methods\)\. Rows are in the order of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2); the values repeat those of Tables[2](https://arxiv.org/html/2610.21300#Sx2.T2)and[S13](https://arxiv.org/html/2610.21300#Sx5.T13)and add the cited\-passages field and the intervals\.Table S11:Experiment 1 addendum: the pipeline’s own prompts sent to the commercial deployments \(49 device–item pairs, the retrieved six chunks of the reference run, 7,200–8,700 characters; 5 runs on each of 14–15 October 2026, provider\-default reasoning\)\. Columns 3–8: prompts \(of 49\) on which the field differed in at least one of the five runs of day 1; then the probability that two runs differ in label \(day 1, with 95% bootstrap interval\) and in label and output within day 1 / within day 2 / across days\. The pipeline’s own judgment stage returned byte\-identical output for these prompts across nine runs \(Table[3](https://arxiv.org/html/2610.21300#Sx2.T3)\)\. Two calls ofclaude\-opus\-5\-5through the Claude API on day 2 failed and are omitted\.
Table S12:Experiment 1 addendum: reasoning minimised \(the 70 prompts of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2), 5 runs on each of 14–15 October 2026\)\. The eight deployments whose interface allows reasoning to be reduced below the default were re\-measured with the lowest documented setting; all other settings as in Table[S2](https://arxiv.org/html/2610.21300#Sx5.T2)\. Columns as in Supplementary Table[S11](https://arxiv.org/html/2610.21300#Sx5.T11); “Default” repeats the day\-1 label and output counts at provider\-default reasoning \(Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\)\. Reasoning tokens were reported on none of the 5,600 calls except 56 calls ofgemini\-3\.8\-flashthrough Agent Platform\.gpt\-6\-lunawith reasoning effort none reached the 16,384\-token output limit without a parsable answer on 24 of 700 calls through the OpenAI API and 29 through Azure \(each such call counts as differing in every field\);claude\-opus\-5\-5andgpt\-6\-astrareject the setting and are not listed\.
Table S13:Experiment 1 across days \(runs 1–5 on day 1, 6–10 on day 2, all tiers\)\. Columns 3–5: the 70 prompts divided by how the label behaved over the ten runs—the same label in all ten; a label that varied within a day; or the same label in all five runs of each day but a different label on the two days\. Columns 6–7: probability that two runs differ \(all tiers\), for pairs within day 1 / within day 2 / across days; the day\-1 label values repeat the last column of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)for comparison\. Rows are in the order of Table[2](https://arxiv.org/html/2610.21300#Sx2.T2)\.Prompts by label behaviour \(of 70\)PP\(two runs differ\): day 1 / day 2 / acrossRouteModelSame inall 10 runsVariedwithin a dayChanged onlybetween daysLabelOutputOurs \(self\-hosted\)gemma\-4\-31b70000\.000 / 0\.000 / 0\.0000\.000 / 0\.000 / 0\.000Gemini APIgemini\-3\.8\-flash69010\.000 / 0\.000 / 0\.0140\.091 / 0\.071 / 0\.183Agent Platformgemini\-3\.8\-flash68020\.000 / 0\.000 / 0\.0290\.074 / 0\.011 / 0\.519Gemini API§gemini\-3\.6\-flash70000\.000 / 0\.000 / 0\.0000\.463 / 0\.443 / 0\.407Agent Platform§gemini\-3\.6\-flash68200\.009 / 0\.017 / 0\.0130\.484 / 0\.497 / 0\.489Claude APIclaude\-haiku\-4\-566400\.034 / 0\.023 / 0\.0260\.319 / 0\.330 / 0\.361Amazon Bedrockclaude\-haiku\-4\-570000\.000 / 0\.000 / 0\.0000\.011 / 0\.023 / 0\.015Claude APIclaude\-opus\-5\-5561400\.081 / 0\.086 / 0\.0771\.000 / 0\.999 / 1\.000Amazon Bedrock‡claude\-opus\-5\-5561400\.080 / 0\.069 / 0\.0780\.999 / 1\.000 / 0\.999Claude API‡claude\-haiku\-5\-5541600\.091 / 0\.083 / 0\.0941\.000 / 1\.000 / 0\.998Amazon Bedrock‡claude\-haiku\-5\-5511900\.104 / 0\.071 / 0\.1010\.996 / 0\.999 / 0\.997OpenAI APIgpt\-6\-astra61900\.046 / 0\.053 / 0\.0590\.981 / 0\.977 / 0\.989Azure OpenAI†gpt\-6\-astra591100\.077 / 0\.043 / 0\.0620\.979 / 0\.974 / 0\.981OpenAI APIgpt\-6\-luna482200\.103 / 0\.123 / 0\.1070\.999 / 0\.997 / 0\.998Azure OpenAI†gpt\-6\-luna432700\.137 / 0\.114 / 0\.1280\.994 / 0\.999 / 0\.997

†7–8 October;‡12–13 October;§13–14 October; the other commercial routes 4–5 October\.

Table S14:Retrieval baselines\. Queries \(of 33, one per distinct question, scored over all 4,977 chunks\) whose top\-6 results were identical in order to the first index of the same kind: hnswlib \(approximate HNSW\), Elasticsearch approximatekk\-nearest\-neighbour search \(ES kNN\) and Elasticsearch BM25\. “Ours” is the pipeline’s exact, scoped retrieval, which matched in every case\.∗BM25 scores changed for all 33 queries\.

Table S15:The pipeline’s components, parameters and rules \(Experiment 2\), as recorded in the manifests\. All hashes are SHA\-256 over raw bytes, a model’s raw output, or the canonical JSON \(sorted keys\) of a structured value\.### Supplementary Note 1: the prompts of Experiment 1

The system prompt \(SHA\-2561181bf6aa31300b525d1fc20e6d24e0c8e00d27c6b69414d47d6e6db46fcc0d9over its UTF\-8 bytes, with straight quotation marks and the line breaks shown\) is reproduced verbatim:

> You review FDA 510\(k\) premarket documents for a blood glucose monitoring system and decide whether the documents show that the device meets each requirement taken from FDA guidance\. Use only the documents provided\. Decide one requirement at a time\. Choose exactly one label: \- Satisfied: The documents show the requirement is met\. \- Not satisfied: The documents show the requirement is not met\. \- Cannot determine: The documents do not address the requirement\. Answer with JSON only, in this form: \{"label":"<one of the labels\>","comment":"<two or three sentences explaining the decision\>","evidence": \[\{"document": <document number\>,"quote":"<a passage copied verbatim from that document\>"\}\]\}\. Quote passages exactly as they appear in the documents\. If the label is Cannot determine, return an empty evidence list\.

The user prompt is the numbered documents, each introduced by a line “=== Documentnn:title===”, followed by “Requirement:item” and “Answer with JSON only\.”\. The item ends with “\(FDA guidance section:heading\)”, where the heading is in fact that of the device’s decision\-summary section matched to the item, not a section of the guidance; the same text is used in both experiments, so this misnomer does not affect the comparisons\. The JSON schema sent where accepted requires an object with exactly the keyslabel\(one of the three labels\),comment\(string\) andevidence\(an array of objects with an integerdocumentand a stringquote\) and no other properties\. Experiment 2 uses the same label definitions\. All 70 prompts and their SHA\-256 are in the supplementary material \(prompts\_index\.jsonl\); full texts are reconstructible from the public documents\.

### Supplementary Note 2: changes during the study

The measures, operations and conditions were fixed before the runs, and every change was recorded when it was made\. The positive\-control runs \(“Ours”\), the concurrency test, the extension of day 2 to all tiers and the infusion\-pump runs were added during the study and are exploratory\. The changes were as follows\.

Experiment 1\.\(i\) Device–item pairs whose 510\(k\) summary had under 1,000 characters of text \(two PDF portfolios\) were skipped in favour of the next pair\. \(ii\) On Amazon Bedrock,claude\-haiku\-4\-5was measured on 4–5 October in place ofclaude\-opus\-5\-5, so that both Claude routes carried the same model;claude\-opus\-5\-5through Bedrock andclaude\-haiku\-5\-5through both Claude routes were added on 12–13 October, withclaude\-opus\-5\-5through the Claude API sent again on those days for the route comparison \(its 4–5 October measurement remains the one reported in Tables[2](https://arxiv.org/html/2610.21300#Sx2.T2)and[S13](https://arxiv.org/html/2610.21300#Sx5.T13)\)\. An earlier attempt on 8 October was interrupted after 105 Bedrock calls and is not used\. \(iii\) The Claude API credit balance ran out for 16 seconds; the 78 failed calls were resent\. \(iv\) The second day was extended from mid prompts to all tiers, and the self\-hosted judgment stage was added with the same prompts and schedule\. \(v\) Azure OpenAI was measured on 7–8 October through an existing subscription, with the OpenAI API’s short prompts sent on the same days for the route comparison\. \(vi\)gemini\-3\.5\-flashthrough the Gemini API had been planned for the mid prompts only; short and long prompts were added on 7–8 October so that every deployment has the same 70 prompts\. On 13 October, the identifiergemini\-3\.5\-flashreturned the versiongemini\-3\.6\-flashthrough the Gemini API \(it had returnedgemini\-3\.5\-flashon 4–8 October\), while Agent Platform still servedgemini\-3\.5\-flashunder it, so the two routes could no longer be compared on the same days under that identifier; we therefore measuredgemini\-3\.6\-flashthrough both Google routes on 13–14 October and report it in place of the earlier measurement, which remains in the supplementary ledger but appears in no table; Google’s release notes of 8 October 2026 record the deprecation and the automatic routing\([71](https://arxiv.org/html/2610.21300#bib.bib71)\)\. \(vii\) The cost cap was raised in steps from $480 to $1,300\. \(viii\) The count of label changes between days was defined after the data were collected, as in Measures and statistics; a count of prompts with any differing pair of runs across days, used in drafts, also counted variation within a day\. The within\- and across\-day probabilities that two runs differ, defined before the runs, are reported unchanged\. \(ix\) The reduced\-reasoning and retrieved\-passage measurements \(Supplementary Tables[S12](https://arxiv.org/html/2610.21300#Sx5.T12)and[S11](https://arxiv.org/html/2610.21300#Sx5.T11)\) were added on 14 October after internal review, pre\-registered before sending, and run on 14–15 October; their first launch sent no calls because the runner skips completed \(route, model, prompt, run\) tuples, so the reduced\-reasoning runs use run numbers 31–40, mapped to 1–10 in the tables\.

Concurrency test\.\(i\) The test was added after Experiment 1 had begun\. \(ii\) The second model was changed before measurement from Qwen3\.8\-27B, which the serving software refused to run in batch\-invariant mode, to Qwen3\-32B on the 60 short and mid prompts, with thinking disabled and the attention kernel fixed; Qwen3\.8\-27B was measured without batch invariance only\.

Experiment 2\.\(i\) After the dry runs, FDA guidance was removed from the corpus, because the items are guidance sentences and retrieval returned the guidance itself, and the item count was corrected to 776 unique device–item pairs\. \(ii\) Ingestion with 16 concurrent documents stalled, because page rendering in the PDF library did not return when documents were held open across long OCR waits; the extractor was changed to render all OCR pages before calling OCR, extracted text is unchanged, and the rerun matched the reference run produced before the fix\. \(iii\) The other\-GPU test was planned on an A100 80GB, which could not run the reference configuration \(in vLLM 0\.29\.0’s Triton attention backend, used with batch invariance, an FP8 KV cache requires compute capability 8\.9, which the A100 lacks, and the FP8 weight conversion kernel failed to compile\); it was run on an H100 80GB with the same settings and the memory share raised from 0\.62 to 0\.74 to give the same absolute memory\. \(iv\) A run on a later day and the infusion\-pump runs were added\. \(v\) T8 and T9 were added on 10 October, after reading Vosti\([21](https://arxiv.org/html/2610.21300#bib.bib21)\); the smaller chunk budget was changed from 2,048 to 4,096 tokens before measurement because the engine rejected 2,048 for this multimodal model\. \(vi\) After internal review, on 14–15 October, we added a second H100 run on a fresh machine created from a disk image of the reference machine, the repeats of the 4,096 budget at 1, 8 and 16 concurrent items, the server\-restart and interrupted\-ingestion runs, the OCR and embedding ablations and the hnswlib and soft\-delete tests \(Methods\); the OCR ablation was stopped by its two\-hour limit after 336 of 337 documents, which are the documents compared\.