Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact

# Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
Source: [https://arxiv.org/html/2607.25589](https://arxiv.org/html/2607.25589)
Mateusz Kozłowski[https://orcid.org/0009-0005-7785-1342](https://orcid.org/0009-0005-7785-1342) Independent Researcher, Kraków, Poland

###### Abstract

Background:Medical\-imaging AI benchmarks are often represented by a manuscript, analysis tables, executable code, provider APIs, and repository releases\. Agreement between these artifacts is assumed but rarely tested directly\.

Purpose:To determine where an archived chest\-radiograph vision\-language model \(VLM\) pilot diverged from its reported protocol, how those divergences affected identifiability of its claims, and which machine\-verifiable controls could have detected them\.

Materials and Methods:This was a forensic audit of an existing local project snapshot and its public repository record\. No model was called again and no report or image was newly annotated\. We traced executed prompt bindings, inventoried DICOM headers and patient aliases, reconciled planned versus nonempty outputs, inspected the automated label\-extraction path, recomputed matched tests by explicit\(case,finding\)\(\\mathrm\{case\},\\mathrm\{finding\}\)keys, and scanned manuscript, figure, submission, and repository artifacts for propagation failures\. The audit unit was an artifact or scientific claim, not a patient outcome\.

Results:The nominal design contained 300 model\-prompt calls; 297 yielded nonempty reports\. Sixty Claude calls labeled A/B were executed with the same C prompt\. The 30 studies represented 28 patients and included 14 PA, 11 AP, one lateral, and four undocumented views; all four undocumented views were MONOCHROME1 and had been rendered without required polarity inversion\. Dataset split was not retained\. The automated extractor was unvalidated; for five reports it received only the first 4000 characters\. Replacing positional truncation with one common cohort of 369 complete case\-finding blocks changed the historical Cochran statistic fromQ=154\.73Q=154\.73to182\.29182\.29\. The number of McNemar comparisons with nominal unadjustedp<0\.05p<0\.05changed from 26/45 to 27/45; after Holm adjustment, the count changed from 18/45 to 20/45\. These arithmetic corrections did not restore the intended prompt estimand\. Stale model names and results persisted in the released repository bundle after manuscript corrections\.

Conclusion:The corrected table was numerically reproducible, but its conditions differed from the reported experiment\. The original performance and prompt\-effect claims are withdrawn, and the legacy benchmark is invalid for outcome inference\. A benchmark contract binding cohort membership, rendered\-image and prompt hashes, provider\-resolved model identity, call status, annotation provenance, keyed analysis, and derived\-artifact checks is designed to expose these failure classes\. The release implements selected guards with synthetic fixtures; the complete contract has not been prospectively validated\.

Keywords:medical imaging artificial intelligence; vision\-language models; radiology; benchmark integrity; reproducibility audit; research integrity; DICOM rendering; prompt provenance; artifact traceability

## 1Introduction

Medical\-imaging AI studies are often read as if the manuscript were the experiment\. In practice, the reported result is the final product of a longer measurement and software chain\. Cohort\-selection code chooses records and sampling units; DICOM processing determines the model\-visible pixels; a provider adapter binds the prompt, endpoint, model identifier, and decoding configuration; and a second system may convert free text into labels\. Analysis code then selects paired and missing observations, after which build scripts and manual copy operations propagate values into tables, figures, submission packages, and public archives\. Arithmetic may remain internally consistent even when the final result no longer describes the named condition\.

Reporting guidance recognizes many components of this problem\. The 2024 update of the Checklist for Artificial Intelligence in Medical Imaging calls for explicit reporting of data selection, preprocessing, reference standards, partitions, missingness, software, and statistical analysis\[[22](https://arxiv.org/html/2607.25589#bib.bib4)\]\. Radiology\-specific guidance for large language models also emphasizes exact prompts, sampling settings, model versions, and expert evaluation\[[2](https://arxiv.org/html/2607.25589#bib.bib3)\]\. General computational guidance recommends preserving raw observations, recording every transformation, avoiding manual data manipulation, and making analysis steps inspectable\[[20](https://arxiv.org/html/2607.25589#bib.bib6),[21](https://arxiv.org/html/2607.25589#bib.bib7)\]\. Machine\-learning reproducibility programs have likewise shown the value of checklists, code release, and structured artifact review\[[7](https://arxiv.org/html/2607.25589#bib.bib8),[19](https://arxiv.org/html/2607.25589#bib.bib9)\]\.

These practices are necessary, but a list of disclosed components does not prove that the disclosed components were the ones that executed\. Model cards and datasheets describe intended systems and datasets\[[16](https://arxiv.org/html/2607.25589#bib.bib13),[6](https://arxiv.org/html/2607.25589#bib.bib14)\]\. FAIR principles improve the findability and reusability of digital objects\[[24](https://arxiv.org/html/2607.25589#bib.bib10)\], while the W3C PROV model provides a vocabulary for entities, activities, and agents\[[14](https://arxiv.org/html/2607.25589#bib.bib11)\]\. Artifact\-review frameworks ask whether released materials are documented, consistent, complete, and exercisable\[[1](https://arxiv.org/html/2607.25589#bib.bib21)\]\. None of these properties alone establishes that a condition label equals the request bytes sent to a provider, that a radiograph was rendered with the intended polarity, or that the figure in a repository was rebuilt after the underlying result changed\.

Benchmarks that depend on remote vision\-language model \(VLM\) services make these boundaries difficult to inspect\. Provider aliases can change, model snapshots may be unavailable later, and request metadata may not be returned in a form suitable for audit\[[2](https://arxiv.org/html/2607.25589#bib.bib3)\]\. The model sees a rendered image rather than a DICOM header, so an intact source file can still become an incorrect model input\. When another language model converts generated reports into labels, the endpoint combines at least three measurement processes: the clinical source report, the report\-generating VLM, and the label extractor\. A single binary “correct” cell can hide disagreement at any of these layers\.

The project examined here was initially presented as a pilot comparison of two reporting prompts across five VLM families using an enriched sample of 30 MIMIC\-CXR studies\[[10](https://arxiv.org/html/2607.25589#bib.bib1)\]\. A pre\-submission review first appeared to require only numerical corrections\. Deeper inspection showed that the preserved client\-side call path bound both nominal Claude conditions to a third prompt\. Four radiographs had been rendered with reversed polarity, the 30 studies belonged to 28 patients, and final split membership had not been retained\. Three planned outputs failed to yield nonempty reports: two files were zero\-byte and one was absent\. Five reports were truncated before automated extraction, and the matched analysis paired observations by position after condition\-specific deletion\. Corrected source text also failed to propagate into every public and submission artifact\.

The audit can correct reconstructable arithmetic, but it cannot recover an unexecuted prompt contrast or an absent clinical reference standard\. It therefore does not produce a repaired leaderboard and instead examines the divergence between the intended protocol and the preserved artifact system\.

This paper addresses three questions:

1. 1\.At which transitions did the intended, executed, observed, analyzed, reported, and released forms of the benchmark diverge?
2. 2\.Which historical quantities can still be reconstructed, and what is the narrowest interpretation that the preserved evidence permits?
3. 3\.Which machine\-verifiable controls would have converted each silent divergence into a visible failure before publication?

The analysis does not compare model performance\. It reconstructs the archived arithmetic with an evidence hierarchy, explicit\(case,finding\)\(\\mathrm\{case\},\\mathrm\{finding\}\)keys, and one stable matched cohort\. Each claim is then adjudicated separately to distinguish recoverable arithmetic from unsupported scientific interpretation\. The proposed fail\-closed contract covers the cohort, pixels, prompts, model identity, outputs, annotation, analysis, and release\. The accompanying aggregate\-only package contains offline regression tests, figure\-provenance records, privacy guards, and deterministic archive checks\.

## 2Conceptual framework

### 2\.1Six states of a computational result

The audit treats a benchmark result as a sequence of state transitions rather than a single table\. Table[1](https://arxiv.org/html/2607.25589#S2.T1)defines the six states used throughout this paper\. The states are related, but they are not interchangeable\. For example, a prompt may exist in the intended protocol and in a source file, yet never become part of the executed request\. A value may be correct in the current manuscript while an older value remains in the released archive\.

Table 1:States of a computational benchmark and the evidence needed to distinguish them\.Evidence had to match the state under review\. Docstrings documented intent, runtime bindings established prompt execution, status fields summarized outputs, and file bytes established whether data existed\. Current local sources documented the author’s corrected position; archived packages showed what readers had received\. The audit therefore used runtime bindings for prompt identity, DICOM attributes and rendering code for pixel identity, file bytes for completion, explicit scientific keys for pairing, and archived package contents for the public release\.

### 2\.2Identity\-preserving transitions

Each comparison required a different identity check\. Prompt identity was checked against exact request text, and image identity against the selected study, documented view, and rendered pixels\. Matched analyses required a common\(case,finding\)\(\\mathrm\{case\},\\mathrm\{finding\}\)key across columns\. Public claims were linked to the code, input ledger, and release that generated them\.

Checksums verify identity at these transitions, including whether prompt strings differ or figures are byte\-identical\. Separate semantic checks are required to assess whether the prompts define a clinically appropriate contrast and whether a figure supports a justified claim\. The proposed contract therefore combines identity checks with semantic invariants, such as allowed views, unique sampling units, explicit missingness, and a fixed multiplicity family\.

### 2\.3What can be corrected without repeating the experiment

The audit used a three\-level repair boundary\. A transcription or arithmetic error can be corrected when the authoritative inputs are preserved and the transformation is identified\. A measurement or denominator problem can sometimes be reconstructed descriptively, but its interpretation must be restricted to the retained measurement system\. A failure of treatment, input, or reference\-standard identity cannot be repaired by reanalysis because the intended observation does not exist\.

The repair boundary explains why the historical Q and McNemar values are reported while the leaderboard is withdrawn\. The keyed binary matrix permits a well\-defined calculation about that archived matrix\. It does not establish that the columns correspond to the intended prompts, that all images satisfy the inclusion rule, or that the automated labels represent clinical truth\. This audit therefore does not claim to rescue the original experiment\. It studies why that rescue is not identified and how the failure could have been detected earlier\.

## 3Materials and methods

### 3\.1Design, audit unit, and frozen scope

This was a retrospective forensic audit of a project state preserved on 27 July 2026\. The audit was initiated after discrepancies were found during pre\-submission correction; it was not preregistered and did not test a prospectively specified clinical hypothesis\. The unit of investigation was an artifact, transformation, or scientific claim\. A patient outcome was not an audit unit\.

The examined materials included source code, prompt definitions, provider adapters, execution logs where available, a case manifest, DICOM headers, generated\-report file metadata, cached four\-state extractor labels, aggregate confusion\-category cells, analysis code and outputs, manuscript versions, generated tables and figures, multiple submission packages, and the public Zenodo record\[[12](https://arxiv.org/html/2607.25589#bib.bib23)\]\. The audit retained a local copy of the version\-specific released archive\. The relevant local files were read in place\. Record\-level clinical content was not copied into the release workspace\.

The scope was deliberately narrower than the original project\. No provider was called again, no image was rendered again for outcome analysis, and no report or reference label was newly annotated\. The audit did not infer unlogged provider snapshots, reconstruct raw extractor responses that had not been retained, or create a clinical reference standard\. Private development cases previously called “Study 1” were excluded because their scoring rules changed during annotation, one source document was not retained, and the same small case set informed prompt development and evaluation\. Including them would have added non\-reproducible clinical material without strengthening the artifact audit\.

### 3\.2Evidence sources and rules of precedence

Evidence was ranked separately for each audited state, as summarized in Table[2](https://arxiv.org/html/2607.25589#S3.T2)\. Human\-readable labels documented claims about objects; object identity required evidence from the corresponding execution or release boundary\.

Table 2:Evidence rules used to prevent a convenient proxy from replacing the object under audit\.The rules distinguish missing identity metadata from a documented mismatch\. An absent resolved model identifier restricts model\-specific verification without showing that the wrong model ran\. Byte\-identical prompt bindings for conditions intended to differ, by contrast, document that the intended contrast did not execute\.

### 3\.3Audit workflow

#### Execution trace\.

We traced imports, function arguments, mutable module state, adapter calls, and request construction from the batch runner to the provider boundary\. Comments, docstrings, display labels, and nominal condition names were recorded as intent evidence but were not allowed to override the executed binding\. Special attention was given to Python imports by value and to runtime monkey\-patching, because changing a name in one module does not necessarily change a value already bound in another module\. For adapters that accepted the prompt as a function argument, the argument path was followed into the request object\. For the Claude path, the import order and module namespace used by the actual request function were inspected\.

#### Cohort and image audit\.

We matched the retained case manifest to study identity, a pseudonymous patient alias, and the DICOM fields for view position and photometric interpretation\. Counts were made at both study and patient level so that repeated patients could not be mistaken for independent sampling units\. The selector was inspected to determine which official MIMIC\-CXR partitions were eligible and whether final split membership survived in the manifest\. The rendering path was checked for modality transformation, windowing, conversion to eight\-bit pixels, JPEG creation, and conditional polarity handling\. The DICOM standard defines the minimum MONOCHROME1 sample value as white and the minimum MONOCHROME2 value as black\[[17](https://arxiv.org/html/2607.25589#bib.bib20)\]; treating both encodings identically therefore changes the displayed polarity for MONOCHROME1 inputs\. This was a bounded audit of retained headers and preserved code\. Images were not re\-rendered, and the full modality LUT, VOI LUT, window selection, bit\-depth, compression, and display chain was not independently validated\. A blankViewPositionwas classified as undocumented, not as proof of a nonfrontal acquisition\.

#### Output reconciliation\.

The frozen design implied30×5×2=30030\\times 5\\times 2=300calls\. We reconciled every expected combination with the corresponding path, byte count, and logged status\. A report was counted as nonempty only if the expected file existed and contained at least one byte after whitespace handling\. A missing file and a zero\-byte file were not treated as interchangeable, and a success status did not override either condition\. The separate states matter because downstream annotation code can silently convert both states into missing cells while an upstream dashboard still reports a successful call\.

#### Annotation and endpoint audit\.

We inspected the automated extraction code from input construction through JSON parsing and confusion\-category assignment\. The audit recorded the extractor model name present in code, maximum input length, retry and fallback behavior, allowed label states, uncertainty mapping, and false\-positive severity rule\. It also checked whether raw extractor responses, full decoding settings, and a human validation sample were retained\. The four candidate states werePOSITIVE,NEGATIVE,UNCERTAIN, andNOT\_MENTIONED\. The reference side used the 14 CheXpert\-style labels, including*No Finding*\[[9](https://arxiv.org/html/2607.25589#bib.bib2)\]\. The audit did not assume that either side constituted image\-adjudicated truth\.

#### Statistical reconstruction\.

We assigned each retained observation a scientific key formed from its case and finding identifiers\. The key, not row position, determined alignment across the ten nominal model and prompt conditions\. The original analysis path was reproduced separately to identify the source of the published values, then replaced for the corrective calculation by one globally complete matrix\. Alternative routes were retained as named sensitivity calculations rather than silently mixed\. No missing value was imputed, and no missing\-completely\-at\-random assumption was made\.

#### Propagation audit\.

We searched for canonical model names, call counts, cohort counts, and headline statistics across manuscript sources, compiled tables, plotting inputs, figure labels, repository README files, compressed archives, submission variants, and copy\-ready metadata\. We treated the dated Zenodo deposit as the released artifact and inspected a locally preserved copy of its version\-specific archive\. We did not infer its contents from the newest local folder\. This tested whether a correct edit in one source had reached every object presented to readers\.

### 3\.4Historical statistical reconstruction

The reconstruction used the archived confusion\-category matrix\. For each of the 420 possible case\-finding blocks, every nominal condition was reduced to a binary indicator: TP and TN were coded 1, while FN and every FP severity category were coded 0\. A block entered the global matrix only when all ten conditions had a defined binary value\. This produced ann×kn\\times kmatrixY=\(yi​j\)Y=\(y\_\{ij\}\), wherek=10k=10,iiindexes a\(case,finding\)\(\\mathrm\{case\},\\mathrm\{finding\}\)key, andjjindexes a nominal condition\. Equivalently, the base index contained30×14=42030\\times 14=420blocks and 4,200 condition\-specific cells\. The global rule retained 369 blocks and 3,690 condition\-specific cells\.

Cochran’s Q was calculated for the globally complete matrix following the matched\-sample formulation\[[4](https://arxiv.org/html/2607.25589#bib.bib16)\]\. LetCj=∑iyi​jC\_\{j\}=\\sum\_\{i\}y\_\{ij\}be the column totals,Ri=∑jyi​jR\_\{i\}=\\sum\_\{j\}y\_\{ij\}the row totals, andT=∑jCjT=\\sum\_\{j\}C\_\{j\}\. The statistic was

Q=\(k−1\)​k​∑jCj2−T2k​T−∑iRi2,Q=\(k\-1\)\\frac\{k\\sum\_\{j\}C\_\{j\}^\{2\}\-T^\{2\}\}\{kT\-\\sum\_\{i\}R\_\{i\}^\{2\}\},
withk−1=9k\-1=9degrees of freedom\. The survival probability was evaluated from the chi\-square distribution without roundingQQ\.

All\(102\)=45\\binom\{10\}\{2\}=45post\-hoc comparisons used the same rows as the omnibus test\. For a pair of columns,bbcounted keys correct only in the first condition andcccounted keys correct only in the second\. The uncorrected McNemar statistic was\(b−c\)2/\(b\+c\)\(b\-c\)^\{2\}/\(b\+c\), and the continuity\-corrected statistic was\(\|b−c\|−1\)2/\(b\+c\)\(\|b\-c\|\-1\)^\{2\}/\(b\+c\)when discordant observations were present\[[15](https://arxiv.org/html/2607.25589#bib.bib17)\]\. Exact two\-sided binomial and exact mid\-ppvalues were also calculated\. These variants have different small\-sample properties\[[5](https://arxiv.org/html/2607.25589#bib.bib19)\], so agreement of the threshold counts was checked rather than assumed\.

Holm adjustment was applied to the unroundedpp\-values within each complete 45\-comparison family\[[8](https://arxiv.org/html/2607.25589#bib.bib18)\]\. No comparison was removed after seeing its result\. A separate pairwise available\-case route was computed to check the reviewer’s 28/45 value\. None of the four implemented McNemar variants reproduced that count: each yielded 27/45 unadjusted and 20/45 after Holm adjustment\. This route nevertheless targets different observation sets because each pair may use a different denominator; it was therefore excluded from the fixed\-cohort family reported here\.

The statistical target was the historical binary matrix, not clinical accuracy\. CheXpert\-derived report labels are a reference procedure rather than image\-level adjudication\[[9](https://arxiv.org/html/2607.25589#bib.bib2),[22](https://arxiv.org/html/2607.25589#bib.bib4)\], and the candidate labels were generated by an extractor that had not been validated against human readers\. P\-values were consequently used to reproduce the archived calculation, not to reinstate a clinical or comparative claim\.

The 369 retained blocks did not constitute 369 independent clinical sampling units\. Fourteen finding labels were nested within each study, and two patients contributed two studies\. The usual chi\-square reference for Cochran’s Q and binomial reference for McNemar’s test therefore do not have defensible population\-level calibration in this dataset\. The statistic, nominalpp\-value, and threshold counts are preserved solely to reconstruct the legacy arithmetic and to explain the difference among analysis routes\. They are not interpreted as inferential evidence of a model, prompt, or clinical difference\.

### 3\.5Aggregate ledger, figures, and offline test harness

We stored publishable findings in a versioned aggregate JSON ledger governed by JSON Schema\. A dependency\-free validator checks the schema during tests and release preflight; additional tests enforce cross\-field arithmetic, such as whether missing, empty, and nonempty outputs sum to the plan\. Every numerical value shown in Figures[1](https://arxiv.org/html/2607.25589#S4.F1)–[4](https://arxiv.org/html/2607.25589#S4.F4)is read from this ledger rather than copied into plotting code\. Figure[5](https://arxiv.org/html/2607.25589#S5.F5)is explicitly a conceptual build diagram and contains no measured value\. The ledger assigns each aggregate variable an explicit field\[[23](https://arxiv.org/html/2607.25589#bib.bib12)\]\. A companion trace links the ledger and script hashes to each figure hash, caption claim, and stated limitation\.

The local harness contains synthetic record\-level fixtures and public aggregate artifacts; it does not contain restricted clinical records\. It tests request\-side prompt hashes, collapsed nominal conditions, DICOM polarity and view rules, duplicate patients, split retention, terminal output validation, joins on explicit keys, Cochran’s Q, four McNemar variants, Holm adjustment, forbidden claim text, identifier\-like privacy patterns, unsafe archive paths, and manifest verification\. Known small matrices and analytically knownpp\-values support the statistical regression tests\. The suite requires only Python’s standard library; Matplotlib is used to regenerate figures\.

These tests verify control logic, not the historical outcomes\. Because the record\-level historical matrix and MIMIC material cannot be redistributed, a third party can execute the statistical functions and failure guards but cannot independently regenerate the study\-specific 27/45 and 20/45 counts from the release candidate alone\. The candidate states this limit explicitly\.

### 3\.6Claim adjudication

Each headline claim was evaluated against the minimum evidence needed for its scientific meaning, then assigned one of four statuses:

- •*retained as an artifact fact*when preserved files directly established the statement without requiring a clinical interpretation;
- •*restricted*when a calculation was valid only for the archived automated\-label matrix;
- •*unverifiable*when a necessary identity, configuration, or source was not retained;
- •*withdrawn*when the executed condition, input, or measurement did not identify the reported contrast\.

The status was not determined by statistical significance\. A smallpp\-value could coexist with withdrawal if the comparison was not the one reported\. Conversely, an artifact fact could be retained without an outcome test\. When several defects affected one claim, the strongest limitation governed its disposition\. Claim adjudication was author\-led and was not independently replicated\.

### 3\.7Data governance and public\-release boundary

MIMIC\-CXR is credentialed data\. Current PhysioNet guidance for use with online services requires verified zero retention, no training, and no human review, and advises against a service whose handling cannot be confirmed\[[18](https://arxiv.org/html/2607.25589#bib.bib22)\]\. This audit initiated no new transfer and cannot infer the historical provider\-account configuration from source code\. It therefore does not certify historical compliance\.

The public candidate contains no MIMIC image, source report, patient or study identifier, generated report, private development case, or record\-level derivative\. It contains aggregate counts, audit code, schemas, offline tests with synthetic record\-level fixtures and public aggregate or source artifacts, figure code, and publication documentation\. A conservative text scanner is included to catch common identifier patterns\. Passing that scanner does not establish de\-identification or permission to redistribute credentialed data\. Venue\-specific ethics and data\-use requirements must be assessed independently\.

## 4Results

### 4\.1Observed divergences

TableLABEL:tab:divergencessummarizes the observed breaks between reported intent and the preserved artifacts\.

Table 3:Divergences between reported intent and the preserved executable or released artifact\.LayerReported or intendedPreserved evidenceConsequencePrompt routingClaude System A versus BThe pipeline imported the default prompt by value before a later module monkey\-patch\. The preserved client\-side call path bound both labels to byte\-identical System C\.The 60 labeled A/B calls identify C/C variability, not a prompt effect\.PhotometryStandard grayscale radiographsFour inputs were MONOCHROME1; the renderer did not invert them\.Model\-visible polarity was reversed for four studies\.View selectionFrontal radiographsHeaders contained 14 PA, 11 AP, one LL, and four blank view positions\.The claimed frontal cohort was not executed\.Sampling unit30 independent casesThe manifest contained 30 studies from 28 patients; two patients each contributed two studies\.Independence and the claimed participant count were unsupported\.PartitionOfficial test setThe selector admitted both test and validation studies; the retained manifest did not store split\.Final partition composition is not recoverable from the release\.CompletenessNear\-complete output setOne Gemini B report was absent and two Qwen A files were zero bytes despite anokstatus\.The correct file\-level count is 297/300\.ExtractionComplete automated annotationThe extractor truncated input at 4000 characters; five Claude reports exceeded that limit\. Raw API responses and decoding settings were not retained\.Candidate labels cannot be fully audited or independently reconstructed\.ScoringClinical FP severity and one binary conventionSeverity was assigned deterministically by finding name\.*No Finding*was pooled with pathologies; generated uncertainty was silently non\-positive\.Severity was not clinical; endpoint directions depended on coding choices\.Matched testsEqual\-length paired vectorsMissing entries were dropped by condition and remaining vectors truncated by position\.Observations after the first gap were mispaired\.ReleaseCorrected source propagated to all artifactsStale model names and values remained in repository tables, figures, and parallel submission packages\.The public bundle did not describe the corrected manuscript\.![Refer to caption](https://arxiv.org/html/2607.25589v1/x1.png)Figure 1:Observed audit findings along the archived artifact path\.The upper row summarizes protocol identity; the lower row follows derived measurements through release\. Arrows show artifact dependency, not estimated causal effects\. Every count comes from the public aggregate ledger\. Recalculation can repair arithmetic but cannot restore unrecorded prompt\-condition or model identity\.
### 4\.2Condition identity and prompt routing

The batch runner exposed two nominal prompt keys, A and B, and passed the selected string as an argument to four provider adapters\. The Claude route was different\. It first imported the report function from a separate pipeline module, then attempted to replace a prompt variable in the originalpromptsmodule before calling that function\. The pipeline module had already imported the default prompt by value\. Reassigning the original module attribute did not update the name bound inside the pipeline module, whose request construction continued to use the default System C text\.

This control\-flow path applied to 30 studies under two nominal Claude conditions, accounting for 60 affected calls\. Filenames, progress labels, and analysis columns still recorded A and B, but both conditions were bound to prompt C\. Repeated execution under one prompt can describe run\-to\-run output variation; it cannot identify an A\-versus\-B effect because the treatment did not vary\.

The preserved Python import and call path deterministically establishes what the function read under the recorded import order\. Because provider\-side request payloads were not retained, this is an inference from code rather than direct API\-log evidence\. The release candidate tests the corresponding guard with synthetic payloads and does not present historical request bytes\.

### 4\.3Cohort, sampling unit, and pixel identity

The four MONOCHROME1 images were exactly the four images with undocumented view position\. Adding the single LL image produced five studies that failed the reported requirement for a documented AP or PA input\. The selector ranked PA ahead of AP and other values, but it did not enforce a frontal\-only filter\. Blank view metadata therefore survived selection\. A blank value does not prove that an acquisition was nonfrontal; it shows that the released evidence cannot establish the reported frontal criterion\.

The same four blank\-view studies were encoded as MONOCHROME1\. The preserved renderer applied windowing and converted the result to eight\-bit JPEG without the conditional polarity inversion required by that photometric interpretation\. The model\-visible grayscale polarity was therefore reversed for those four inputs\. The audit did not attempt to estimate how this changed a particular output\. The scientific problem occurs earlier: nominal calls using those four study inputs did not receive the image representation defined by the intended benchmark\.

Two patients each contributed two studies\. One contributed aMONOCHROME1study and the LL study; the other contributed two otherwise frontal studies\. Thus the most compact description is 30 studies from 28 patients, not 30 independent patients\. Repeated studies need not be forbidden in every design, but they require a declared sampling unit, patient\-level split policy, and clustered analysis\. None was preserved here\.

The partition claim was likewise unresolved\. The selection code admitted both the official test and validation partitions, while the retained manifest did not store final split membership\. This establishes that “test only” was not enforced\. The final mixture was not reconstructed in this audit and is excluded from the release candidate; controlled re\-linkage to official split metadata may be possible if a credentialed review preserves the necessary restricted keys\.

### 4\.4Output completeness and the post hoc boundary

Of 300 planned report files, 297 were nonempty\. Because two Qwen A files were zero bytes and one Gemini B file was absent, affecting three different studies, available denominators differed by condition\. A count of 299 resulted from treating zero\-byte files with loggedokstatus as completed\. The file\-level reconciliation distinguishes these states because an optimistic status can otherwise propagate into a claimed denominator even when no report is available for annotation\.

A mechanical post hoc eligibility subset would remove the four undocumented, non\-inverted MONOCHROME1 studies and the LL study, leaving 25 AP/PA MONOCHROME2 studies\. After also removing the two non\-identifiable Claude columns, eight nominal conditions would remain\. Those 25 studies supplied 350 potential case\-finding blocks\. Eight blocks had an uncertain reference label\. A remaining study lacked an entire Qwen A output, affecting 14 blocks, two of which overlapped the uncertain\-reference set\. The union was therefore8\+14−2=208\+14\-2=20incomplete blocks, leaving 330 complete blocks from 24 studies\.

This calculation is a technical eligibility boundary, not a corrected benchmark\. The exclusions were defined after inspecting the defects, the final partition was still unknown, and the candidate\-label measurement remained unvalidated\. Reporting a new leaderboard from the 330 blocks would create a post hoc experiment that was neither planned nor clinically adjudicated\. No such table was produced\.

![Refer to caption](https://arxiv.org/html/2607.25589v1/x2.png)Figure 2:Aggregate accounting of views, report files, and matched case\-finding blocks\. Panel \(a\) shows the retained DICOM header counts\. Panel \(b\) distinguishes a nonempty file from zero\-byte and missing outputs\. Panel \(c\) uses one 420\-block frame for all ten nominal conditions\. The visualization contains no record\-level membership data\.
### 4\.5Annotation semantics

The extractor assigned each report one of four labels per finding:POSITIVE,NEGATIVE,UNCERTAIN, orNOT\_MENTIONED\. For five reports it received only the first 4000 characters\. Per\-report label caches survived, but raw extractor responses and the complete historical decoding configuration did not\. The released aggregate retained only the derived TP, TN, FN, or severity\-coded FP cell\.

Table[4](https://arxiv.org/html/2607.25589#S4.T4)makes the historical reduction explicit\. Candidate uncertainty was treated as non\-positive rather than preserved as a third state in the binary analysis\. Reference uncertainty excluded the block\. These choices are analytic conventions, not clinical facts\.

Table 4:Historical mapping from report\-level label states to the archived binary endpoint\. Severity suffixes were assigned by finding name, not by clinical adjudication\.*No Finding*supplied 10 of 38 positive reference cells in the pooled 14\-label endpoint despite having opposite semantics to the 13 pathology labels\. Removing it reversed the reported A/B direction for one nominal model family\. Treating generatedUNCERTAINas positive instead of non\-positive reversed the direction for another nominal model family\. These sensitivity analyses show that the archived direction depended on endpoint conventions; they do not establish that either mapping is clinically correct\. The non\-identifying totals are frozen in the public aggregate ledger, while study\-level verification requires the retained local caches\.

The severity labels were not adjudicated by a radiologist\. Of 551 historical FP cells, 550 were classified minor and one moderate by a deterministic finding\-name rule; none was severe\. All clinical interpretations of this variable were therefore withdrawn\. The endpoint measures agreement between two automated report\-labeling paths\. It cannot determine whether the image supports a finding, whether the source report omitted a visible abnormality, or whether the extractor misunderstood the generated text\.

### 4\.6Reconstructed historical arithmetic

The archived matrix contained 420 possible case\-finding blocks\. Exactly 369 were complete across all ten nominal conditions\. The original procedure first deleted missing values independently within each condition and then truncated all remaining vectors to the shortest length,n=383n=383\. Equal length did not restore matching\. If one condition lacks key 2, for example, its second retained value belongs to key 3 while the second value of a complete condition still belongs to key 2\. Every position after the first asymmetric gap can therefore compare different case\-finding blocks\.

After joining by key, one common cohort of 369 blocks was used for all calculations\. It yielded Cochran’sQ=182\.28998Q=182\.28998with 9 degrees of freedom and a nominalp=1\.684×10−34p=1\.684\\times 10^\{\-34\}\. For every implemented McNemar variant, 27 of 45 unadjusted comparisons had nominalp<0\.05p<0\.05, and 20 of 45 remained below0\.050\.05after Holm adjustment\. The threshold counts were identical for the asymptotic, continuity\-corrected, exact binomial, and exact mid\-ppvariants; the public aggregate ledger does not identify whether the same pairs crossed the threshold in every variant\.

The reviewer\-suggested 28/45 count was not reproduced\. Allowing each pair to use its own available blocks still yielded 27/45 unadjusted comparisons under the asymptotic, continuity\-corrected, exact binomial, and exact mid\-ppvariants; each route also yielded 20/45 after Holm adjustment\. Pair\-specific deletion is a legitimate descriptive route under an appropriate analysis plan, but here it changes the observations across contrasts\. The audit selected the common 369\-block set so that Cochran’s Q and the full 45\-test family referred to one archived cohort\. Neither route repairs prompt\-condition identity, potentially informative missingness, or within\-study clustering\.

These values correct the reportedQ=154\.73Q=154\.73, 26/45 raw, and 18/45 adjusted counts\. They do not validate the benchmark: the ten columns include two mislabeled Claude conditions, invalid image representations, an incompletely preserved partition, and unvalidated extracted labels\. The nominalpp\-values are also uncalibrated for the nesting of findings within studies and repeated studies within two patients\. They are retained as forensic outputs of a historical analysis, not evidence of population heterogeneity\.

![Refer to caption](https://arxiv.org/html/2607.25589v1/x3.png)Figure 3:Three versions of the historical arithmetic\. The original report used positional truncation\. The reviewer claim of 28 raw comparisons was not reproduced: all four pairwise available\-case variants yielded 27 raw and 20 Holm\-adjusted comparisons\. The keyed reconstruction used one globally complete cohort for the omnibus test and all 45 McNemar comparisons\. These values describe archived automated labels, not clinical performance\.
### 4\.7Propagation into released artifacts

The propagation audit found that source correction and public correction occurred separately\. At the time of inspection, the Zenodo record with DOI10\.5281/zenodo\.21427298containedreproducibility\_bundle\_REVISED\.zip, dated 18 July 2026\. It predated later corrections\. Stale model names remained in a README, generated results tables, performance figures, and the runner source\. The archive did not contain the provider adapters needed to resolve the contradictory names\.

The stale labels also appeared in parallel journal and arXiv sources and in copy\-ready metadata\. Several manually maintained publication branches remained internally consistent while describing an obsolete state\. Correcting the local manuscript did not change the bytes attached to the public record\.

The audit used a locally preserved copy of the inspected release\. The corrective record provides a version\-specific file inventory and checksums that distinguish the audited bundle from later public changes\.

### 4\.8Impact on scientific claims

Table[5](https://arxiv.org/html/2607.25589#S4.T5)separates findings that remain supportable from claims that the retained evidence cannot identify\.

Table 5:Disposition and public verifiability of the original headline claims\.The audit reports arithmetic tied to preserved inputs and withholds claims that require valid clinical measurement or prompt\-condition identity\. Numerical correction alone does not establish that the associated conclusion remains valid\.

### 4\.9Executable safeguards

A clean local run completed all 36 offline tests\. They use synthetic record\-level fixtures, the public aggregate ledger, and the public source tree; no restricted clinical record enters the suite\. The tests require only Python’s standard library, while Matplotlib is needed to regenerate the figures\. Table[6](https://arxiv.org/html/2607.25589#S4.T6)links each family to the historical failure it is designed to expose\.

Table 6:Offline regression\-test inventory\. Record\-level fixtures are synthetic; aggregate and source\-tree checks use release\-candidate artifacts\.![Refer to caption](https://arxiv.org/html/2607.25589v1/x4.png)Figure 4:Distribution of the 36 offline tests\. Solid bars denote controls over scientific identity; hatched bars denote release\-integrity controls\. Counts are an inventory, not code coverage or clinical validation\.

## 5A machine\-verifiable benchmark contract

### 5\.1Design principles and threat model

The observed failures arose through different mechanisms, but each involved an unchecked mapping between a label and the object it was meant to identify\. The contract checks four transitions: planning resolves sampling\-unit, image, prompt, and model identity; execution creates complete atomic records; analysis requires annotation provenance and scientific keys; and release rebuilds and reconciles derived artifacts\.

“Fail closed” means that an unmet invariant changes the pipeline state to a named failure and prevents downstream publication\. It does not mean that every unexpected observation is deleted\. A lateral image, repeated patient, or length\-limited response may be scientifically useful in a protocol that anticipates it\. The requirement is that the protocol declare how it is handled before inference and that the implementation enforce that declaration\.

The threat model covers accidental drift, stale configuration, incomplete materialization, ambiguous missingness, unsafe joins, and failed propagation\. It excludes deliberate fabrication by an actor able to alter source data and manifests together, and it does not assess whether the clinical question is valid\. The contract instead preserves the identity of the objects that a benchmark claims to compare and exposes divergence before result generation\.

### 5\.2Eight contract clauses

1. 1\.Cohort contract\.Each sampling unit receives an immutable case key, a pseudonymous patient key, source and split provenance, declared eligibility fields, and an inclusion decision\. The contract must state whether repeated patients are prohibited or handled by a patient\-level split and clustered analysis\. In the audited project, this clause would have rejected an undocumented split, unexpected view values under the stated frontal\-only rule, and an unacknowledged difference between studies and patients before any API call\.
2. 2\.Pixel contract\.Preprocessing parameters, relevant DICOM attributes, software version, conditional photometric transformation, and the final rendered\-image hash are recorded\. Every paired model or prompt condition must receive the same frozen bytes unless image transformation is itself the planned treatment\. A syntactically valid hash establishes identity, not image quality; separate modality\-specific tests and visual checks remain necessary\.
3. 3\.Prompt contract\.The nominal condition points to an immutable expected prompt text and hash\. Every call records the text and hash at the request boundary, after template expansion and adapter logic\. A test must inspect the actual request object or a faithful interceptor, because reading a module constant would not have detected the import\-binding defect observed here\. Conditions expected to differ fail if their executed prompt hashes collapse\.
4. 4\.Model contract\.The planned target, provider endpoint, requested identifier, provider\-resolved identifier, SDK and adapter version, decoding settings, timestamp, and request ID remain linked to each call\. Mutable aliases such as “latest” are rejected unless the provider response resolves and records a stable snapshot\. When a provider does not expose a resolved identifier, the limitation is explicit and model\-version claims are restricted\.
5. 5\.Output contract\.A planned\-call ledger is reconciled with exactly one terminal record per call\. Completion requires nonempty bytes, a request ID, and a terminal finish state\. Empty, transport\-failed, safety\-blocked, invalid, and length\-limited responses remain distinct statuses\. Files are written atomically with sidecar metadata so that existence alone cannot masquerade as successful completion\.
6. 6\.Annotation contract\.The clinical reference, candidate extraction, reader decisions, adjudication, extractor request and raw response, model and decoding configuration, text\-length handling, and uncertainty policy remain separate fields\. A derived TP, TN, FN, or FP cell is generated from these fields and never replaces them\. This clause makes sensitivity to*No Finding*, uncertainty, and severity mapping inspectable instead of embedding those choices in one irreversible label\.
7. 7\.Analysis contract\.Joins operate on explicit scientific keys\. The analysis plan names the sampling unit, missingness rule, denominator, clustering structure, multiplicity family, test variant, effect measure, and random seed where applicable\. Invariants verify that an omnibus test and its post\-hoc family use the declared cohort\. A changing denominator can still be allowed, but only as an explicitly named estimand rather than an unnoticed side effect of list deletion\.
8. 8\.Release contract\.One versioned registry supplies canonical names and corrective values to manuscript text, tables, figures, metadata, and archive builders\. Every generated artifact records its input and script hash\. A closed manifest rejects unlisted or changed files, while text guards reject withdrawn claims and stale headline values\. Under this contract, each release receives a version and checksum; folder contents and modification times do not define release state\.

### 5\.3Reference implementation and current coverage

The release candidate is a reference implementation of selected guards, not a validated framework\. Table[7](https://arxiv.org/html/2607.25589#S5.T7)distinguishes executable controls from specifications so that 36 passing tests are not mistaken for complete coverage of all eight clauses\.

Table 7:Implementation status of the proposed benchmark contract in the corrective release candidate\.The fileartifact\_contract\.jsonencodes fail\-closed text rules for submission artifacts\. It is not a complete machine\-readable representation of the eight\-clause benchmark contract\. A dependency\-free validator enforces the JSON Schema during tests and release preflight, while Python tests enforce cross\-field arithmetic that the schema does not express conveniently\.

### 5\.4Practical adoption and limits

A small project can implement the contract with a frozen CSV or JSON call plan, standard\-library hashes, adapter tests of request objects, one sidecar per call, composite\-key joins, and whitelist release builds\. The specific tools may vary, provided that each state transition has a machine\-checkable precondition and leaves evidence for the next state\.

Hashes document whether an object changed and whether an archive matches a frozen list\. Expert review must still assess clinical appropriateness, consent, data\-use compliance, and statistical adequacy\. A reproducible pipeline may still implement an invalid experiment\. The contract makes the evidence chain inspectable for review but does not establish validity by itself\.

![Refer to caption](https://arxiv.org/html/2607.25589v1/x5.png)Figure 5:Local release gates and outcomes for the corrective artifacts\.Versioned aggregate evidence, synthetic fixtures, and source files pass through four local gates before the Zenodo and arXiv archives are built\. If any gate fails, neither archive is built\. These gates verify the local package, not the external platforms\.

## 6Discussion

### 6\.1Interpretation of the reconstructed arithmetic

Joining observations by explicit keys produced one historical Q value and the same 27/45 unadjusted and 20/45 Holm threshold counts across the four implemented McNemar variants\. Those aggregate counts answer a narrow forensic question about one declared reduction of the archived matrix\. They do not show that the columns represent the intended prompt\-condition identities, that every row represents an eligible image, or that a binary agreement cell represents a clinical judgment\.

The estimand also depends on the treatment or condition, sampling unit, input representation, measurement procedure, and population to which the result refers\. Here, failures in prompt\-condition identity, input eligibility, sampling\-unit definition, partition provenance, and outcome measurement were cumulative\. A corrected formula cannot reconstruct components that were never created or were not preserved\.

The nominal Q statistic therefore does not restore the original claim\. Even before accounting for nested findings, it describes differences among columns with compromised identities\. The correction should report the historical arithmetic accurately and withdraw interpretations unsupported by condition and measurement identity\.

### 6\.2Why the failures interacted

These failures interacted\. Prompt routing and DICOM handling altered the conditions and inputs\. Output state and extraction rules determined which measured cells existed, and positional truncation changed how those cells were paired\. Because the defects were not isolated, this audit does not estimate the contribution of any single one\.

Consider the prompt defect\. The repository contained three real prompt strings and the runner printed two plausible condition labels\. Output directories and analysis columns were named consistently with A and B\. Most artifact\-level checks would therefore see a balanced design\. Only execution\-path tracing showed that a value imported into another module remained C after the attempted patch\. Because the prompt defined the experimental condition, the binding error changed the executed comparison\. A one\-call adapter test asserting the request text or hash would have exposed the failure before 60 outputs were produced\.

The DICOM finding has the same structure: MONOCHROME1 changes how sample values should be displayed\[[17](https://arxiv.org/html/2607.25589#bib.bib20)\], so omitting polarity inversion broke the equivalence between the source file and the model\-visible input\. The audit did not re\-render or re\-evaluate the images; the warranted conclusion is that the model inputs for four studies fell outside the intended pipeline, not that a particular performance penalty occurred\.

Missingness linked execution to analysis because file status, byte count, reference uncertainty, and candidate\-label availability determined whether a block entered the matrix\. Pairwise available\-case analysis and the globally complete matrix used different denominators, but all four implemented pairwise variants and the selected common\-cohort analysis yielded 27/45 raw and 20/45 Holm\-adjusted comparisons\. The reviewer’s 28/45 value was therefore not reproduced\. Explicit keys and a named missingness rule make the analysis sets and this numerical discrepancy visible\.

### 6\.3Measurement language and the term hallucination

The original project used report\-derived labels as both reference and candidate measurements\. That design can be useful for scalable report comparison, but it does not directly observe image\-level correctness\. A source report may omit a visible finding, describe it with uncertain language, or reflect a different acquisition context\. A generated report may express the same concept with wording that an extractor misses\. The extractor may then add a third disagreement\.

Pooling*No Finding*with 13 pathology labels further shows that the meaning of a binary endpoint depends on its construction\. A positive*No Finding*label has different semantics from a positive pathology label\. Likewise, treating candidate uncertainty as non\-positive is one possible decision rule, not a neutral fact\. The observed directional sensitivities demonstrate that these conventions mattered in the archived matrix, but they do not establish which convention is clinically preferable\.

The manuscript therefore uses*automated report\-label discordance*rather than*hallucination*\. Recent radiology taxonomy work also calls for a precise definition of the evaluated contingency\[[3](https://arxiv.org/html/2607.25589#bib.bib5)\]\. The narrower term describes the measurement without implying image\-level adjudication, clinical severity, or a causal explanation for the generated text\.

### 6\.4Release engineering is part of the method

The public\-release failure shows that source correction and release correction require separate verification\. Correcting one authoritative source did not update the repository ZIP, generated tables, figures, parallel submission variants, or copy\-ready metadata\. Without a machine\-readable dependency graph, the author must identify and update each dependent object manually\. The version\-specific Zenodo bundle inspected in the audit confirms that the public correction was incomplete\.

A whitelist build, input and script hashes, a closed manifest, and archive verification identify the source state that produced each released object\. These controls make corrections traceable and allow reviewers to establish which artifact they assessed; they do not validate the underlying scientific claim\.

### 6\.5Relationship to existing reproducibility frameworks

The proposed contract complements the reporting, documentation, stewardship, provenance, and artifact\-review frameworks discussed in the Introduction\. The contract adds identity checks at execution boundaries: request hashes link prompt descriptions to request bytes; rendered\-image hashes link DICOM selections to model\-visible inputs; scientific keys preserve case\-finding identity across conditions; and release manifests link public archives to source states\. These controls add machine\-verifiable provenance where narrative documentation does not establish object identity\.

The leakage problem was not limited to partition membership\[[11](https://arxiv.org/html/2607.25589#bib.bib15)\]\. The condition label remained in filenames after the prompt binding collapsed\. Accounting treated anokstatus as completion even when a file was empty, and one manuscript was corrected while the public release retained the old value\. These are provenance failures even when no classic train\-test leakage occurs\.

### 6\.6What can be published without rerunning the study

Without a rerun, the preserved evidence supports a forensic case study, not a new benchmark claim\. A new benchmark paper would require a prospectively frozen protocol, valid inputs, logged model and prompt identities, human\-validated measurement, an appropriate sampling plan, and calibrated analysis\. None is created by this audit\. Presenting corrected performance estimates would therefore exceed the evidence\.

A forensic case study can document how preserved artifacts diverged, determine which historical calculations remain reconstructable, define the boundary between correction and non\-identifiability, and test controls against synthetic reproductions of the failure classes\. These contributions require explicit evidence grading and exclusion of outcomes that the retained evidence does not identify\. When the evidence does not identify a replacement estimate, the correction should state that limit\. Here, the post hoc 330\-block boundary shows how far mechanical cleaning could proceed; because the subset was selected after the defects were known, it is not reported as a new benchmark estimate\.

### 6\.7Implications for authors, reviewers, and repositories

For authors, several controls can be applied before expensive inference: validate cohort and patient units, freeze rendered\-image hashes, intercept one request from each adapter, and assert distinct condition hashes\. During inference, record resolved model identity and terminal output states\. Before analysis, require explicit keys and print a missingness table\. Before release, rebuild from a whitelist and verify the archive after extraction\.

Reviewers cannot be expected to reverse\-engineer every pipeline, but they can ask boundary questions that expose weak provenance: Where was the prompt captured, before or after adapter logic? Are the reported cases patients, studies, images, or finding blocks? Does every paired comparison use the same keys? What does a successful call mean when a file is empty? Can the authors show that the downloadable archive was built from the submitted source?

Editors and repositories can support these checks by accepting compact machine\-readable ledgers, preserving version history, displaying relations between corrected and superseded records, and encouraging executable artifact review\. Persistent citation metadata does not validate a record’s contents or show that a later version reproduces an earlier state\. Versioned correction notices and checksums help readers distinguish record persistence from content validation\.

### 6\.8Limitations and researcher reflexivity

This is a single case from one small radiology VLM project\. It cannot estimate how often similar failures occur in medical AI, and the proposed controls should not be interpreted as a prevalence\-derived taxonomy\. The same author designed the original project, performed the audit, adjudicated claims, and prepared this report\. That access enabled detailed reconstruction but creates risks of selective attention and confirmation bias\. The audit was retrospective, not preregistered, and no independent laboratory repeated it\.

Several conclusions rely on evidence that is stronger than a manuscript label but weaker than a complete historical log\. The C/C prompt result follows deterministically from the preserved import path, yet provider\-side payloads were not retained\. Model\-resolved identifiers and historical account settings remain incomplete\. The DICOM review established a missing MONOCHROME1 inversion in preserved code but did not validate the entire rendering chain\. Cached extractor labels permitted semantic sensitivity checks, while raw responses and full decoding settings could not be reconstructed\.

The release candidate intentionally excludes record\-level MIMIC material, so readers cannot independently regenerate the aggregate results from the candidate\. Controlled verification by an appropriately credentialed reviewer may be necessary\. The included privacy scanner is only a guardrail and does not certify data\-use compliance\.

The historical matrix has an additional statistical limitation\. Fourteen finding blocks are nested within each study, and two patients contributed repeated studies\. The nominal Q and McNemar reference distributions do not account for this clustering\. No population\-level effect size, confidence interval, or power calculation is offered because the outcome estimand itself is not being reclaimed\. The reportedpp\-values are forensic reproductions of historical arithmetic, not a corrected inferential analysis\.

The benchmark contract was derived after observing these failures\. Its offline tests show that selected validators react to synthetic record\-level fixtures and the public aggregate or source artifacts; they do not establish prospective validity of the complete framework\. Model and annotation clauses remain specifications, the JSON and text contracts do not encode every semantic requirement, and the workflow has not been tested by an independent project\. Future work should prospectively implement the contract, measure false alarms and missed failures, and evaluate whether the controls remain usable at larger scale\.

## 7Conclusion

The audited benchmark diverged from its reported identity at the prompt, pixel, cohort, output, annotation, analysis, and release layers\. Recomputing the statistics corrected three historical values; the condition, input, cohort, and measurement defects remained\. The paper therefore withdraws all clinical\-performance, safety, ranking, and prompt\-effect claims\. The publishable contribution without rerunning the study is the documented failure chain and a contract designed to make the observed failure classes machine\-detectable before release\. Its reference validators are testable now; prospective validation remains future work\.

## Data and code availability

Release candidate 1\.1\.0 contains audit code, 36 offline tests, a dependency\-free JSON Schema validator, an artifact contract, aggregate forensic findings, versioned figure\-generation code, a privacy scanner, a gated deterministic archive builder, and SHA\-256 manifests\. Record\-level test fixtures are synthetic; other checks read only the aggregate ledger or source\-tree artifacts included in this candidate\. The tests run with Python’s standard library\. The figure generator reads all numerical values from the aggregate ledger; Figure 5 is conceptual\.

The reproducibility boundary has three levels\. Public verification covers aggregate accounting, schema and invariant checks, statistical algorithms on synthetic matrices, traced figure generation, claim and privacy guards, manuscript compilation, and archive checksums\. Controlled verification may cover the record\-level historical matrix, DICOM headers, and local label caches for an appropriately credentialed reviewer\. Provider\-side payloads, raw extractor responses, and some resolved model identities were not retained and cannot be reconstructed at either level\. Final split membership was absent from the audited manifest and is unavailable publicly; a controlled re\-linkage may be possible if the necessary restricted keys survive\.

The release candidate intended for public deposit excludes MIMIC images, source reports, patient or study identifiers, generated reports, private development cases, and record\-level derivatives\.

## Declarations

#### Ethics and governance\.

No new model inference, annotation, or data collection was performed for this audit, and no credentialed clinical material is redistributed\. Historical PhysioNet/provider compliance has not been independently verified\. No claim of ethics exemption is made\.

#### Funding\.

This work received no external funding\.

#### Competing interests\.

The author declares no competing interests\.

#### Author contributions\.

Mateusz Kozłowski: conceptualization, investigation, methodology, software, validation, visualization, data curation, and writing\.

#### Use of generative AI\.

OpenAI Codex assisted with code review, synthetic test scaffolding, figure generation code, and language editing\. It did not supply clinical labels or new experimental observations\. The author inspected the preserved evidence, verified numerical outputs and cited sources, reviewed every generated artifact, and retains responsibility for the manuscript\.

## References

- \[1\]Association for Computing Machinery\(2020\-08\)Artifact review and badging, version 1\.1\.Note:[https://www\.acm\.org/publications/policies/artifact\-review\-and\-badging\-current](https://www.acm.org/publications/policies/artifact-review-and-badging-current)Accessed 27 July 2026Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p3.1)\.
- \[2\]C\. Bluethgen, D\. Van Veen, C\. Zakka, K\. E\. Link, A\. H\. Fanous, R\. Daneshjou, T\. Frauenfelder, C\. P\. Langlotz, S\. Gatidis, and A\. Chaudhari\(2025\)Best practices for large language models in radiology\.Radiology315\(1\),pp\. e240528\.External Links:[Document](https://dx.doi.org/10.1148/radiol.240528)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1),[§1](https://arxiv.org/html/2607.25589#S1.p4.1)\.
- \[3\]F\. J\. Brooks and M\. A\. Anastasio\(2026\)A taxonomy of machine hallucination in radiology\.Radiology: Artificial Intelligence8\(2\),pp\. e250203\.External Links:[Document](https://dx.doi.org/10.1148/ryai.250203)Cited by:[§6\.3](https://arxiv.org/html/2607.25589#S6.SS3.p3.1)\.
- \[4\]W\. G\. Cochran\(1950\)The comparison of percentages in matched samples\.Biometrika37\(3–4\),pp\. 256–266\.External Links:[Document](https://dx.doi.org/10.1093/biomet/37.3-4.256)Cited by:[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p2.3)\.
- \[5\]M\. W\. Fagerland, S\. Lydersen, and P\. Laake\(2014\)Recommended tests and confidence intervals for paired binomial proportions\.Statistics in Medicine33\(16\),pp\. 2850–2875\.External Links:[Document](https://dx.doi.org/10.1002/sim.6148)Cited by:[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p5.6)\.
- \[6\]T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daumé III, and K\. Crawford\(2021\)Datasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.External Links:[Document](https://dx.doi.org/10.1145/3458723)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p3.1)\.
- \[7\]B\. J\. Heil, M\. M\. Hoffman, F\. Markowetz, S\. Lee, C\. S\. Greene, and S\. C\. Hicks\(2021\)Reproducibility standards for machine learning in the life sciences\.Nature Methods18,pp\. 1132–1135\.External Links:[Document](https://dx.doi.org/10.1038/s41592-021-01256-7)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1)\.
- \[8\]S\. Holm\(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.External Links:[Link](https://www.jstor.org/stable/4615733)Cited by:[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p6.1)\.
- \[9\]J\. Irvin, P\. Rajpurkar, M\. Ko, Y\. Yu, S\. Ciurea\-Ilcus, C\. Chute, H\. Marklund, B\. Haghgoo, R\. Ball, K\. Shpanskaya, J\. Seekins, D\. A\. Mong, S\. S\. Halabi, J\. K\. Sandberg, R\. Jones, D\. B\. Larson, C\. P\. Langlotz, B\. N\. Patel, M\. P\. Lungren, and A\. Y\. Ng\(2019\)CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison\.Proceedings of the AAAI Conference on Artificial Intelligence33\(01\),pp\. 590–597\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v33i01.3301590)Cited by:[§3\.3](https://arxiv.org/html/2607.25589#S3.SS3.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p7.1)\.
- \[10\]A\. E\. W\. Johnson, T\. J\. Pollard, S\. J\. Berkowitz, N\. R\. Greenbaum, M\. P\. Lungren, C\. Deng, R\. G\. Mark, and S\. Horng\(2019\)MIMIC\-CXR, a de\-identified publicly available database of chest radiographs with free\-text reports\.Scientific Data6,pp\. 317\.External Links:[Document](https://dx.doi.org/10.1038/s41597-019-0322-0)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p5.1)\.
- \[11\]S\. Kapoor and A\. Narayanan\(2023\)Leakage and the reproducibility crisis in machine\-learning\-based science\.Patterns4\(9\),pp\. 100804\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100804)Cited by:[§6\.5](https://arxiv.org/html/2607.25589#S6.SS5.p2.1)\.
- \[12\]M\. Kozłowski\(2026\)Reproducibility bundle: iterative prompt calibration of a vision\-language model system for automated radiology report generation\.Zenodo\.Note:[https://zenodo\.org/records/21427298](https://zenodo.org/records/21427298)Historical release record, DOI 10\.5281/zenodo\.21427298; the audit used a locally preserved copy of the version\-specific bundle inspected in July 2026Cited by:[§3\.1](https://arxiv.org/html/2607.25589#S3.SS1.p2.1),[Data and code availability](https://arxiv.org/html/2607.25589#Sx1.p4.1)\.
- \[13\]M\. Kozłowski\(2026\)VLM radiology prompt\-calibration benchmark: versioned reproducibility archive\.Zenodo\.Note:[https://doi\.org/10\.5281/zenodo\.21629849](https://doi.org/10.5281/zenodo.21629849)External Links:[Document](https://dx.doi.org/10.5281/zenodo.21629849)Cited by:[Data and code availability](https://arxiv.org/html/2607.25589#Sx1.p4.1)\.
- \[14\]T\. Lebo, S\. Sahoo, D\. McGuinness, K\. Belhajjame, J\. Cheney, D\. Corsar, D\. Garijo, S\. Soiland\-Reyes, S\. Zednik, and J\. Zhao\(2013\-04\)PROV\-O: the PROV ontology\.W3C RecommendationWorld Wide Web Consortium\.External Links:[Link](https://www.w3.org/TR/prov-o/)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p3.1)\.
- \[15\]Q\. McNemar\(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.External Links:[Document](https://dx.doi.org/10.1007/BF02295996)Cited by:[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p5.6)\.
- \[16\]M\. Mitchell, S\. Wu, A\. Zaldivar, P\. Barnes, L\. Vasserman, B\. Hutchinson, E\. Spitzer, I\. D\. Raji, and T\. Gebru\(2019\)Model cards for model reporting\.InProceedings of the Conference on Fairness, Accountability, and Transparency,pp\. 220–229\.External Links:[Document](https://dx.doi.org/10.1145/3287560.3287596)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p3.1)\.
- \[17\]National Electrical Manufacturers Association\(2026\)DICOM PS3\.3: information object definitions, image pixel module\.Note:[https://dicom\.nema\.org/medical/dicom/current/output/chtml/part03/sect\_C\.7\.6\.3\.html](https://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.7.6.3.html)Current online edition, accessed 27 July 2026Cited by:[§3\.3](https://arxiv.org/html/2607.25589#S3.SS3.SSS0.Px2.p1.1),[§6\.2](https://arxiv.org/html/2607.25589#S6.SS2.p3.1)\.
- \[18\]PhysioNet\(2025\-09\)Use of MIMIC data with large language models and online services\.Note:[https://physionet\.org/news/post/llm\-responsible\-use/](https://physionet.org/news/post/llm-responsible-use/)Accessed 27 July 2026Cited by:[§3\.7](https://arxiv.org/html/2607.25589#S3.SS7.p1.1)\.
- \[19\]J\. Pineau, P\. Vincent\-Lamarre, K\. Sinha, V\. Lariviere, A\. Beygelzimer, F\. d’Alché\-Buc, E\. Fox, and H\. Larochelle\(2021\)Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program\.Journal of Machine Learning Research22\(164\),pp\. 1–20\.External Links:[Link](https://jmlr.org/papers/v22/20-303.html)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1)\.
- \[20\]G\. K\. Sandve, A\. Nekrutenko, J\. Taylor, and E\. Hovig\(2013\)Ten simple rules for reproducible computational research\.PLoS Computational Biology9\(10\),pp\. e1003285\.External Links:[Document](https://dx.doi.org/10.1371/journal.pcbi.1003285)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1)\.
- \[21\]M\. Taschuk and G\. Wilson\(2017\)Ten simple rules for making research software more robust\.PLoS Computational Biology13\(4\),pp\. e1005412\.External Links:[Document](https://dx.doi.org/10.1371/journal.pcbi.1005412)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1)\.
- \[22\]A\. S\. Tejani, M\. E\. Klontzas, A\. A\. Gatti, J\. T\. Mongan, L\. Moy, S\. H\. Park, and C\. E\. Kahn\(2024\)Checklist for artificial intelligence in medical imaging \(CLAIM\): 2024 update\.Radiology: Artificial Intelligence6\(4\),pp\. e240300\.External Links:[Document](https://dx.doi.org/10.1148/ryai.240300)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p2.1),[§3\.4](https://arxiv.org/html/2607.25589#S3.SS4.p7.1)\.
- \[23\]H\. Wickham\(2014\)Tidy data\.Journal of Statistical Software59\(10\),pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.18637/jss.v059.i10)Cited by:[§3\.5](https://arxiv.org/html/2607.25589#S3.SS5.p1.1)\.
- \[24\]M\. D\. Wilkinsonet al\.\(2016\)The FAIR guiding principles for scientific data management and stewardship\.Scientific Data3,pp\. 160018\.External Links:[Document](https://dx.doi.org/10.1038/sdata.2016.18)Cited by:[§1](https://arxiv.org/html/2607.25589#S1.p3.1)\.