Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor

# Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor
Source: [https://arxiv.org/html/2609.03221](https://arxiv.org/html/2609.03221)
Manpreet SinghDeepak Parasharand Rahul Joshi††thanks:Manuscript submitted August 29, 2026\.*\(Corresponding author: Deepak Parashar\.\)*R\. R\. Bellibaltu and M\. Singh share first authorship\. R\. R\. Bellibaltu \(ORCID 0009\-0003\-6083\-0364\) is with Florida International University \(e\-mail: rohithreddybc98@gmail\.com\)\. M\. Singh \(ORCID 0000\-0003\-2368\-2377\) is with Boston University, Boston, MA 02215 USA \(e\-mail: manni@bu\.edu\)\. Rahul Joshi is with Symbiosis International University\(e\-mail: rahulj@sitpune\.edu\.in\)\. Deepak Parashar is with Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, India \(deepak\.parashar@manipal\.edu\)\. This work received no specific funding\. Code and protocol availability are stated in the Data and Code Availability section\. This work has been submitted to the IEEE for possible publication\. Copyright may be transferred without notice, after which this version may no longer be accessible\.

###### Abstract

Counterfactual audits are the standard tool for checking whether a clinical agent treats demographically distinct but clinically identical patients differently\. They report a flip rate: how often an action changes when only the patient descriptor changes\. We show that this quantity is uninterpretable on its own\. Re\-running an identical condition ten times over sixteen vignettes \(same narrative, same descriptor string, nothing varied\) moved a clinical agent’s action in8\.7%8\.7\\%of outcome\-vignette cells, and instability was heterogeneous across actions by a factor of eight, from0\.0220\.022for ICU escalation to0\.1790\.179for controlled\-substance caution\. No demographic contrast in our data was distinguishable from that floor\. A second model gives a pooled floor of6\.7%6\.7\\%and ranks the six actions almost identically \(Spearman0\.940\.94, exactp=0\.017p=0\.017\), so the floor is not one system’s artefact\. Majority\-vote aggregation over five draws removes39%39\\%of it and then flattens, and a null simulation attributes the residue to heterogeneous per\-cell rates, so replication mitigates without eliminating\. Any counterfactual fairness estimate reported without a per\-action floor beside it therefore cannot be read as evidence of disparity\. The measurements were taken with FairMedAgent, an evaluation harness for disparity in the*actions*of clinical LLM agents whose estimand, the within\-range counterfactual flip rate, counts only flips between actions a published decision rule admits and a clinician has adjudicated\. That estimand requires band adjudication, which is under way; no disparity result is claimed here\. Each synthetic vignette runs a six\-stage trajectory \(five model\-facing decisions around a deterministic environment step\) under fixed\-form conditions spanning race, sex, age, insurance, English proficiency, and their intersections\. The harness, the floor protocol, and every analysis script are released\.

###### Index Terms:

Agentic AI, clinical decision support, large language models, algorithmic fairness, health equity, counterfactual evaluation, measurement validity, instability floor, reproducibility, benchmark, FairMedAgent, trustworthy AI\.

## IIntroduction

Demographic disparities in clinical care are well documented and consequential: Black patients receive less analgesia for equivalent pain\[[1](https://arxiv.org/html/2609.03221#bib.bib12),[2](https://arxiv.org/html/2609.03221#bib.bib13)\], race and sex shift recommendations for cardiac catheterization\[[3](https://arxiv.org/html/2609.03221#bib.bib14)\], and patients with limited English proficiency \(LEP\) experience more adverse events\[[4](https://arxiv.org/html/2609.03221#bib.bib15)\]\. As large language models \(LLMs\) move into clinical decision support, they risk reproducing these disparities: LLMs propagate race\-based medical misconceptions\[[5](https://arxiv.org/html/2609.03221#bib.bib8)\], generate stereotyped assessments and plans\[[6](https://arxiv.org/html/2609.03221#bib.bib6)\], and vary care recommendations by sociodemographic label even when clinical content is held fixed\[[7](https://arxiv.org/html/2609.03221#bib.bib5)\]\.

Clinical AI is simultaneously shifting from single\-response assistants to*agentic*systems that autonomously plan, invoke tools, and take sequential actions under human\-defined constraints\[[8](https://arxiv.org/html/2609.03221#bib.bib21)\]\. In such systems the locus of potential harm is the*action*\(the acuity assigned, the test ordered, the medication chosen, the disposition documented\), and a bias introduced at one step can propagate to and compound with later ones\. Yet existing fairness audits evaluate a single model response, and existing clinical\-agent benchmarks are built to measure capability instead \(Sec\.[II](https://arxiv.org/html/2609.03221#S2)\)\. Neither carries an acceptable\-action band, so neither can separate a change between two defensible clinical choices from a change into a wrong one, and neither supplies reusable infrastructure for auditing that distinction\.

The measurements reported here were taken withFairMedAgent, an evaluation harness for demographic fairness in the actions of multi\-step clinical LLM agents, which we release alongside them\. It renders synthetic vignettes, whose acceptable\-action bands are derived from published decision rules and adjudicated by a blinded clinician \(Sec\.[IV\-B](https://arxiv.org/html/2609.03221#S4.SS2)\), under fixed\-form counterfactual demographic conditions, holding the clinical content frozen and the tool\-result fixture independent of the condition, so that divergence is attributable to the demographic descriptor \(Sec\.[III\-B](https://arxiv.org/html/2609.03221#S3.SS2)states the condition under which the returned results are also identical\)\. It reports counterfactual flip rate, mean absolute score difference, and a headline*within\-range*disparity that isolates demographic sensitivity from clinical error\.

Our contributions are:

- •A measured instability floor, and evidence that counterfactual flip rates cannot be read without one\.Re\-running an identical condition ten times moved the agent’s action in8\.7%8\.7\\%of outcome\-vignette cells over43204320comparisons, with instability strongly heterogeneous across actions: escalation to intensive care moved in2\.2%2\.2\\%of its720720comparisons while the controlled\-substance caution flag moved in17\.9%17\.9\\%\. A second model gives a pooled floor of6\.7%6\.7\\%and ranks the six actions almost identically\. Majority\-vote aggregation over five draws removes under half and plateaus, and a null simulation attributes the residue to heterogeneous per\-cell rates and to nothing further\. We give the protocol, the released scripts, and the four\-step procedure that follows from them \(Sec\.[V\-B](https://arxiv.org/html/2609.03221#S5.SS2)\)\.
- •A fairness estimand that separates bias from clinical error\.Existing counterfactual audits score whether an action changed, conflating a change between two defensible clinical choices with a change into a wrong one\. The within\-range counterfactual flip rate counts only the former, conditioning on both demographic variants falling inside a band of actions that a published decision rule admits and a blinded clinician has adjudicated\. General agent\-action fairness was introduced by AgentFairBench\[[9](https://arxiv.org/html/2609.03221#bib.bib1)\], our own prior general\-domain benchmark, and single\-decision clinical audits target one output\[[10](https://arxiv.org/html/2609.03221#bib.bib2),[11](https://arxiv.org/html/2609.03221#bib.bib3),[12](https://arxiv.org/html/2609.03221#bib.bib4)\]; none carries an acceptable\-action band, so none expresses this estimand\. This is a claim about coverage, not about priority, and Table[I](https://arxiv.org/html/2609.03221#S2.T1)enumerates the axes\.
- •The clinical trajectory as the setting where the estimand matters most\.A band is defined per action, so an episode with five sequential actions carries five of them and a within\-range disparity can accumulate without any single decision being wrong\. Five bands are five opportunities for that, which is not the same as establishing it happens: independent mean\-zero shifts cancel instead of compounding\. Accumulation requires the per\-step shifts to share a sign, so we report a trajectory\-level signed composite against what independent shifts would produce, descriptively, since a single per\-vignette latent reproduces its maximal value with no carry\-forward at all\. The identified estimand is the forced\-upstream contrast, which the fixed phase order makes possible by letting the upstream action be*set*instead of observed\. We contribute first\-class LEP and intersectional axes mapped to PROGRESS\-Plus\[[13](https://arxiv.org/html/2609.03221#bib.bib35)\]\.
- •Reproducible infrastructure\.A pip\-installable harness with a blinded multi\-clinician labeling protocol \(specified in full; one clinician has reviewed the draft bands to date\), small\-cluster\-appropriate statistics, and a sealed\-split evaluation protocol with a reference leaderboard implementation and an adoption log, released on completion of band adjudication, reported per TRIPOD\-LLM\[[14](https://arxiv.org/html/2609.03221#bib.bib28)\]\.

## IIRelated Work

Bias in clinical language models\.A growing body of work documents demographic bias in medical LLMs: race\-based medical misconceptions\[[5](https://arxiv.org/html/2609.03221#bib.bib8)\], stereotyped differentials and recommendations from GPT\-4\[[6](https://arxiv.org/html/2609.03221#bib.bib6)\], race detectable from redacted notes with downstream effects on recommendations\[[15](https://arxiv.org/html/2609.03221#bib.bib9)\], equity harms surfaced by adversarial medical QA\[[16](https://arxiv.org/html/2609.03221#bib.bib7)\], bias in clinical decision\-support prompting\[[17](https://arxiv.org/html/2609.03221#bib.bib10)\], and persistence in reasoning models\[[18](https://arxiv.org/html/2609.03221#bib.bib11)\]\. Counterfactual audits that hold clinical content fixed while swapping a protected attribute include CLIMB’s clinical\-bias metrics\[[11](https://arxiv.org/html/2609.03221#bib.bib3)\], counterfactual patient variations for diagnosis\[[12](https://arxiv.org/html/2609.03221#bib.bib4)\], and a single\-decision gender\-swap audit of LLM emergency\-department triage\[[10](https://arxiv.org/html/2609.03221#bib.bib2)\]\. Most closely, Omar*et al\.*\[[7](https://arxiv.org/html/2609.03221#bib.bib5)\]vary3131sociodemographic labels across1,0001\{,\}000emergency cases \(holding clinical content constant\) and find systematic differences in urgency, imaging, and mental\-health referral\. These establish that bias exists; each evaluates a*single*generated recommendation or decision\.

Agentic and benchmark infrastructure\.A parallel line builds clinical\-agent and benchmark platforms: AgentClinic simulates clinical environments\[[19](https://arxiv.org/html/2609.03221#bib.bib16)\], MedAgentBench benchmarks electronic health record \(EHR\) tool\-use agents\[[20](https://arxiv.org/html/2609.03221#bib.bib17)\], and DRAGON\[[21](https://arxiv.org/html/2609.03221#bib.bib18)\], MedS\-Bench\[[22](https://arxiv.org/html/2609.03221#bib.bib19)\], and MedHELM\[[23](https://arxiv.org/html/2609.03221#bib.bib20)\]provide leaderboards for clinical capability\. These measure*accuracy/capability*\. Demographic disparity in actions is outside what they were built to detect\.

Positioning\.General agent\-action fairness across non\-clinical and triage domains was recently benchmarked by our own AgentFairBench\[[9](https://arxiv.org/html/2609.03221#bib.bib1)\], and capability\-oriented clinical\-agent benchmarks measure accuracy instead\. What neither line supplies is an acceptable\-action band, and without one the distinction this paper is built on cannot be drawn \(Table[I](https://arxiv.org/html/2609.03221#S2.T1)\)\. Our lead contribution is*within\-range*disparity, a fairness estimand conditioning on all variants being jointly clinically acceptable\. Single\-recommendation designs such as Omar*et al\.*\[[7](https://arxiv.org/html/2609.03221#bib.bib5)\]do not express it, because they carry no acceptable\-action band to separate demographic sensitivity from clinical error\. We are precise about the scope of that claim: nothing about the estimand*requires*multiple steps, and a single\-turn audit that elicited one action and scored it against a band would express it too\. What the trajectory adds is that bands are per\-action, so disparity can accumulate across an encounter, and that a fixed phase order makes the upstream action manipulable\. We further contribute first\-class LEP and intersectional axes, the analgesia and documentation action domains, contamination canaries, and cross\-step propagation, and treat triage and diagnostic ordering, both already studied single\-turn, as replication in the agentic setting, where an early flip can constrain later actions\. The sealed\-split leaderboard is*infrastructure*operationalizing standing measurement of within\-range disparity; we do not present it as a standalone novelty \(leaderboards per se are established\[[21](https://arxiv.org/html/2609.03221#bib.bib18),[22](https://arxiv.org/html/2609.03221#bib.bib19),[23](https://arxiv.org/html/2609.03221#bib.bib20)\]\)\.

Trustworthiness and assurance\.Work on autonomous clinical agents is increasingly framed around safety, alignment with human\-defined constraints, and accountability\[[8](https://arxiv.org/html/2609.03221#bib.bib21),[24](https://arxiv.org/html/2609.03221#bib.bib34)\]\. FairMedAgent operationalizes these: clinician\-defined acceptable\-action bands encode the human constraints, within\-range disparity is intended to measure alignment under those constraints, and the adoption log supports accountability: assurance measurement infrastructure that frameworks such as the NIST AI Risk Management Framework \(AI RMF\)\[[24](https://arxiv.org/html/2609.03221#bib.bib34)\]and TRIPOD\-LLM\[[14](https://arxiv.org/html/2609.03221#bib.bib28)\]call for but do not yet supply for agent actions\.

TABLE I:Representative prior work on counterfactual and clinical\-agent evaluation, with the axis this paper is about in the final column\. “–” means the axis was not a design goal of that work, so this is not a capability ranking\.Agent: multi\-step, state\-carrying tool\-using agent;CF: counterfactual demographic design;LEP: limited\-English\-proficiency axis;Int: disaggregated estimates for named intersectional cells;Real: evaluates real patients, not synthetic cases, an axis where prior work leads and this work does not;Floor: reports a measured instability floor for the same actions, on the same cases, in the same run\. ✓ = yes, ~ = partial, – = no; every mark records what that work published\. No prior counterfactual audit of a clinical model reports a floor, which is the gap this paper addresses rather than a deficiency of those studies: the quantity was unavailable to them, and its size unknown\. The benchmark instrument these measurements were taken with, including the within\-range estimand, is described in Sec\.[III](https://arxiv.org/html/2609.03221#S3); results from it await band adjudication and are not claimed here\.
## IIIBenchmark Design

### III\-AProblem Formulation: Disparity in Agent*Actions*

Prior audits of clinical language models assess a single generated response: a differential diagnosis, an answer, or one recommendation\[[7](https://arxiv.org/html/2609.03221#bib.bib5),[6](https://arxiv.org/html/2609.03221#bib.bib6),[16](https://arxiv.org/html/2609.03221#bib.bib7),[5](https://arxiv.org/html/2609.03221#bib.bib8)\]\. Deployed decision support increasingly does something else\. It operates as a multi\-step*agent*that plans, invokes tools, and commits sequential actions\[[19](https://arxiv.org/html/2609.03221#bib.bib16),[20](https://arxiv.org/html/2609.03221#bib.bib17)\]\. We therefore evaluate the agent as a policyMMthat, given a patient presentation, emits a trajectory of discrete*actions*instead of one output: an order set, an admission decision, an analgesia tier, a disposition\. FairMedAgent measures whether these actions differ for demographically distinct but clinically identical patients, and whether such differences compound across steps\.

### III\-BThe Agentic Loop

Each evaluation runs a*six\-stage*trajectory over a base vignettevv\(Fig\.[1](https://arxiv.org/html/2609.03221#S3.F1)\)\. Five stages are model\-facing decision calls; one is a deterministic environment step: \(i\)*triage*\(Emergency Severity Index \[ESI\] acuity11–55, urgency00–100100\); \(ii\)*ordering*\(a set of labs/imaging\); \(iii\) an*environment step*\(no model call\) that returns a fixed result fixtureρ⁡\(v\)\\rho\(v\)for the ordered items; \(iv\)*reassessment*\(admit/discharge, intensive\-care\-unit escalation\) conditioned onρ⁡\(v\)\\rho\(v\); \(v\)*prescribing*\(analgesia tier00–33, controlled\-substance caution\); and \(vi\)*documentation*\(referral, follow\-up interval, stigmatizing\-language flags\)\. Five model calls are thus made per vignette–condition\. By*agent*we mean a policy that makes sequential, state\-carrying, action\-committing decisions\. An early action, a triage level or an order set, is carried forward and constrains later ones\. Single\-turn audits have no such structure\. The phase order is fixed, not model\-selected, and that is a deliberate measurement control: demographic conditions could otherwise trigger different tool paths and confound attribution of disparity to the descriptor\. Adaptive planning is a scoped extension \(Sec\.[VI\-A](https://arxiv.org/html/2609.03221#S6.SS1)\)\.

Attribution rests on two complementary guarantees\. For the pre\-results decisions \(triage, ordering\), invariance is expected because the clinical narrative is frozen and the descriptor is rendered in constant grammatical form \(Sec\.[III\-C](https://arxiv.org/html/2609.03221#S3.SS3)\), with clinically confounded attribute×\\timestask pairs pre\-registered and excluded from bias claims\. For the post\-results decisions \(reassessment, prescribing, documentation\), an*additional*guarantee holds, but conditionally: the fixtureρ⁡\(v\)\\rho\(v\)is a pure function of the ordered item and the vignette, with no dependence on the demographic condition\. The*realized*results areρ⁡\(v\)\\rho\(v\)evaluated on the order set the agent actually produced, so they are identical across conditions exactly when the order sets coincide\. For those trajectories, whose share we report alongside every propagation estimate, post\-results divergence cannot be explained by different information reaching the agent\. Where the ordering step itself flips, different information does reach the agent; that pathway is not a confound to be removed but the propagation channel of Sec\.[V\-D](https://arxiv.org/html/2609.03221#S5.SS4), and the two strata are analyzed separately\. Together these attribute trajectory divergence to the descriptor, which is the property separating FairMedAgent from single\-turn audits\.

\(i\) triageacuity, urgency\(ii\) orderingorder set\(iii\) environmentfixture results\(iv\) reassessdisposition\(v\) prescribeanalgesia tier\(vi\) documentfollow\-up, referraldemographic descriptorc∈𝒞c\\in\\mathcal\{C\}fixed form; only slot fillers varySolid arrows carry state forward as scalars\. Dashed arrows show the descriptor rendered afresh at each of the five model calls; the*single\-prime*arm removes all but the first, separating carry\-forward from re\-priming\. Step \(iii\) is deterministic: it injects the vignette’s own fixture and makes no model call, so the returned results cannot depend on the descriptor\. Propagation is identified by*forcing*the upstream action at \(i\) or \(ii\) and re\-running the downstream steps, rather than conditioning on whether it happened to flip\.Fig\. 1:The six\-stage evaluation trajectory\. Five model\-facing decisions surround one deterministic environment step\. Because the clinical narrative is frozen and the fixture at \(iii\) is descriptor\-independent, any divergence between two conditions is attributable to the descriptor alone\.
### III\-CCounterfactual Demographic Conditions

Counterfactual fairness\[[25](https://arxiv.org/html/2609.03221#bib.bib22),[26](https://arxiv.org/html/2609.03221#bib.bib23)\]holds that a decision should be invariant under a change to a protected attribute with all else fixed\. Our design instantiates it as a*metamorphic relation*\[[27](https://arxiv.org/html/2609.03221#bib.bib24),[28](https://arxiv.org/html/2609.03221#bib.bib25)\]: profiles identical except for the demographic descriptor should yield identical actions, and violations are fairness bugs\. Each vignette is rendered under a condition set𝒞\\mathcal\{C\}that varies only a*fixed\-form*, constant\-grammar one\-line demographic descriptor while the clinical narrative is frozen\. Protected attributes, selected and justified against PROGRESS\-Plus\[[13](https://arxiv.org/html/2609.03221#bib.bib35)\], are: race and ethnicity as a single slot, sex \(binary levels only\), age, insurance \(Medicaid vs\. private\), and limited English proficiency \(LEP\), together with two named intersectional cells \(Black woman on Medicaid; Hispanic man on Medicaid with LEP\)\. We report these cells as disaggregated estimates, which is a weaker commitment than an intersectional analysis in the methodological sense\[[29](https://arxiv.org/html/2609.03221#bib.bib47)\], and we name which cells are powered, and make no claim about the others\.

Only the descriptor’s slot fillers change, and every slot is present in every condition\. This is a correction to an earlier design in which the phrase “with limited English proficiency” was appended only to LEP conditions, which made them strictly longer than every other condition and confounded the flagship LEP axis with surface form\. Language proficiency is now a slot filled in all conditions \(“full English” or “limited English”\), matched in word count, so the LEP contrast varies one filler and nothing else\.

Holding grammatical form constant does not hold token identity, token frequency, or embedding neighbourhood constant, and tokens are not meaning\. A flip therefore has a rival explanation\. Surface perturbation is at least as parsimonious as demographic reasoning, and no amount of replicate agreement addresses it, replicate agreement measures sampling stability, which is a different thing\.

The condition set consequently carries three control arms\. An*identical re\-render*isolates decoding nondeterminism\. A*sham\-attribute*arm carries a real but clinically irrelevant personal detail in the same slot position\. A*rare\-token*arm carries a plausible but non\-referring nationality token matched in word count, isolating sensitivity to token rarity, which differs systematically across real race terms and would otherwise be indistinguishable from demographic sensitivity\. A disparity is credible only as the excess over the largest of the three, and is reported that way\. Because no phrasing can equalize every descriptor, since “Native American” is two tokens where “Black” is one, the harness also reports per\-condition character and word counts, so residual length differences get measured instead of assumed away\.

All three arms are implemented in the released condition set, and the pilot of Sec\.[V\-C](https://arxiv.org/html/2609.03221#S5.SS3)runs them alongside the demographic conditions instead of assuming it\. That pilot is the reason the commitment in the preceding sentence is stated as strongly as it is: the identical re\-render moved the action in a fifth of the cells it was run on, which is more often than most of the demographic contrasts moved\.

##### Confounded\-pair registry\.

Certain attribute×\\timestask pairs carry legitimate clinical signal, and a flip in such a pair should not be conflated with appropriate care\. The registry that encodes this is*directional*, because a symmetric exemption would excuse the very disparities the benchmark exists to detect\. Deciding which demographic conditioning is legitimate is contested territory in its own right\[[30](https://arxiv.org/html/2609.03221#bib.bib45)\], and we treat each registered pair as a claim requiring justification, not a technical convenience\. More interpreter\-dependent documentation and*closer*follow\-up for a patient with LEP are appropriate care;*less*follow\-up and*fewer*referrals is the harm the literature documents\[[4](https://arxiv.org/html/2609.03221#bib.bib15)\], and it remains a confirmatory bias candidate\. We therefore pre\-register, for each registered pair, the direction that is clinically defensible: differences in that direction are demoted to descriptive reporting, and differences in the opposite direction are not\. The registry keys on the*sub\-action*, not the domain, and that granularity is load\-bearing\. Analgesia tiering sits inside medication management, so a domain\-level insurance exemption intended for formulary\-driven drug choice would also exempt an analgesia\-tier difference for a Medicaid patient, which is the disparity this benchmark most needs to catch\. We therefore register no insurance exemption at all: the released action space encodes an analgesia*tier*and not a drug identity, so no formulary justification is expressible in it, and a payer\-conditioned analgesia or acuity difference stays confirmatory\.

Age is likewise left unregistered\. Geriatric under\-triage and end\-of\-life over\-escalation are both documented harms, so registering either direction would define one of them away; we report age contrasts as confirmatory and name the ambiguity instead of resolving it by fiat\.

Stigmatizing language can never be exempted under any attribute: excess negative and social\-risk labeling of patients with LEP runs in the*more*direction, so a rule that treats more documentation as appropriate care would have protected exactly the wrong thing\. What remains registered is LEP against follow\-up interval and referral, where a shorter interval and more referral are defensible and the documented harms run the other way\. It is deliberately*not*registered for acuity, screening, or analgesia: a payer\-conditioned triage decision is not an access constraint but the kind of differential treatment that emergency\-care obligations exist to prohibit, and exempting it would launder discrimination as appropriate care\.

### III\-DMetrics

LetVVbe the vignette set andc0∈𝒞c\_\{0\}\\in\\mathcal\{C\}the reference condition\. Writea⁡\(v,c\)a\(v,c\)for the*ordinal*action the agent commits \(ESI acuity11–55, analgesia tier00–33\),ϕ⁡\(v,c\)∈\{0,1\}\\phi\(v,c\)\\in\\\{0,1\\\}for its pre\-registered threshold dichotomization \(e\.g\., any opioid≡\\equivtier≥2\\geq 2on the scale 0 none, 1 NSAID, 2 weak opioid, 3 strong opioid; high acuity≡\\equivESI≤2\\leq 2\), andg⁡\(v,c\)∈\[0,100\]g\(v,c\)\\in\[0,100\]for a continuous output \(urgency\)\. The distinction betweenaaandϕ\\phiis load\-bearing for the headline metric below and is not merely notational\. Ordinal actions enter MASD as an ordinal distance; set\-valued order sets are summarized by a per\-task guideline\-indicated\-order indicator \(and, where reported, a Jaccard set disparity\)\. Each task’s mapping is fixed a priori in the Supplement\.

The*counterfactual flip rate*\(CFR\) for a contrast\(c0,c\)\(c\_\{0\},c\)is

CFR\(c0,c\)=1\|V\|∑v∈V\[ϕ\(v,c0\)≠ϕ\(v,c\)\]\.\\mathrm\{CFR\}\(c\_\{0\},c\)=\\frac\{1\}\{\|V\|\}\\sum\_\{v\\in V\}\\mathbb\{1\}\\\!\\left\[\\phi\(v,c\_\{0\}\)\\neq\\phi\(v,c\)\\right\]\.\(1\)The*mean absolute score difference*\(MASD\) and*signed action\-level disparity*are

MASD⁡\(c0,c\)=1\|V\|​∑v∈V\|g⁡\(v,c0\)−g⁡\(v,c\)\|,\\mathrm\{MASD\}\(c\_\{0\},c\)=\\frac\{1\}\{\|V\|\}\\sum\_\{v\\in V\}\\bigl\|g\(v,c\_\{0\}\)\-g\(v,c\)\\bigr\|,\(2\)and the signed action\-level disparity isΔ⁡\(c0,c\)=ϕ¯​\(c\)−ϕ¯​\(c0\)\\Delta\(c\_\{0\},c\)=\\bar\{\\phi\}\(c\)\-\\bar\{\\phi\}\(c\_\{0\}\)withϕ¯​\(c\)=\|V\|−1​∑vϕ⁡\(v,c\)\\bar\{\\phi\}\(c\)=\|V\|^\{\-1\}\\sum\_\{v\}\\phi\(v,c\);Δ\\Deltais a signed counterfactual action\-rate disparity in the spirit of demographic parity and counterfactual fairness\[[25](https://arxiv.org/html/2609.03221#bib.bib22)\]and the Aequitas audit taxonomy\[[31](https://arxiv.org/html/2609.03221#bib.bib26)\]; unlike equality of opportunity\[[32](https://arxiv.org/html/2609.03221#bib.bib27)\], it does not condition on a ground\-truth label\.

Our headline metric is the*within\-range*flip rate \(WCFR\), which isolates demographic sensitivity from correctness\. Given a clinician\-defined set of acceptable actionsA⁡\(v\)A\(v\)for each task \(Sec\.[IV](https://arxiv.org/html/2609.03221#S4)\),

WCFR⁡\(c0,c\)=∑v𝟙\[ϕ\(v,c0\)≠ϕ\(v,c\)\]ι\(v\)∑vι⁡\(v\),\\mathrm\{WCFR\}\(c\_\{0\},c\)=\\frac\{\\sum\_\{v\}\\mathbb\{1\}\\bigl\[\\phi\(v,c\_\{0\}\)\\neq\\phi\(v,c\)\\bigr\]\\,\\iota\(v\)\}\{\\sum\_\{v\}\\iota\(v\)\},\(3\)whereι⁡\(v\)=𝟙​\[a⁡\(v,c0\),a⁡\(v,c\)∈A⁡\(v\)\]\\iota\(v\)=\\mathbb\{1\}\\bigl\[a\(v,c\_\{0\}\),a\(v,c\)\\in A\(v\)\\bigr\]marks the vignettes whose two variants both fall inside the band\. The band is defined on the ordinal actionaa, not on its dichotomizationϕ\\phi, and the design of the estimand depends on that\. WereA⁡\(v\)A\(v\)a subset of\{0,1\}\\\{0,1\\\}, the metric would carry no information beyond CFR\. An admissible set of both values leaves every pair in range and returns CFR exactly\. A singleton set forcesϕ⁡\(v,c0\)=ϕ⁡\(v,c\)\\phi\(v,c\_\{0\}\)=\\phi\(v,c\)and returns a mechanical zero\. Placing the band onaamakes WCFR non\-zero exactly for vignettes whose acceptable range*straddles*the pre\-registered threshold\. That is the clinically meaningful case: two guideline\-concordant actions that a dichotomy nonetheless separates\.

That straddling count is reported with every estimate and is the denominator of the headline rate\. A band lying wholly on one side of the threshold can never produce a flip, so counting such vignettes would make the rate a function of which bands the set happens to contain rather than of how the agent behaves\. Being itself a function of band width, it is what bounds precision in practice\. A within\-range flip, where*both*demographic variants are clinically acceptable, is pure demographic sensitivity and not a medical error, and is the cleanest evidence of bias\. Within\-range is not the same as without consequence: a systematic within\-band shift, such as consistently assigning one group the lower of two guideline\-concordant analgesia tiers, is an inequity that accumulates across encounters even though no single decision is erroneous\. WCFR and the propagation estimator of Sec\.[V\-D](https://arxiv.org/html/2609.03221#S5.SS4)are implemented and unit\-tested in the released harness; neither is reported here\. One distinction matters enough to state directly, because the natural assumption is the wrong one: the estimator does not return NA when a band lacks an adjudicator\. It returns a numeric rate together withground\_truth: false, a count of unadjudicated bands, and a provenance note\. NA is reserved for the separate case of an empty straddling denominator\. A reporting path must therefore consult the provenance flag and not rely on a missing value to make an unwarranted number impossible to print, and the harness refuses the call outright when the bands argument is omitted\.

Because both MASD and a threshold dichotomization discard direction \(MASD is an absolute value, so systematic under\-triage of one group and symmetric noise return the same number, while a shift from ESI33to ESI44never crosses a high\-acuity cut at all\), we report alongside them a signed paired\-ordinal estimand: the mean signed difference inaa, a Wilcoxon signed\-rank statistic over non\-zero differences, and the full paired transition tally\. Direction is what identifies who is disadvantaged, so we report it\. We report the denominator \(the number of both\-in\-range vignettes\) alongside every WCFR, as it is the effective sample size and shrinks under band tightening; WCFR is undefined \(reported as NA, not00\) when no vignette has both variants inA⁡\(v\)A\(v\), and its variance grows as the denominator shrinks\.

### III\-EContamination and Anti\-Gaming

Every vignette carries a canary string and profiles are generated from parameterized templates, so verbatim memorization is detectable\. A development split is published and a sealed test split withheld until clinician validation completes\.

Ranking on within\-range disparity alone would be perverse, in two ways that the released harness gates against\. Any demographically invariant constant policy minimizes WCFR: an agent that always assigns the same acuity has a large in\-band denominator, no flips, a perfect fairness score, and no clinical use\. The subtler route exploits the denominator itself\. Because that denominator is the set of jointly in\-band pairs, a model that pushes borderline cases*out*of band shrinks it and can report a lower WCFR than a more capable competitor\. A submission therefore qualifies only above three pre\-registered floors: a capability score on the same trajectories, an action\-entropy floor, and a minimum count of band\-straddling vignettes\. The first two catch the constant policy; only the third closes the denominator\-shrinking route\. Entries failing any floor are published as withheld with the reason attached, never silently dropped\.

## IVExperimental Setup

We report the benchmark following TRIPOD\-LLM\[[14](https://arxiv.org/html/2609.03221#bib.bib28)\]and its parent TRIPOD\+AI\[[33](https://arxiv.org/html/2609.03221#bib.bib29)\]\(completed checklist in the Supplement\) and self\-audit its evaluation validity against the applicable signalling questions of PROBAST\+AI\[[34](https://arxiv.org/html/2609.03221#bib.bib30)\]\.

### IV\-AScenarios and Vignettes

N=300N\{=\}300synthetic vignettes span four action domains: triage/escalation, diagnostic and laboratory ordering, medication management \(with analgesia tiering as a construct\-validity anchor for documented pain\-treatment disparities\[[1](https://arxiv.org/html/2609.03221#bib.bib12),[2](https://arxiv.org/html/2609.03221#bib.bib13),[35](https://arxiv.org/html/2609.03221#bib.bib37),[36](https://arxiv.org/html/2609.03221#bib.bib38)\], following the measurement\-modeling account of construct validity for fairness instruments\[[37](https://arxiv.org/html/2609.03221#bib.bib46)\]\)\. An anchor is only an anchor if it can fail, so we state the failure condition in advance: if the Black\-versus\-White analgesia contrast, run at the pre\-registered vignette count and powered as Sec\.[IV\-D](https://arxiv.org/html/2609.03221#S4.SS4)describes, does not show a negative signed ordinal shift, we treat the instrument as not having demonstrated construct validity for pain\-treatment disparity, and we report that rather than reinterpreting the null as an equity finding\. One asymmetry has to be admitted here\. Severe under\-treatment, where the descriptor drives analgesia below any clinically acceptable tier, moves the action out of band and is therefore excluded from the within\-range estimand by construction; the anchor is evaluated on the signed and unconditioned metrics, which do see it, and documentation/disposition\.

All vignettes are synthetic and contain no protected health information; because synthetic data does not by itself remove human\-research\-protection obligations, a*written*determination for the clinician\-labeling activity will be obtained from the institutional human\-research\-protection program before any annotation begins, and is not assumed\. No annotation has taken place\. The generation procedure is documented in an accompanying Datasheet\[[38](https://arxiv.org/html/2609.03221#bib.bib32)\]\.NNfollows from the discordant\-pair floor of Sec\.[IV\-D](https://arxiv.org/html/2609.03221#S4.SS4)and not from convenience: each vignette contributes one pair per contrast, so a contrast reaching the pre\-registered2525discordant pairs requires25/πd25/\\pi\_\{d\}vignettes, whereπd\\pi\_\{d\}is the discordance rate\. The pilot’s single\-draw rate of0\.210\.21gives119119; the replicated estimate of Sec\.[V\-A](https://arxiv.org/html/2609.03221#S5.SS1)is0\.0870\.087, which gives289289\. We pre\-registerN=300N\{=\}300, leaving margin and note the consequence plainly: a contrast whose true discordance rate falls below that value will not reach the confirmatory floor, and will be reported as hypothesis\-generating, and never padded to significance\. The figure has now moved twice, from119119on a single draw to184184on four vignettes to289289on sixteen, in each case because the floor estimate improved, not because the design changed\. Sizing a fairness study before measuring the instrument’s own noise is the error this sequence illustrates\.

Band adjudication is a protocol commitment that has not yet been carried out: the vignettes reported in Sec\.[V\-C](https://arxiv.org/html/2609.03221#S5.SS3)are drafts, no acceptable\-action band has yet been labeled, and consequently the within\-range estimand is undefined on them\. Where this manuscript describes adjudicated bands it describes the benchmark as specified, and every quantity gated on that adjudication is marked accordingly\.

### IV\-BClinician Labeling Protocol

Ground truth is the set of*clinically acceptable*actionsA⁡\(v\)A\(v\)per task, in place of a single “correct” answer, so that within\-range disparity is well defined\. Choosing which action to label is itself a consequential decision: an algorithm can be unbiased with respect to the label it was given and still cause harm when that label is the wrong proxy for the outcome of interest\[[39](https://arxiv.org/html/2609.03221#bib.bib44)\]\. SoA⁡\(v\)A\(v\)is treated as a substantive equity commitment, not a neutral annotation step\. Bands are*derived*from published decision rules and then*adjudicated*by a clinician, rather than generated from clinical judgment alone\. Each band records the rule it came from, the clause relied on, and the reasoning that yields this set for this presentation: the Emergency Severity Index Handbook, fifth edition\[[40](https://arxiv.org/html/2609.03221#bib.bib49)\], for acuity; emergency\-medicine acute\-pain guidance\[[41](https://arxiv.org/html/2609.03221#bib.bib50)\]for analgesia tier; and diabetes transitions\-of\-care guidance\[[42](https://arxiv.org/html/2609.03221#bib.bib51),[43](https://arxiv.org/html/2609.03221#bib.bib52)\]for follow\-up interval\. A reviewer can therefore check the mapping against a document instead of taking a panel’s word for it, and the width ofA⁡\(v\)A\(v\)is fixed by reference to something outside this study\. Band width otherwise sits unconstrained under the headline estimand: an author who widens it manufactures within\-range flips, and one who narrows it deletes them\.

A clinician who is not an author then reviews the derivation, blinded to the demographic conditions, and either signs off on it or corrects it\. The protocol specifies role composition where the task determines it: emergency\-severity acuity is assigned by triage nurses in US emergency departments, so a triage nurse should adjudicate the acuity bands, and because the LEP registry turns on what interpreter\-dependent documentation should look like, a medical interpreter should review the documentation band\. That composition has not yet been achieved, and what has been done falls short of it in two ways we state here instead of leaving them to be discovered\. One licensed clinician has reviewed all four draft bands against their cited sources\. The review is recorded in the released repository under a pseudonym at her request, with her name, credentials and registration held by the corresponding author and available to the editor\.

She is a geriatrician\. Two of the four bands sit squarely inside her practice: the discharge follow\-up interval after a resolving metabolic emergency, and the referral decision\. Two sit outside it, since emergency\-department triage acuity and acute\-pain tiering in renal colic are emergency\-medicine judgements\. Her verdicts on those two are recorded and are not treated as adjudication; the released registry emits a scope warning for exactly this pairing, so the mismatch is visible to any reader of the artifact, not resting on this sentence\. No triage nurse and no medical interpreter has yet reviewed any band, and her signed attestation has not been returned, so every band in the released set remains author\-derived and no result in this paper rests on any of them\.

Every band carries its provenance level, author\-derived, guideline\-derived, or clinician\-adjudicated, and the harness refuses to treat an unadjudicated band as ground truth\. The count of bands at each level is given, so a reader can see how much of the estimand rests on adjudicated judgment\.

Inter\-rater agreement is reported as Krippendorff’sα\\alpha\[[44](https://arxiv.org/html/2609.03221#bib.bib36)\], chosen overκ\\kappabecause it accommodates ordinal actions, more than two raters, and missing labels within a single coefficient, and it is computed only over the subset of bands carrying two or more independent raters, with the size of that subset printed beside it\. Where a band has a single adjudicator, we say so and report no agreement coefficient for it, since an agreement statistic over one rater is not an agreement statistic\. At present that subset is empty: every band carries at most one reviewer, so noα\\alphais reported and none can be\. Recruiting a second independent rater is therefore not a refinement but the step that brings the agreement machinery into force at all\.

Disagreements are resolved by intersection: an action entersA⁡\(v\)A\(v\)only if every labeler marks it acceptable\. This rule and the union sensitivity analysis bind only on the subset of bands carrying two or more independent raters, whose size we report; with a single adjudicator there are no disagreements, intersection and union coincide, and no safeguard is in force\. Union was rejected deliberately\. A wider band admits more pairs and dilutes the flip rate, so the choice sets both the magnitude and the precision of the headline number, and the more conservative rule is the one that cannot manufacture a within\-range flip out of a single labeler’s permissiveness\. We report the union\-band estimate alongside as a sensitivity analysis, and report each labeler’s band separately\.

Because every disparity estimate depends on the width ofA⁡\(v\)A\(v\), each estimate is accompanied by a sensitivity analysis that recomputes it under tightened and widened bands, so that a reader can see how much of the reported number survives a different but still defensible reading of the same rule\. The bands must be adjudicated before that analysis has anything to vary, so it is specified here and reported with the primary results\.

### IV\-CModel Panel and Scaffolds

We evaluate frontier and open\-weight models under three*reasoning\-style*scaffolds: direct \(C0\), chain\-of\-thought \(C1\), and a single\-call deliberation persona in which the model argues both sides before committing \(C2\)\. These vary how much explicit reasoning precedes the action, not autonomy: the phase order is fixed in every scaffold and C2 is one model call, so calling it increased autonomy would overstate the implementation\. The question is nonetheless agent\-specific: whether deliberation attenuates a demographic prior or supplies a rationalization for it\.

The powered evaluation uses temperature00with pinned model identifiers and per\-run decoding records, which fixes the sampling step without making inference reproducible \(Sec\.[V\-B](https://arxiv.org/html/2609.03221#S5.SS2)\), so the control arms run there too\. Neither the pilot of Sec\.[V\-C](https://arxiv.org/html/2609.03221#S5.SS3)nor the floor study of Sec\.[V\-A](https://arxiv.org/html/2609.03221#S5.SS1)had that recording available, and both ran at provider\-default sampling; this is stated wherever their numbers appear\. Each cell runsRRreplicates with per\-cell agreement reported, so determinism is measured and never assumed\. External models are admitted through the leaderboard rather than run here\.

### IV\-DStatistical Analysis

Within each cell theRRreplicates collapse to a single summary before inference, soNNcounts independent vignettes, not API calls\. Binary and ordinal actions collapse by majority vote and continuous outputs by the per\-cell mean; a tied vote is reported as undetermined; resolving it arbitrarily would invent a result\.

Majority vote is a nonlinear map, so the collapsed rate is not the underlying probability: near one half a gap is amplified, and at the low rates characterizing the actions of interest here it is attenuated, in simulation by roughly a factor of four atR=5R\{=\}5\. Because attenuation hides disparity, replicate\-level models with the vignette as a random effect are the primary analysis for*every*scaffold and collapsed results are a sensitivity analysis\. We pre\-registerR=5R\{=\}5, odd so the vote is always defined, and report per\-cell agreement\.

Each disparity metric is the coefficient of an intercept\-only cluster\-robust regression clustered on vignettes, with an interval from the restricted wild cluster bootstrap using Webb six\-point weights and at⁡\(G−1\)t\(G\-1\)reference, imposing the null by test inversion\[[45](https://arxiv.org/html/2609.03221#bib.bib39),[46](https://arxiv.org/html/2609.03221#bib.bib40),[47](https://arxiv.org/html/2609.03221#bib.bib41)\]\. That procedure, its estimator, the accumulation composite, the single\-prime ablation, the forced\-upstream contrast, the permutation test and the control arms are implemented and unit\-tested; the replicate\-level mixed model remains specified and unimplemented\. The weight distribution decides whether the test can reject at all at these cluster counts: a two\-point Rademacher weight admits only2G2^\{G\}vectors, and enumeration puts the smallest attainable two\-sidedppat0\.180\.18forG=4G\{=\}4and0\.090\.09forG=5G\{=\}5, so a nominal5%5\\%test is inert there whatever the data say\. Six\-point weights\[[46](https://arxiv.org/html/2609.03221#bib.bib40)\]reach0\.0210\.021and0\.0080\.008, which the harness asserts and does not assume\. On clustered simulated data atG=8G\{=\}8the interval covers in94\.0%94\.0\\%of200200replications \(Monte Carlo standard error1\.71\.7points\), the mild under\-coverage the small\-GGliterature reports; the generating process and script are released\.

Both the bootstrap and the permutation test are restricted to*signed*estimands, for one reason with two consequences\. CFR, MASD and WCFR are each symmetric in the pair, so under the sharp null \(where the two labels are exchangeable within a vignette, giving an exact reference distribution with no distributional assumption\), swapping leaves them unchanged and their permutation distribution is a point mass no dataset could reject\. The same symmetry breaks the bootstrap differently: being non\-negative by construction, their null sits on the boundary of the parameter space, where a negative weight maps residuals outside the support\. Signedness is not recoverable from values, since a signed statistic may be one\-sided by chance, so the harness requires the caller to declare it, refuses a symmetric statistic by name, and does not return a vacuousp=1p=1\. Inference therefore runs on the signed action\-rate disparityΔ\\Deltaand the signed ordinal shift\. The harness also retains a nonparametric percentile cluster bootstrap, which produced the interval on the floor in Sec\.[V\-A](https://arxiv.org/html/2609.03221#S5.SS1); the two are not comparable and each interval is labelled with its method\. We reportGGbeside every interval and label those withG≲12G\\lesssim 12approximate\.

Paired binary flips are tested with the exact McNemar test on vignette\-level discordant pairs\[[48](https://arxiv.org/html/2609.03221#bib.bib43)\], whose countsbbandcc*are*directional\. It is valid at any discordant\-pair count, has negligible power when that count is small, and is undefined, and reported as such, when there are none\. A contrast is*confirmatory*only with at least2525independent discordant pairs\. That is a convention, not a power calculation, and its consequence is computable: at2525pairs the exact two\-sided test atα=0\.05\\alpha=0\.05has77%77\\%power at a19:619\{:\}6split, a discordance odds ratio of3\.23\.2, reaching80%80\\%between19:619\{:\}6and20:520\{:\}5\. Smaller asymmetries are not confirmable at this cell size\. Contrasts below the floor, and all confounded pairs, are reported as hypothesis\-generating without significance claims\.

Multiplicity is controlled at a false\-discovery rate of0\.050\.05over the family of confirmatory contrasts: those meeting the floor and lying outside the confounded registry\. Because the discordant\-pair count is ancillary for the odds ratio in the conditional McNemar model, nullpp\-values within the selected family remain uniform and the guarantee holds conditional on that family; the number of contrasts screened out is given\. Contrasts sharing a reference condition are positively correlated, so the pre\-registered procedure is Benjamini–Yekutieli\[[49](https://arxiv.org/html/2609.03221#bib.bib48)\], valid under arbitrary dependence, with Benjamini–Hochberg\[[50](https://arxiv.org/html/2609.03221#bib.bib42)\]alongside as a sensitivity analysis and never as primary\. Both are implemented\.

### IV\-EReproducibility

The harness is released as a semantically versioned, pip\-installable package with fixed analysis seeds and pinned model identifiers\. The model\-call layer is supplied by the caller, so decoding settings are recorded per run and not seeded\. The development split is published as a versioned dataset with a Datasheet\[[38](https://arxiv.org/html/2609.03221#bib.bib32)\]and per\-model Model Cards\[[51](https://arxiv.org/html/2609.03221#bib.bib33)\]on completion of clinician validation, and a citable archival snapshot is minted via Zenodo at that release\. Every table and figure in this paper is regenerated from stored artifacts by a single command\.

## VResults

### V\-AThe instability floor

A counterfactual flip rate is a difference between two conditions, so it is interpretable only against how often the agent differs from itself\. We estimated that quantity instead of assuming it\.

The reference condition was run ten independent times over sixteen vignettes\. Nothing varied: the same clinical narrative, the same descriptor string, the same harness\-constructed prompts, each replicate answered in a separate context with no access to the others\. Ten replicates give forty\-five pairwise comparisons over800800model calls, and every difference within one is decoding nondeterminism by construction\. Decoding used the provider’s default sampling configuration, as Sec\.[IV\-C](https://arxiv.org/html/2609.03221#S4.SS3)records\.

The pooled floor is0\.0870\.087\(374/4320374/4320outcome–vignette comparisons\), with a percentile cluster bootstrap interval of\[0\.056,0\.116\]\[0\.056,0\.116\]clustered on the vignette atG=16G\{=\}16\.

TABLE II:Null flip rate per action over ten independent replicates of an identical condition \(4545pairwise comparisons over1616vignettes, so720720comparisons per action\)\. Nothing varies between replicates; every flip is decoding instability\.The floor is not one number\. Table[II](https://arxiv.org/html/2609.03221#S5.T2)reports it per action, and the spread is eightfold: ICU escalation moves in one comparison in fifty, controlled\-substance caution in nearly one in five\. A single pooled floor describes neither well, and an audit reporting one flip rate for a whole trajectory averages quantities whose noise differs by that factor\.

That spread invites an objection we should answer, not leave standing\. Disagreement between two draws is maximised when the underlying rate is one half and vanishes as it approaches either extreme, and these actions differ sharply in prevalence: intensive\-care escalation is indicated in14%14\\%of draws while admission is indicated in64%64\\%\. An eightfold spread in raw flip rates could therefore be prevalence dressed up as instability\.

It is not\. Comparing each action against a homogeneous\-rate null, what disagreement would be if every vignette shared that action’s marginal rate at2​p¯​\(1−p¯\)2\\bar\{p\}\(1\-\\bar\{p\}\), the ratios run from0\.050\.05for any\-opioid to0\.420\.42for controlled\-substance caution, a factor of eight that survives having prevalence divided out\. Both models agree on which action tops that adjusted ranking\. Every ratio also sits well below one, which says something the raw rates conceal: the instability is not spread evenly across vignettes but concentrated in a minority of contested cells, with most cells decided the same way every time\. The seventy\-five unanimous cells give the same picture\.

One caveat belongs with the top\-ranked action\. Acuity is defined by the ESI handbook and analgesia tier rests on published acute\-pain guidance, but the controlled\-substance caution flag is put to the model without operational criteria: no prescription\-monitoring rule, no diversion\-risk rubric, nothing a clinician could apply consistently either\. Its position therefore admits two readings we cannot separate here\. The judgement may be genuinely contested, or an underspecified construct may invite different plausible interpretations on different draws\. The second is the less flattering reading of our own instrument, and we name it, since a benchmark that leaves an action undefined should expect this\. Giving that flag a definition is a change the powered run needs, and it is where our action space is weakest\.

The study was then repeated on a second model, six replicates over the same sixteen vignettes with identical prompts, so the floor is not one system’s artefact\. Its pooled floor is0\.0670\.067against0\.0870\.087: two points apart, and neither near zero\. More informative is what happens to the ordering\. Ranking the six actions by instability gives a Spearman correlation of0\.940\.94between models, and both place the same action at each end: escalation to intensive care is the most stable for both \(0\.0220\.022and0\.0210\.021\), and the controlled\-substance caution flag the least stable for both \(0\.1790\.179and0\.2000\.200\)\. Two actions swap adjacent ranks; the rest hold\. The exact permutation test over all720720orderings givesp=0\.017p=0\.017two\-sided, and we report the exact value rather than the asymptotic approximation, which returns0\.0050\.005and overstates the evidence at thisnn\. The attainablepp\-values are coarse here: one further rank swap would give0\.0580\.058\. The correlation is therefore consistent with a shared ordering, though it does not establish one, and we treat it as such\.

#### The floor\-reporting procedure

The recommendation this paper makes is small enough to state completely, and we state it separately so it can be adopted without reading the rest\. An audit reporting counterfactual flip rates should:

1. 1\.run the reference conditionRRtimes over the same cases, varying nothing: same narrative, same descriptor string, same prompts, independent contexts;
2. 2\.report the resulting flip rate*per action*, not pooled, since the actions in our data differ by a factor of eight and a pooled figure describes none of them;
3. 3\.report each demographic contrast beside the floor for that same action, and treat a contrast that does not exceed it as indistinguishable from noise; and
4. 4\.state the decoding configuration, because pinning the temperature reduces the floor without eliminating it \(Sec\.[V\-B](https://arxiv.org/html/2609.03221#S5.SS2)\)\.

The cost is one additional condition, reusing the cases already in the run\. AtR=10R\{=\}10over sixteen cases this was800800calls, roughly what a single demographic arm costs\. An audit that cannot afford it can still report step 4 and the value ofRRit managed, which is more than current practice supplies\.

That separation is the useful part, within limits we should name: two models and six ranked actions cannot establish a general law, and what follows is a working hypothesis, and a third model would test it\. The overall level looks like a property of the model, so an audit should not borrow a floor from another system’s report\. Which decisions are unstable looks largely like a property of the decision, so an audit can reasonably expect the same judgements to be the difficult ones\. It also converges with the dispersion result below: a decision that is hard for two unrelated models is more plausibly hard because of where it sits relative to a clinical boundary than because of anything either model does\. One asymmetry is worth stating: the second arm has six replicates against ten, so its per\-action rates carry wider uncertainty and we draw no conclusion from a few points of difference between them\.

These figures are given having already seen a smaller version of this study mislead us, which is the strongest argument we can offer for the sample size\. At four vignettes the pooled floor read0\.1360\.136with an interval of\[0\.063,0\.209\]\[0\.063,0\.209\], two of the six actions appeared never to move at all, and the ordering of the rest bore little resemblance to Table[II](https://arxiv.org/html/2609.03221#S5.T2)\. All three were artifacts\. Sixteen vignettes give an interval less than half as wide, no action with a zero rate, and a different ranking\. A floor estimated from a handful of cases is not a conservative version of the real one; it is simply wrong, in whichever direction the sample happens to fall\.

### V\-BHow much of the floor is removable

An obvious objection is that the fix is one decoding parameter: pin the temperature to zero and the floor disappears, so a warning aimed at default sampling is a warning about bad practice rather than about the instrument\.

The premise does not hold\. Temperature governs the sampling step, not the logits that step draws from, and those logits are produced by kernels whose reduction order depends on batch composition, GPU count and numerical precision\. Yuan*et al\.*measure this directly and report accuracy variation of up to nine points, and response\-length differences of thousands of tokens, attributable to hardware and batch configuration with sampling held fixed\[[52](https://arxiv.org/html/2609.03221#bib.bib53)\]\. Greedy decoding therefore removes one source of the floor and leaves another in place, and it is a source an auditor of a hosted model does not control and usually cannot observe: batch composition is a function of who else is querying the endpoint at the time\. We report our own figures at the provider’s default sampling and make no claim about the magnitude that would remain at temperature zero, which we could not set; the point is that the remaining magnitude is an empirical question, not a settled one\.

We can, however, measure the other route to removing the same variance, which the protocol already pre\-registers: replicates aggregated by majority vote and compared across*disjoint*groups, since overlapping groups share draws and would report agreement driven by the overlap\.

The floor falls from0\.0870\.087at a single draw to0\.0630\.063atR=3R\{=\}3and0\.0530\.053atR=5R\{=\}5\. Aggregation removes39%39\\%of it and the curve flattens\.

It is tempting to read that residue as evidence of something beyond sampling noise, and an earlier draft of this paper did\. It is not\. Majority vote converges at a rate governed by how far a cell’s flip probability sits from one half: a cell answered the same way nine times in ten collapses almost immediately, while a cell split evenly barely moves however many draws are averaged\. A population mixing the two produces this fast\-then\-slow shape with no cell holding two stable answers\. Simulating independent Bernoulli noise at the observed per\-cell rates and applying the identical disjoint\-split procedure places the observed curve inside the simulated null at every depth: atR=5R\{=\}5, observed0\.0530\.053against a null mean of0\.0480\.048and a central interval of\[0\.025,0\.076\]\[0\.025,0\.076\]\.

The objection deserves a direct answer, and the data give a partial one\. Seventy\-five of the ninety\-six vignette–action cells were unanimous across all ten replicates, so the floor is carried by twenty\-one, and eleven have observed rates between0\.30\.3and0\.70\.7\. Those eleven are the slow ones: majority vote moves a cell toward certainty at a rate set by its distance from one half, so cells near the boundary barely move at any feasible depth\.

What we cannot do is estimate the floor a deterministic decode would face\. It is tempting to read the three cells that split exactly five–five as the residue that would survive, since only a cell at exactly one half fails to converge\. That reasoning does not hold at ten draws\. A cell atp=0\.5p\{=\}0\.5ties only25%25\\%of the time and a cell atp=0\.4p\{=\}0\.4ties20%20\\%of the time, so if all eleven near\-boundary cells sat near one half we would expect2\.72\.7ties and we observe three\. The tie count is therefore consistent with cells merely close to the boundary, not on it,p=0\.5p\{=\}0\.5andp=0\.4p\{=\}0\.4are not separable at this depth, and we report no estimate of the deterministic floor\.

The convergence rate is estimable, and it is the quantity an audit actually needs\. AtR=5R\{=\}5the floor is0\.0530\.053, still61%61\\%of its single\-draw value, and the last two points of the curve fall slowly enough that a linear continuation would not reach zero until aboutR=15R\{=\}15\. The curve is convex, so that figure is optimistic\. This bears directly on our own protocol, which pre\-registersR=5R\{=\}5: at that depth most of the floor is still present, and a study replicating five times and treating the result as converged would be wrong about its own instrument\. The recommendation that follows does not depend on which decoding an audit uses\. Whether it pins the temperature or replicates, it has to demonstrate that the floor is gone by measuring it, rather than assume that the parameter or the depth settled the question\. The control arm costs one condition and reuses the vignettes already in the run\.

We looked for a mechanism and did not find one that survives\. On four vignettes, the cases whose continuous urgency score varied most across replicates were exactly the cases whose discrete acuity flipped, which suggested that instability tracks proximity to a decision boundary\. At sixteen vignettes the relationship reverses: the wide\-dispersion half contributes no acuity flips at all and the tight\-dispersion half contributes all5555\. We report the reversal rather than the version that agreed with our expectation\. Whatever governs which cells are unstable, a single continuous proxy measured on ten draws does not capture it, and we make no case\-difficulty claim\.

### V\-CDemographic contrasts against the measured floor

The pilot that exercised the harness end to end supplies the only demographic data reported here: the same four vignettes under six conditions, plus the three control arms of Sec\.[III\-C](https://arxiv.org/html/2609.03221#S3.SS3), at one draw per cell\. Every contrast falls inside the range the re\-render itself spans\. The inference is narrower than it looks\. A demographic contrast measures the floor*plus*whatever signal exists, so it should if anything sit above the null rather than below it, and observing one below a single null draw is evidence that it is close to the null rather than evidence that the null exceeds it\. The defensible statement is that no contrast is distinguishable from decoding instability at this sample size, and no disparity is claimed from any of them\.

Two cells nonetheless looked like signal, and the episode is worth reporting because the arithmetic that dismissed them did not survive better measurement\. Controlled\-substance caution flipped in three of four vignettes for both the Black\-woman\-on\-Medicaid and the LEP contrast\. Against the pilot’s own single\-draw floor of0\.210\.21such a cell arises with probability0\.0310\.031, giving0\.920\.92expected across the thirty inspected and making two unremarkable\. Against the replicated floor the probability is0\.0020\.002, or0\.0200\.020using the action\-specific floor for that outcome, so two observed cells are well beyond what chance predicts\. What disqualifies them is not the arithmetic but a defect in the arm: the reference arm’s prompt at the prescribing step differs from the comparison arms’, because the first dispatch was declined by the answering model and was reissued with added framing about the synthetic provenance of the vignettes\. Both analgesia outcomes therefore compare a reworded reference against unmodified comparisons and cannot be attributed to the descriptor whether or not they exceed chance\. The harness now emits one canonical dispatch instruction per step so a reissued prompt reuses it\. A floor estimated from one draw was wrong in the direction that made an anomaly look benign, which is the paper’s own thesis turned on itself\.

No within\-range result is reported\. WCFR is undefined until the bands carry adjudicators, and the harness enforces that and does not leave it to discretion:within\_range\_flip\_ratereturnsground\_truth: falsewith a count of unadjudicated bands whenever any band lacks one, refuses the call outright when the bands argument is omitted, and the band constructor refuses to build a clinician\-adjudicated band with an empty adjudicator list\. Adjudication is under way\. What will be reported, fixed here before the numbers exist, is the within\-range flip rate with its straddling denominator, the unconditioned flip rate, the mean absolute score difference and the signed action\-level disparity for each attribute×\\timesdomain cell, each with its cluster count and interval, each accompanied by the per\-action floor of Sec\.[V\-A](https://arxiv.org/html/2609.03221#S5.SS1), and each routed through the confounded registry with the number screened out reported\. A cell failing the adjudication, capability or provenance gates is reported as not evaluated, and never as null\.

### V\-DDisparity propagation

A difference at an early step can propagate to later ones, and that is the estimand a single\-turn design cannot express\. Conditioning on whether the upstream action*happened*to flip would not identify it, because that is a post\-treatment variable: the flipped stratum selects the vignettes on which the model is demographically sensitive at all, and an elevated downstream rate there is equally well explained by stable vignette\-level threshold proximity\. Propagation is therefore identified by*intervening*on the upstream action instead\. The fixed phase order makes this possible: each downstream step is run twice on the same vignette, with the upstream action forced to its reference value in one arm and its flipped value in the other, while the descriptor continues to vary within each arm\. Vignette\-level borderline\-ness is held fixed by construction, and the forced step is recorded as forced, so no manipulated trajectory can pass as free\-running\.

One confound has to be removed before any of this reads as propagation\. The descriptor is rendered at every model call, so a late\-step difference could be carry\-forward or simply the descriptor priming that call afresh\. The single\-prime arm, in which the descriptor appears at the first step only and the trajectory state carries it onward, separates the two\. A scope limit belongs here as well: state passes between steps as a small set of scalars, not the agent’s own prior text, so a deployed agent that keeps its generations in context has a richer channel and this design measures a lower bound on carry\-forward, not an estimate of it\.

No propagation result is reported, and the pilot is not evidence either way: with four vignettes an association between an early\-step and a late\-step difference is indistinguishable from chance\. What will be reported is the forced\-upstream contrast against the single\-call baseline, with the observational stratification alongside it as a descriptive quantity only\. The trajectory\-level accumulation composite accompanies both and identifies nothing on its own\.

## VIDiscussion

FairMedAgent reframes clinical AI fairness evaluation around the*action*taken by a multi\-step agent, not a single generated response, and around*within\-range*disparity that isolates demographic sensitivity from clinical error\. The clinical narrative is frozen and the tool\-result fixture does not depend on the demographic condition, so trajectory divergence is attributable to the descriptor, under the order\-set condition stated in Sec\.[III\-B](https://arxiv.org/html/2609.03221#S3.SS2)\. The agentic loop then makes visible a phenomenon single\-turn audits cannot reach: a disparity introduced at one step may propagate through and compound with later actions\. The powered evaluation quantifies it\.

A null is a result\.The benchmark is designed so that a well\-powered null, meaning within\-range disparity indistinguishable from zero across the model panel, is an informative outcome and is reported as such, with the detectable effect size stated\. Measurement infrastructure whose value depended on finding disparity would be the wrong instrument for an assurance role\.

Scope of claims\.Findings are*in\-silico*signals of model behavior on synthetic vignettes, and they estimate nothing about patient outcomes\. The audit is also a component\-level one\. There is no retrieval, no guideline grounding at inference time, no guardrail layer and no order\-entry validation, and a deployed triage assistant sits behind at least some of these; a flip rate here characterizes a bare model’s action policy, not a fielded system’s behaviour\. We deliberately do not translate flip rates into claims of clinical harm; establishing that a within\-range flip changes real\-world outcomes requires prospective clinical evaluation, for which reporting standards such as DECIDE\-AI and CONSORT\-AI would apply downstream\. The urgency score is a model self\-report and is anchored against the ordinal ESI acuity rather than treated as an independent outcome\.

Relation to prior work\.On triage and diagnostic ordering our design is positioned to test, in the agentic setting, disparities previously reported single\-turn\[[7](https://arxiv.org/html/2609.03221#bib.bib5)\]; whether they replicate here is an open question the powered evaluation answers, and we report no result for it here\. Our contribution is the within\-range construct\. The LEP and intersectional axes, the analgesia and documentation action domains, and the cross\-step propagation analysis extend it instead of standing beside it as separate claims\.

### VI\-ALimitations

The most serious threat to this instrument is one the design bounds but does not remove: a flip may reflect sensitivity to surface perturbation instead of demographic reasoning, and the control arms measure that floor without eliminating it\. Any disparity we report is an excess over that floor, and a disparity smaller than it is not interpretable\. Relatedly, the headline estimand is by construction blind to the most severe case, where a descriptor moves an action out of the acceptable band entirely: that is bias causing clinical error, it is excluded from both numerator and denominator, and it is captured only by the signed and unconditioned metrics\.

The benchmark also uses synthetic vignettes and coarse demographic descriptors; the number of vignettes per domain, and thus the cluster countGG, is modest, so per\-domain intervals are wide and some contrasts are estimation\-only and not confirmatory \(Sec\.[IV\-D](https://arxiv.org/html/2609.03221#S4.SS4)\)\. Ceiling and floor effects may mask disparity where a single action is near\-universal\. The axes evaluated here exclude nonbinary gender, disability, and religion, which we name as out\-of\-scope limitations and do not claim as coverage\.

Ground truth rests on guideline derivation with clinician adjudication rather than on a multi\-clinician generating panel, and the adjudicating panel is small\. A reviewer may reasonably prefer independent generation by several clinicians, and that is the obvious next step; what this design offers instead is that the derivation is documented, sourced, and contestable by anyone holding the same guidelines\. A deeper limit concerns where the band sits, not how wide it is\. Bands are derived from published decision rules, and those rules were written from an evidence base drawn from the same care system whose disparities motivate this work\. If a guideline’s acceptable range was itself calibrated on a population already under\-treated, an agent that reproduces that under\-treatment uniformly across demographic conditions lies inside the band, and the within\-range rate is zero\. WCFR measures demographic sensitivity relative to a published standard; it cannot audit the standard\. The signed metrics detect a shift*within*the band and so retain some purchase, but a floor effect located entirely below defensible practice would register on none of our estimands\. Auditing guidelines themselves for demographic bias is a different study, and we claim no coverage of it\. A related limit sits in the analgesia scale itself\. Reviewing the medication band, the adjudicator observed that a three\-point ordinal cannot express combination therapy, sequential rescue dosing, acetaminophen, intravenous lidocaine or ketamine, each of which is an ordinary choice for the presentation that vignette describes\. The scale records escalation along a single axis and is silent about the multimodal practice that axis sits inside, so the construct\-validity anchor can demonstrate less than a richer action space would allow\. We keep the coarse scale because it is what makes the band checkable against a published rule\. The acceptable\-range width still bounds every estimate; we report band\-width sensitivity accordingly, and treat within\-range disparity as headline only when inter\-rater agreement and the per\-cell within\-range count are adequate, and as exploratory otherwise\.

The manipulation is an explicit one\-line descriptor, so what the benchmark measures is sensitivity to a stated demographic label\. Real records carry demographic signal implicitly, through names, ZIP codes, prior utilisation, insurer, documentation style and accumulated history, and a model may respond to those cues while showing no sensitivity to an explicit label\. A null result here therefore bounds label sensitivity and says nothing about implicit encoding, which is the channel the observational literature on real notes implicates\. The documentation step’s stigmatizing\-language flags are a model self\-assessment, not an external instrument, and are reported descriptively\. The phase order is fixed; adaptive tool selection is out of scope\.

## VIIConclusion

FairMedAgent measures demographic disparity in the actions of multi\-step clinical LLM agents via counterfactual synthetic profiles, with a within\-range metric to be read against its control arms, LEP and intersectional axes, small\-cluster\-appropriate statistics, and a sealed\-split evaluation protocol with a reference leaderboard\. We release the harness now, with the dataset and evaluation protocol to follow on completion of clinician validation, to enable reproducible, re\-runnable fairness audits of clinical agents as the field moves toward autonomous, trustworthy, human\-centric systems\.

## Ethical Considerations

The vignettes are entirely synthetic and contain no protected health information; the study involves no patient data\. Clinician labelers are nonetheless human research contributors: synthetic, zero\-PHI data does not by itself remove human\-research\-protection obligations, so a*written*determination \(including a “not human subjects research” determination\) will be obtained from the institutional human\-research\-protection program*before*any annotation, and is not assumed\.*Dual use:*a public disparity\-elicitation resource could be misused to probe or worsen model behavior\. We release the harness openly and will release the development split on the same terms, gating exploit\-specific details\. Where a contrast shows a systematic directional difference on a clinically consequential action, we notify the model vendor and allow a fixed window before posting publicly\. This is a courtesy notification and a precaution, not a determination that harm occurred: as Sec\.[VI](https://arxiv.org/html/2609.03221#S6)states, this design does not license claims about patient outcomes\.*Governance:*the leaderboard protects held\-out labels, logs submissions for adoption transparency, is not self\-populated, and carries a stated maintenance and sunset plan\. This work situates fairness auditing within recognized assurance frameworks, including the NIST AI Risk Management Framework\[[24](https://arxiv.org/html/2609.03221#bib.bib34)\]and dataset\-diversity guidance\[[53](https://arxiv.org/html/2609.03221#bib.bib31)\]\.

## Data and Code Availability

The evaluation harness is released as version 0\.1\.1, a semantically versioned, pip\-installable package with a publicCITATION\.cff, at[https://github\.com/rohithreddybc/FairMedAgent](https://github.com/rohithreddybc/FairMedAgent), archived on Zenodo at10\.5281/zenodo\.22165979\. The trajectories from which every reported number is recomputed are published at[https://huggingface\.co/datasets/Rohithreddybc/FairMedAgent](https://huggingface.co/datasets/Rohithreddybc/FairMedAgent)\. The development split, its Datasheet, the submission protocol, and the reference leaderboard are published on completion of clinician validation\. The held\-out test split is sealed and scored under the submission protocol\. The synthetic cohort is fully described in Sec\.[IV](https://arxiv.org/html/2609.03221#S4)and the accompanying Datasheet, and the benchmark is citable via its public repository; the dataset identifier/accession is intentionally not printed here pending a separate eligibility determination for a companion dataset descriptor\. The leaderboard URL is registered at that release and is not printed here, since it does not exist until the release is cut; the dataset identifier is added only if and when the companion\-descriptor eligibility determination permits it\.

## Acknowledgment

The clinical band review was performed by a licensed clinician who is not an author and whose identity is available to the editor\.

## References

- \[1\]K\. M\. Hoffman, S\. Trawalter, J\. R\. Axt, and M\. N\. Oliver\(2016\)Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites\.Proceedings of the National Academy of Sciences113\(16\),pp\. 4296–4301\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1516047113)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p1.1)\.
- \[2\]M\. K\. Goyal, N\. Kuppermann, S\. D\. Cleary, S\. J\. Teach, and J\. M\. Chamberlain\(2015\)Racial disparities in pain management of children with appendicitis in emergency departments\.JAMA Pediatrics169\(11\),pp\. 996–1002\.External Links:[Document](https://dx.doi.org/10.1001/jamapediatrics.2015.1915)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p1.1)\.
- \[3\]K\. A\. Schulman, J\. A\. Berlin, W\. Harless, J\. F\. Kerner, S\. Sistrunk, B\. J\. Gersh, R\. Dube, C\. K\. Taleghani, J\. E\. Burke, S\. Williams, J\. M\. Eisenberg, W\. Ayers, and J\. J\. Escarce\(1999\)The effect of race and sex on physicians’ recommendations for cardiac catheterization\.New England Journal of Medicine340\(8\),pp\. 618–626\.External Links:[Document](https://dx.doi.org/10.1056/NEJM199902253400806)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1)\.
- \[4\]C\. Divi, R\. G\. Koss, S\. P\. Schmaltz, and J\. M\. Loeb\(2007\)Language proficiency and adverse events in us hospitals: a pilot study\.International Journal for Quality in Health Care19\(2\),pp\. 60–67\.External Links:[Document](https://dx.doi.org/10.1093/intqhc/mzl069)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.SSS0.Px1.p1.1)\.
- \[5\]J\. A\. Omiye, J\. C\. Lester, S\. Spichak, V\. Rotemberg, and R\. Daneshjou\(2023\)Large language models propagate race\-based medicine\.npj Digital Medicine6\(1\),pp\. 195\.External Links:[Document](https://dx.doi.org/10.1038/s41746-023-00939-z)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1)\.
- \[6\]T\. Zack, E\. Lehman, M\. Suzgun, J\. A\. Rodriguez, L\. A\. Celi, J\. Gichoya, D\. Jurafsky, P\. Szolovits, D\. W\. Bates, R\. E\. Abdulnour, A\. J\. Butte, and E\. Alsentzer\(2024\)Assessing the potential of GPT\-4 to perpetuate racial and gender biases in health care: a model evaluation study\.The Lancet Digital Health6\(1\),pp\. e12–e22\.External Links:[Document](https://dx.doi.org/10.1016/S2589-7500%2823%2900225-X)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1)\.
- \[7\]M\. Omar, S\. Soffer, R\. Agbareia, N\. L\. Bragazzi, D\. U\. Apakama, C\. R\. Horowitz, A\. W\. Charney, R\. Freeman, B\. Kummer, B\. S\. Glicksberg, G\. N\. Nadkarni, and E\. Klang\(2025\)Sociodemographic biases in medical decision making by large language models\.Nature Medicine31\(6\),pp\. 1873–1881\.External Links:[Document](https://dx.doi.org/10.1038/s41591-025-03626-6)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p1.1),[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.3.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p3.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1),[§VI](https://arxiv.org/html/2609.03221#S6.p4.1)\.
- \[8\]G\. Xu, X\. Li, Y\. Chen, Y\. Duan, S\. Wu, H\. Yu, C\. Chiu, J\. Ni, N\. Tang, T\. J\. Li,et al\.\(2026\)A comprehensive survey of ai agents in healthcare\.Journal of Biomedical Informatics,pp\. 105045\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2026.105045),[Link](https://www.sciencedirect.com/science/article/pii/S1532046426000699)Cited by:[§I](https://arxiv.org/html/2609.03221#S1.p2.1),[§II](https://arxiv.org/html/2609.03221#S2.p4.1)\.
- \[9\]T\. Morla, R\. R\. Bellibaltu, M\. Singh, and M\. S\. Kapoor\(2026\)AgentFairBench: do llm agents discriminate when they act?\.arXiv preprint arXiv:2606\.16723\.External Links:2606\.16723Cited by:[2nd item](https://arxiv.org/html/2609.03221#S1.I1.i2.p1.1),[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.2.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p3.1)\.
- \[10\]R\. J\. Young and A\. M\. Matthews\(2026\)EQUITRIAGE: a fairness audit of gender bias in LLM\-based emergency department triage\.arXiv preprint arXiv:2605\.03998\.External Links:2605\.03998Cited by:[2nd item](https://arxiv.org/html/2609.03221#S1.I1.i2.p1.1),[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.4.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[11\]Y\. Zhang, S\. Hou, M\. D\. Ma, W\. Wang, M\. Chen, and J\. Zhao\(2024\)CLIMB: a benchmark of clinical bias in large language models\.arXiv preprint arXiv:2407\.05250\.External Links:2407\.05250Cited by:[2nd item](https://arxiv.org/html/2609.03221#S1.I1.i2.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[12\]K\. Benkirane, J\. Kay, and M\. Perez\-Ortiz\(2024\)How can we diagnose and treat bias in large language models for clinical decision\-making?\.arXiv preprint arXiv:2410\.16574\.External Links:2410\.16574Cited by:[2nd item](https://arxiv.org/html/2609.03221#S1.I1.i2.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[13\]J\. O’Neill, H\. Tabish, V\. Welch, M\. Petticrew, K\. Pottie, M\. Clarke, T\. Evans, J\. Pardo Pardo, E\. Waters, H\. White, and P\. Tugwell\(2014\)Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health\.Journal of Clinical Epidemiology67\(1\),pp\. 56–64\.External Links:[Document](https://dx.doi.org/10.1016/j.jclinepi.2013.08.005)Cited by:[3rd item](https://arxiv.org/html/2609.03221#S1.I1.i3.p1.1),[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1)\.
- \[14\]J\. Gallifant, M\. Afshar, S\. Ameen, Y\. Aphinyanaphongs, S\. Chen, G\. Cacciamani, D\. Demner\-Fushman, D\. Dligach, R\. Daneshjou, C\. Fernandes, L\. H\. Hansen, A\. Landman, L\. Lehmann, L\. G\. McCoy, T\. Miller, A\. Moreno, N\. Munch, D\. Restrepo, G\. Savova, R\. Umeton, J\. W\. Gichoya, G\. S\. Collins, K\. G\. M\. Moons, L\. A\. Celi, and D\. S\. Bitterman\(2025\)The TRIPOD\-LLM reporting guideline for studies using large language models\.Nature Medicine31\(1\),pp\. 60–69\.External Links:[Document](https://dx.doi.org/10.1038/s41591-024-03425-5)Cited by:[4th item](https://arxiv.org/html/2609.03221#S1.I1.i4.p1.1),[§II](https://arxiv.org/html/2609.03221#S2.p4.1),[§IV](https://arxiv.org/html/2609.03221#S4.p1.1)\.
- \[15\]H\. Adam, M\. Y\. Yang, K\. Cato, I\. Baldini, C\. Senteio, L\. A\. Celi, J\. Zeng, M\. Singh, and M\. Ghassemi\(2022\)Write it like you see it: detectable differences in clinical notes by race lead to differential model recommendations\.InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society \(AIES ’22\),pp\. 7–21\.External Links:[Document](https://dx.doi.org/10.1145/3514094.3534203)Cited by:[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[16\]S\. R\. Pfohl, H\. Cole\-Lewis, R\. Sayres, D\. Neal, M\. Asiedu, A\. Dieng, N\. Tomasev, Q\. M\. Rashid, S\. Azizi, N\. Rostamzadeh, L\. G\. McCoy, L\. A\. Celi, Y\. Liu, M\. Schaekermann, A\. Walton, A\. Parrish, C\. Nagpal, P\. Singh, A\. Dewitt, P\. Mansfield, S\. Prakash, K\. Heller, A\. Karthikesalingam, C\. Semturs, J\. Barral, G\. Corrado, Y\. Matias, J\. Smith\-Loud, I\. Horn, and K\. Singhal\(2024\)A toolbox for surfacing health equity harms and biases in large language models\.Nature Medicine30\(12\),pp\. 3590–3600\.External Links:[Document](https://dx.doi.org/10.1038/s41591-024-03258-2)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.5.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p1.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1)\.
- \[17\]R\. Poulain, H\. Fayyaz, and R\. Beheshti\(2024\)Bias patterns in the application of LLMs for clinical decision support: a comprehensive study\.arXiv preprint arXiv:2404\.15149\.External Links:2404\.15149Cited by:[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[18\]J\. J\. Docking, L\. X\. Li, B\. D\. Menz, S\. Bacchi, A\. M\. Hopkins, and M\. J\. Sorich\(2026\)Evaluating the potential of reasoning large language models to perpetuate racial and gender disease stereotypes in health care\.Journal of Medical Internet Research28,pp\. e82256\.External Links:[Document](https://dx.doi.org/10.2196/82256)Cited by:[§II](https://arxiv.org/html/2609.03221#S2.p1.1)\.
- \[19\]S\. Schmidgall, R\. Ziaei, C\. Harris, E\. Reis, J\. Jopling, and M\. Moor\(2024\)AgentClinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments\.arXiv preprint arXiv:2405\.07960\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2405.07960),[Link](https://arxiv.org/abs/2405.07960)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.6.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p2.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1)\.
- \[20\]Y\. Jiang, K\. C\. Black, G\. Geng, D\. Park, J\. Zou, A\. Y\. Ng, and J\. H\. Chen\(2025\)MedAgentBench: a realistic virtual ehr environment to benchmark medical llm agents\.arXiv preprint arXiv:2501\.14654\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.14654),[Link](https://arxiv.org/abs/2501.14654)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.7.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p2.1),[§III\-A](https://arxiv.org/html/2609.03221#S3.SS1.p1.1)\.
- \[21\]J\. S\. Bosmaet al\.\(2025\)The dragon benchmark for clinical nlp\.npj Digital Medicine8\(1\),pp\. 289\.External Links:[Document](https://dx.doi.org/10.1038/s41746-025-01626-x),[Link](https://www.nature.com/articles/s41746-025-01626-x)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.8.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p2.1),[§II](https://arxiv.org/html/2609.03221#S2.p3.1)\.
- \[22\]C\. Wu, P\. Qiu, J\. Liu, H\. Gu, N\. Li, Y\. Zhang, Y\. Wang, and W\. Xie\(2025\)Towards evaluating and building versatile large language models for medicine\.npj Digital Medicine8\(1\),pp\. 58\.External Links:[Document](https://dx.doi.org/10.1038/s41746-024-01390-4),[Link](https://www.nature.com/articles/s41746-024-01390-4)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.9.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p2.1),[§II](https://arxiv.org/html/2609.03221#S2.p3.1)\.
- \[23\]S\. Bedi, H\. Cui, M\. Fuentes, A\. Unell, M\. Wornow, J\. M\. Banda,et al\.\(2026\)Holistic evaluation of large language models for medical tasks with MedHELM\.Nature Medicine32,pp\. 943–951\.Note:PMID: 41559415\. Preprint: arXiv:2505\.23802External Links:[Document](https://dx.doi.org/10.1038/s41591-025-04151-2)Cited by:[TABLE I](https://arxiv.org/html/2609.03221#S2.T1.23.10.1.1),[§II](https://arxiv.org/html/2609.03221#S2.p2.1),[§II](https://arxiv.org/html/2609.03221#S2.p3.1)\.
- \[24\]National Institute of Standards and Technology\(2023\)Artificial intelligence risk management framework \(AI RMF 1\.0\)\.NIST AITechnical Report100\-1,National Institute of Standards and Technology, U\.S\. Department of Commerce,Gaithersburg, MD\.External Links:[Document](https://dx.doi.org/10.6028/NIST.AI.100-1)Cited by:[§II](https://arxiv.org/html/2609.03221#S2.p4.1),[Ethical Considerations](https://arxiv.org/html/2609.03221#Sx1.p1.1)\.
- \[25\]M\. J\. Kusner, J\. R\. Loftus, C\. Russell, and R\. Silva\(2017\)Counterfactual fairness\.InAdvances in Neural Information Processing Systems 30 \(NeurIPS 2017\),pp\. 4066–4076\.External Links:1703\.06856,[Link](https://arxiv.org/abs/1703.06856)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1),[§III\-D](https://arxiv.org/html/2609.03221#S3.SS4.p2.3)\.
- \[26\]S\. Garg, V\. Perot, N\. Limtiaco, A\. Taly, E\. H\. Chi, and A\. Beutel\(2019\)Counterfactual fairness in text classification through robustness\.InProceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society \(AIES ’19\),pp\. 219–226\.External Links:[Document](https://dx.doi.org/10.1145/3306618.3317950),[Link](https://doi.org/10.1145/3306618.3317950)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1)\.
- \[27\]P\. Ma, S\. Wang, and J\. Liu\(2020\)Metamorphic testing and certified mitigation of fairness violations in NLP models\.InProceedings of the Twenty\-Ninth International Joint Conference on Artificial Intelligence \(IJCAI\-20\),pp\. 458–465\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2020/64),[Link](https://www.ijcai.org/proceedings/2020/64)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1)\.
- \[28\]Z\. Chen, J\. M\. Zhang, M\. Hort, M\. Harman, and F\. Sarro\(2024\)Fairness testing: a comprehensive survey and analysis of trends\.ACM Transactions on Software Engineering and Methodology\.Note:arXiv:2207\.10223, 2022External Links:2207\.10223,[Link](https://arxiv.org/abs/2207.10223)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1)\.
- \[29\]G\. R\. Bauer\(2014\)Incorporating intersectionality theory into population health research methodology: challenges and the potential to advance health equity\.Social Science & Medicine110,pp\. 10–17\.External Links:[Document](https://dx.doi.org/10.1016/j.socscimed.2014.03.022)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.p1.1)\.
- \[30\]D\. A\. Vyas, L\. G\. Eisenstein, and D\. S\. Jones\(2020\)Hidden in plain sight—reconsidering the use of race correction in clinical algorithms\.New England Journal of Medicine383\(9\),pp\. 874–882\.External Links:[Document](https://dx.doi.org/10.1056/NEJMms2004740)Cited by:[§III\-C](https://arxiv.org/html/2609.03221#S3.SS3.SSS0.Px1.p1.1)\.
- \[31\]P\. Saleiro, B\. Kuester, L\. Hinkson, J\. London, A\. Stevens, A\. Anisfeld, K\. T\. Rodolfa, and R\. Ghani\(2018\)Aequitas: a bias and fairness audit toolkit\.arXiv preprint arXiv:1811\.05577\.External Links:1811\.05577,[Document](https://dx.doi.org/10.48550/arXiv.1811.05577)Cited by:[§III\-D](https://arxiv.org/html/2609.03221#S3.SS4.p2.3)\.
- \[32\]M\. Hardt, E\. Price, and N\. Srebro\(2016\)Equality of opportunity in supervised learning\.InAdvances in Neural Information Processing Systems 29 \(NIPS 2016\),Vol\.29,pp\. 3315–3323\.External Links:1610\.02413,[Link](https://arxiv.org/abs/1610.02413)Cited by:[§III\-D](https://arxiv.org/html/2609.03221#S3.SS4.p2.3)\.
- \[33\]G\. S\. Collins, K\. G\. M\. Moons, P\. Dhiman, R\. D\. Riley, A\. L\. Beam, B\. Van Calster, M\. Ghassemi, X\. Liu, J\. B\. Reitsma, M\. van Smeden, A\. Boulesteix, J\. C\. Camaradou, L\. A\. Celi, S\. Denaxas, A\. K\. Denniston, B\. Glocker, R\. M\. Golub, H\. Harvey, G\. Heinze, M\. M\. Hoffman, A\. P\. Kengne, E\. Lam, N\. Lee, E\. W\. Loder, L\. Maier\-Hein, B\. A\. Mateen, M\. D\. McCradden, L\. Oakden\-Rayner, J\. Ordish, R\. Parnell, S\. Rose, K\. Singh, L\. Wynants, and P\. Logullo\(2024\)TRIPOD\+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods\.BMJ385,pp\. e078378\.External Links:[Document](https://dx.doi.org/10.1136/bmj-2023-078378)Cited by:[§IV](https://arxiv.org/html/2609.03221#S4.p1.1)\.
- \[34\]K\. G\. M\. Moons, J\. A\. A\. Damen, T\. Kaul, L\. Hooft, C\. Andaur Navarro, P\. Dhiman, A\. L\. Beam, B\. Van Calster, L\. A\. Celi, S\. Denaxas, A\. K\. Denniston, M\. Ghassemi, A\. P\. Kengne, B\. A\. Mateen, M\. D\. McCradden, R\. D\. Riley, K\. Singh, L\. Wynants, and G\. S\. Collins\(2025\)PROBAST\+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods\.BMJ388,pp\. e082505\.External Links:[Document](https://dx.doi.org/10.1136/bmj-2024-082505)Cited by:[§IV](https://arxiv.org/html/2609.03221#S4.p1.1)\.
- \[35\]S\. H\. Meghani, E\. Byun, and R\. M\. Gallagher\(2012\)Time to take stock: a meta\-analysis and systematic review of analgesic treatment disparities for pain in the united states\.Pain Medicine13\(2\),pp\. 150–174\.External Links:[Document](https://dx.doi.org/10.1111/j.1526-4637.2011.01310.x)Cited by:[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p1.1)\.
- \[36\]R\. Hirani, B\. Benkli, C\. A\. Odonkor, Z\. Hirani, T\. Oso, S\. Bohacek, J\. Wiedrick, A\. Hildebrand, C\. Osuagwu, V\. Orhurhu, W\. M\. Hooten, S\. Abdi, and S\. H\. Meghani\(2024\)Racial disparities in opioid prescribing in the united states from 2011 to 2021: a systematic review and meta\-analysis\.Journal of Pain Research17,pp\. 3639–3649\.External Links:[Document](https://dx.doi.org/10.2147/jpr.s477128)Cited by:[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p1.1)\.
- \[37\]A\. Z\. Jacobs and H\. Wallach\(2021\)Measurement and fairness\.InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency \(FAccT ’21\),pp\. 375–385\.External Links:[Document](https://dx.doi.org/10.1145/3442188.3445901)Cited by:[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p1.1)\.
- \[38\]T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daumé III, and K\. Crawford\(2021\)Datasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.External Links:[Document](https://dx.doi.org/10.1145/3458723)Cited by:[§IV\-A](https://arxiv.org/html/2609.03221#S4.SS1.p2.1),[§IV\-E](https://arxiv.org/html/2609.03221#S4.SS5.p1.1)\.
- \[39\]Z\. Obermeyer, B\. Powers, C\. Vogeli, and S\. Mullainathan\(2019\)Dissecting racial bias in an algorithm used to manage the health of populations\.Science366\(6464\),pp\. 447–453\.External Links:[Document](https://dx.doi.org/10.1126/science.aax2342)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p1.1)\.
- \[40\]Emergency Nurses Association\(2023\)Emergency severity index handbook\.5 edition\.Note:Five\-level triage algorithm; the high\-risk\-situation and danger\-zone\-vitals criteria relied on here are carried forward from version 4External Links:[Link](https://media.emscimprovement.center/documents/Emergency_Severity_Index_Handbook.pdf)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p1.1)\.
- \[41\]American College of Emergency Physicians\(2017\)Optimizing the treatment of acute pain in the emergency department\.Clinical Policy,American College of Emergency Physicians\.Note:Multimodal non\-opioid first\-line therapy; opioids reserved for refractory painExternal Links:[Link](https://www.acep.org/patient-care/clinical-policies/)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p1.1)\.
- \[42\]American Diabetes Association Professional Practice Committee\(2024\)16\. Diabetes care in the hospital: standards of care in diabetes—2024\.Diabetes Care47\(Supplement 1\),pp\. S295–S306\.External Links:[Document](https://dx.doi.org/10.2337/dc24-S016)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p1.1)\.
- \[43\]G\. E\. Umpierrez, G\. M\. Davis, N\. A\. ElSayed,et al\.\(2024\)Hyperglycemic crises in adults with diabetes: a consensus report\.Diabetes Care47\(8\),pp\. 1257–1275\.External Links:[Document](https://dx.doi.org/10.2337/dci24-0032)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p1.1)\.
- \[44\]A\. F\. Hayes and K\. Krippendorff\(2007\)Answering the call for a standard reliability measure for coding data\.Communication Methods and Measures1\(1\),pp\. 77–89\.External Links:[Document](https://dx.doi.org/10.1080/19312450709336664)Cited by:[§IV\-B](https://arxiv.org/html/2609.03221#S4.SS2.p5.1)\.
- \[45\]A\. C\. Cameron, J\. B\. Gelbach, and D\. L\. Miller\(2008\)Bootstrap\-based improvements for inference with clustered errors\.The Review of Economics and Statistics90\(3\),pp\. 414–427\.External Links:[Document](https://dx.doi.org/10.1162/rest.90.3.414)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p3.1)\.
- \[46\]M\. D\. Webb\(2023\)Reworking wild bootstrap\-based inference for clustered errors\.Canadian Journal of Economics/Revue canadienne d’économique56\(3\),pp\. 839–858\.Note:Circulated 2013 as Queen’s Economics Department Working Paper No\. 1315External Links:[Document](https://dx.doi.org/10.1111/caje.12661)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p3.1)\.
- \[47\]J\. G\. MacKinnon and M\. D\. Webb\(2017\)Wild bootstrap inference for wildly different cluster sizes\.Journal of Applied Econometrics32\(2\),pp\. 233–254\.External Links:[Document](https://dx.doi.org/10.1002/jae.2508)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p3.1)\.
- \[48\]M\. W\. Fagerland, S\. Lydersen, and P\. Laake\(2013\)The McNemar test for binary matched\-pairs data: mid\-p and asymptotic are better than exact conditional\.BMC Medical Research Methodology13,pp\. 91\.External Links:[Document](https://dx.doi.org/10.1186/1471-2288-13-91)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p5.1)\.
- \[49\]Y\. Benjamini and D\. Yekutieli\(2001\)The control of the false discovery rate in multiple testing under dependency\.The Annals of Statistics29\(4\),pp\. 1165–1188\.External Links:[Document](https://dx.doi.org/10.1214/aos/1013699998)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p6.1)\.
- \[50\]Y\. Benjamini and Y\. Hochberg\(1995\)Controlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B \(Methodological\)57\(1\),pp\. 289–300\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1995.tb02031.x)Cited by:[§IV\-D](https://arxiv.org/html/2609.03221#S4.SS4.p6.1)\.
- \[51\]M\. Mitchell, S\. Wu, A\. Zaldivar, P\. Barnes, L\. Vasserman, B\. Hutchinson, E\. Spitzer, I\. D\. Raji, and T\. Gebru\(2019\)Model cards for model reporting\.InProceedings of the Conference on Fairness, Accountability, and Transparency \(FAT\* ’19\),pp\. 220–229\.External Links:[Document](https://dx.doi.org/10.1145/3287560.3287596)Cited by:[§IV\-E](https://arxiv.org/html/2609.03221#S4.SS5.p1.1)\.
- \[52\]J\. Yuan, H\. Li, X\. Ding, W\. Xie, Y\. Li, W\. Zhao, K\. Wan, J\. Shi, X\. Hu, and Z\. Liu\(2025\)Understanding and mitigating numerical sources of nondeterminism in LLM inference\.arXiv preprint arXiv:2506\.09501\.Note:Reports accuracy variation of up to nine points, and response\-length differences of thousands of tokens, attributable to GPU count, GPU type and batch size with sampling held fixedExternal Links:[Link](https://arxiv.org/abs/2506.09501)Cited by:[§V\-B](https://arxiv.org/html/2609.03221#S5.SS2.p2.1)\.
- \[53\]J\. E\. Alderman, J\. Palmer, E\. Laws, M\. D\. McCradden, J\. Ordish, M\. Ghassemi, S\. R\. Pfohl, N\. Rostamzadeh, H\. Cole\-Lewis, B\. Glocker, M\. Calvert, T\. Panch, J\. W\. Gichoya, V\. I\. Madai, B\. A\. Mateen, X\. Liu, A\. K\. Denniston, A\. Ganapathi, and STANDING Together Consortium\(2025\)Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations\.The Lancet Digital Health7\(1\),pp\. e64–e88\.External Links:[Document](https://dx.doi.org/10.1016/S2589-7500%2824%2900224-3)Cited by:[Ethical Considerations](https://arxiv.org/html/2609.03221#Sx1.p1.1)\.