Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA

# Which Medical Questions Deserve Rationales?Perturbation-Sensitive Selection for Robust QA
Source: [https://arxiv.org/html/2609.09684](https://arxiv.org/html/2609.09684)
Dayou YuVasile RusAffiliation:Department of Computer Science, University of MemphisAffiliation:ywu10@memphis\.edu dayou\.yu@memphis\.edu vrus@memphis\.edu

###### Abstract

Medical question\-answering datasets often contain answer labels, whereas high\-quality rationales remain scarce, noisy, or costly to validate\. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already\-labeled questions should receive rationale supervision under a fixed token budget\. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected\. We propose root\-mean\-square Robustness\-based Sample Prioritization \(RMS\-RSP\), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold\-versus\-best\-distractor margin\. Across five medical QA datasets, MedGemma\-4B\-IT, three training seeds, ten budgeted non\-RSP selectors, and an unbudgeted full\-supervision reference, RMS\-RSP provides a deliberately qualified result\. Its locked\-budget accuracy is 60\.61% on average versus 60\.08% for Random, with a statistically resolved gain only on AfriMed\-QA \(\+1\.44 points\)\. Its full\-budget accuracy area is not better than Random\. However, after three answer\-option reorderings, RMS\-RSP improves robust accuracy and semantic consistency by 1\.91 and 2\.85 points on average, respectively, with the same direction on all five datasets\. Training on every pool rationale raises macro accuracy to 63\.74%, but consumes 29–254 times more rationale tokens and does not uniformly improve robustness\. These findings do not establish universal accuracy gains; they instead suggest that rationale\-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes\.

## 1Introduction

Medical QA datasets commonly provide answer labels at much greater scale than carefully checked explanations\. Rationales are longer, require domain expertise to write or verify, and may contain irrelevant or incorrect intermediate statements even when the final answer is correct\. The relevant allocation problem is therefore not only*which questions should be labeled*, as in conventional active learning\([Settles, 2009](https://arxiv.org/html/2609.09684#bib.bib1)\), but*which already\-labeled questions deserve additional rationale supervision*\. This distinction matters in medical settings: applying all available rationales indiscriminately can spend substantial annotation or training budget while exposing the model to noisy reasoning traces\.

Recent reasoning\-data selectors rank traces by answer uncertainty, likelihood, local step compatibility, or early training dynamics\([Goncharov et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib12);[Yang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib13);[Just et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib14);[Wang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib15);[Jin et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib16)\)\. These methods have advanced data\-efficient reasoning distillation, but most were developed for mathematics, coding, or general science\. Moreover, their scores do not directly ask whether the rationale is coupled to the medical decision boundary\. In multiple\-choice medical QA, this boundary is especially important: a useful rationale should support the correct clinical option over its strongest distractor rather than merely be fluent or difficult\.

We revisit Robustness\-based Sample Prioritization \(RSP\) with a normalized, rationale\-local score\. Given an answer\-trained model and a candidate rationale, we inject RMS\-scaled Gaussian noise into the rationale\-token hidden states at several late layers\. We then measure how much the gold\-versus\-best\-distractor log\-probability margin changes\. The canonical RMS\-RSP score is the root mean square of these shifts\. It is scale\-normalized, sensitive to both positive and negative boundary changes, and tied to the tokens that would be acquired as supervision\.

Our evaluation is designed around the limits of the current evidence\. We use five medical QA datasets spanning African medical examinations, multilingual medical exams, underrepresented specialties, biomedical literature, and Indian entrance examinations\([Nimo et al\., 2025](https://arxiv.org/html/2609.09684#bib.bib8);[Alonso et al\., 2024](https://arxiv.org/html/2609.09684#bib.bib9);[Kim et al\., 2024](https://arxiv.org/html/2609.09684#bib.bib10);[Jin et al\., 2019](https://arxiv.org/html/2609.09684#bib.bib11);[Pal et al\., 2022](https://arxiv.org/html/2609.09684#bib.bib7)\)\. We compare against Random, answer entropy and margin, rationale length, and five recent reasoning\-data selectors\. We report locked\-budget accuracy, macro\-F1, full\-curve Token\-AUBC, and invariance to answer\-option permutations\. Our contributions are:

- •a precise formulation of*budgeted rationale selection*for answer\-labeled medical QA, separating answer supervision from rationale supervision;
- •RMS\-RSP, a rationale\-local, relative\-scale perturbation score based on gold–distractor margin shifts, together with a signed ablation;
- •a five\-dataset comparison showing heterogeneous standard accuracy but consistent improvements in option\-order robust accuracy and semantic consistency, together with a high\-resource all\-rationales reference\.

## 2Related work

#### Rationale supervision in medical QA\.

Chain\-of\-thought prompting and rationale fine\-tuning can improve multi\-step reasoning\([Wei et al\., 2022](https://arxiv.org/html/2609.09684#bib.bib2);[Zelikman et al\., 2022](https://arxiv.org/html/2609.09684#bib.bib3)\), but generated explanations need not faithfully describe the computation that produced an answer\([Turpin et al\., 2023](https://arxiv.org/html/2609.09684#bib.bib4)\)\. This concern is amplified in healthcare, where an incorrect intermediate claim can be consequential\. Medical benchmarks vary widely in explanation provenance: MedMCQA supplies short explanations, MedExpQA provides physician\-written reference explanations, MedExQA provides explanation pairs, and PubMedQA pairs decisions with article conclusions\([Pal et al\., 2022](https://arxiv.org/html/2609.09684#bib.bib7);[Alonso et al\., 2024](https://arxiv.org/html/2609.09684#bib.bib9);[Kim et al\., 2024](https://arxiv.org/html/2609.09684#bib.bib10);[Jin et al\., 2019](https://arxiv.org/html/2609.09684#bib.bib11)\)\. We treat these rationales as an offline acquisition oracle rather than assuming that every rationale is equally useful\.

#### Reasoning\-data selection\.

Uncertainty sampling selects difficult inputs from model outputs\. Complexity\-aware fine\-tuning uses answer entropy to reserve reasoning supervision for complex items\([Goncharov et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib12)\)\. RSR balances token rank and surprisal to identify trajectories aligned with but informative to a student\([Yang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib13)\)\. LALP scores each step from a restricted local context\([Just et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib14)\), while ASLEC\-DROP removes low\-probability first\-step tokens that confound naturalness scores with step length\([Wang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib15)\)\. TEMP uses losses under random and fine\-tuning\-direction parameter perturbations and observes that useful reasoning traces can be identified early\([Jin et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib16)\)\. These are close baselines because they address the same supervision\-allocation question\. RSP differs by perturbing activations at rationale positions and measuring an answer\-boundary response\.

#### Multiple\-choice robustness\.

LLMs can change predictions when answer options are reordered even though their semantic content is unchanged\([Pezeshkpour and Hruschka, 2024](https://arxiv.org/html/2609.09684#bib.bib17);[Zheng et al\., 2024](https://arxiv.org/html/2609.09684#bib.bib18)\)\. We therefore evaluate more than original\-order accuracy\. Option permutations are not claimed to simulate every clinical distribution shift; they are a controlled invariance test aligned with RSP’s decision\-boundary motivation\.

## 3Budgeted rationale selection

### 3\.1Problem setup

Let the pool be𝒟=\{\(xi,𝒪i,yi,ri\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},\\mathcal\{O\}\_\{i\},y\_\{i\},r\_\{i\}\)\\\}\_\{i=1\}^\{N\}, wherexix\_\{i\}is a medical question \(and context, when present\),𝒪i\\mathcal\{O\}\_\{i\}is its option set,yiy\_\{i\}is the known correct option, andrir\_\{i\}is a candidate rationale\. We first train an answer\-only modelfθ0f\_\{\\theta\_\{0\}\}on all\(xi,yi\)\(x\_\{i\},y\_\{i\}\)pairs\. Rationales are not used in this stage\.

Each rationale has token costcic\_\{i\}\. A selector ranks candidates and choosesS⊆\{1,…,N\}S\\subseteq\\\{1,\\ldots,N\\\}such that∑i∈Sci≤B\\sum\_\{i\\in S\}c\_\{i\}\\leq B\. Only\{ri:i∈S\}\\\{r\_\{i\}:i\\in S\\\}are unlocked for the rationale\-training branch\. In our offline experiments,rir\_\{i\}is visible to rationale\-aware scoring functions; consequently, the setup models selection from existing or cheaply generated candidate explanations for validation/training, not selection before any rationale has been produced\. This scope distinction is central to our claims\.

### 3\.2Rationale\-local perturbations

For candidateii, we append the rationale and the suffix “Final Answer:” to the question and options\. Letai​ja\_\{ij\}be the logit for optionjjat the answer position\. The clean gold–distractor margin is

mi=ai​yi−maxj≠yi⁡ai​j\.m\_\{i\}=a\_\{iy\_\{i\}\}\-\\max\_\{j\\neq y\_\{i\}\}a\_\{ij\}\.\(1\)At transformer layerℓ\\ell, letHi\(ℓ\)∈ℝT×dH\_\{i\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{T\\times d\}be the hidden states andMi,R∈\{0,1\}T×1M\_\{i,R\}\\in\\\{0,1\\\}^\{T\\times 1\}mask only the rationale tokens\. We perturb

H~i\(ℓ,k\)=Hi\(ℓ\)\+α​RMS⁡\(Hi,R\(ℓ\)\)​\(Mi,R⊙Zi\(ℓ,k\)\),Zi\(ℓ,k\)∼𝒩⁡\(0,I\),\\widetilde\{H\}\_\{i\}^\{\(\\ell,k\)\}=H\_\{i\}^\{\(\\ell\)\}\+\\alpha\\,\\operatorname\{RMS\}\\\!\\left\(H\_\{i,R\}^\{\(\\ell\)\}\\right\)\\left\(M\_\{i,R\}\\odot Z\_\{i\}^\{\(\\ell,k\)\}\\right\),\\quad Z\_\{i\}^\{\(\\ell,k\)\}\\sim\\mathcal\{N\}\(0,I\),\(2\)whereRMS⁡\(Hi,R\(ℓ\)\)\\operatorname\{RMS\}\(H\_\{i,R\}^\{\(\\ell\)\}\)is computed across masked tokens and hidden dimensions\. Relative scaling avoids comparing a fixed absolute noise level across layers with different activation magnitudes\. Restricting noise to rationale positions asks how strongly the answer boundary depends on that candidate explanation rather than on the question generally\.

Letm~i\(ℓ,k\)\\widetilde\{m\}\_\{i\}^\{\(\\ell,k\)\}be the perturbed margin anddi\(ℓ,k\)=mi−m~i\(ℓ,k\)d\_\{i\}^\{\(\\ell,k\)\}=m\_\{i\}\-\\widetilde\{m\}\_\{i\}^\{\(\\ell,k\)\}\. For layersℒ\\mathcal\{L\}andKKperturbations per layer, the canonical score is

siRMS\-RSP=1\|ℒ\|​K​∑ℓ∈ℒ∑k=1K\(di\(ℓ,k\)\)2\.s\_\{i\}^\{\\text\{RMS\-RSP\}\}=\\sqrt\{\\frac\{1\}\{\|\\mathcal\{L\}\|K\}\\sum\_\{\\ell\\in\\mathcal\{L\}\}\\sum\_\{k=1\}^\{K\}\\left\(d\_\{i\}^\{\(\\ell,k\)\}\\right\)^\{2\}\}\.\(3\)We select larger scores first, greedily skipping an item when its cost would exceed the remaining token budget\. RMS magnitude is direction\-agnostic: either a reduction or increase in the gold margin indicates that the boundary is locally sensitive to the rationale representation\. Our Signed\-RSP ablation instead uses the mean signed drop,1\|ℒ\|​K​∑ℓ,kdi\(ℓ,k\)\\frac\{1\}\{\|\\mathcal\{L\}\|K\}\\sum\_\{\\ell,k\}d\_\{i\}^\{\(\\ell,k\)\}\.

### 3\.3Downstream rationale training

Starting from the answer\-only adapter, each selected item contributes an answer replay record and a rationale\-generation record ending in the correct answer\. We additionally sample one answer\-only replay item from the unselected pool per selected item\. This paired design reduces the chance that a selector wins merely by changing the amount of answer supervision\. At inference, the model directly scores answer letters without generating a rationale\.

## 4Experimental design

### 4\.1Datasets and splits

Table[1](https://arxiv.org/html/2609.09684#S4.T1)summarizes the frozen splits\. We use the English subset of MedExpQA and the expert multiple\-choice portion of AfriMed\-QA\. Multi\-answer AfriMed\-QA rows are removed\. MedExQA and PubMedQA use deterministic derived splits after exact deduplication; the others preserve official test partitions where available\. No test item is used to choose a token budget\. The MedMCQA test set had been evaluated in earlier exploratory work, so its result is protocol\-aligned rather than a pristine confirmation\.

Table 1:Datasets and frozen protocol\. “RSPnn” is the number of rationales selected by canonical RMS\-RSP at the development\-locked token budget\.
### 4\.2Model, acquisition, and training

All runs use MedGemma\-4B\-IT\([Sellergren et al\., 2025](https://arxiv.org/html/2609.09684#bib.bib5)\)with LoRA adapters\([Hu et al\., 2022](https://arxiv.org/html/2609.09684#bib.bib6)\)\. The answer\-only adapter is trained for one epoch on every pool answer \(r=16r=16, LoRAα=32\\alpha=32, learning rate2×10−42\\times 10^\{\-4\}\)\. Each rationale branch starts from that adapter and trains for two epochs at10−410^\{\-4\}with sequence length 1,024, batch size 1, gradient accumulation 8, and one\-to\-one unselected answer replay\. We use training seeds 13, 23, and 37\.

For RMS\-RSP,ℒ=\{−2,−4,−8\}\\mathcal\{L\}=\\\{\-2,\-4,\-8\\\},K=4K=4, and perturbation scaleα=0\.10\\alpha=0\.10\. Candidate budgets are 256, 512, and 1,024 rationale tokens\. For each dataset, development data select one shared budget by maximizing RMS\-RSP accuracy minus the strongest non\-RSP selector at that budget, breaking ties toward fewer tokens\. We then evaluate all methods once at the locked budget\. Because this rule is centered on RMS\-RSP, we also report Token\-AUBC over all budgets to prevent a favorable single budget from carrying the conclusion\.

Random uses three acquisition seeds crossed with the three training seeds \(nine runs per budget\)\. Deterministic selectors use three training seeds\. The same selected set is used across downstream seeds; random acquisition varies independently\.

As a high\-resource reference, we additionally train from the same answer\-only adapters on every pool rationale for the same two epochs\. This full\-supervision condition consumes 29,399–129,935 scored rationale tokens, depending on the dataset, and has no unselected replay pool\. It is neither token\- nor update\-matched to the budgeted branches and is therefore reported as an unbudgeted reference rather than a competing selector\.

### 4\.3Baselines

We compare with answer entropy, negative top\-two answer margin, rationale length, and Random\. Five recent selectors are adapted to the shared candidate\-rationale pool:

- •Complexity\-aware FTranks by answer\-position vocabulary entropy\([Goncharov et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib12)\)\.
- •RSRuses token\-rank over surprisal, with the sign reversed so larger scores are selected\([Yang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib13)\)\.
- •LALPaverages step likelihood from the question and the preceding 25% of steps\([Just et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib14)\)\.
- •ASLEC\-DROPaverages rationale\-token likelihood after dropping the first token of each step\([Wang et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib15)\)\.
- •TEMPcombines losses under calibrated random parameter perturbations and checkpoints extrapolated along a small rationale\-LoRA direction\([Jin et al\., 2026](https://arxiv.org/html/2609.09684#bib.bib16)\)\.

Because the medical datasets lack consistent step annotations, LALP and ASLEC share a deterministic newline\-and\-sentence segmentation\. These are protocol adaptations rather than claims of exact reproduction under the original models and datasets\.

### 4\.4Metrics and uncertainty

The primary task metric is multiple\-choice accuracy; macro\-F1 is secondary\. We also compute Brier score\([Brier, 1950](https://arxiv.org/html/2609.09684#bib.bib19)\)and expected calibration error\([Guo et al\., 2017](https://arxiv.org/html/2609.09684#bib.bib20)\)in the released result artifacts\.Token\-AUBCis trapezoidal area under test accuracy at 0, 256, 512, and 1,024 acquired rationale tokens, normalized by 1,024; the zero\-token point is the answer\-only model\.

For option robustness, each item receives three deterministic, distinct, non\-identity permutations\. Predictions are mapped back to the original semantic options\.ReAccrequires a correct prediction on the original and all three permutations\.ReConrequires the same semantic prediction across all four versions, irrespective of correctness\. No model is retrained for this evaluation\. Paired 95% intervals use 10,000 hierarchical bootstrap draws over downstream seeds and test items; comparisons with Random additionally resample acquisition seeds\.

## 5Results

### 5\.1Standard accuracy is positive in some settings, not universal

Table[2](https://arxiv.org/html/2609.09684#S5.T2)reports the shared development\-locked budget for each dataset\. Canonical RMS\-RSP is best on AfriMed\-QA \(64\.16%\) and exceeds Random by 1\.44 points with a 95% interval of\[0\.21,2\.77\]\[0\.21,2\.77\]\. Its point estimate also exceeds Random on MedExpQA \(\+0\.09\) and PubMedQA \(\+1\.61\), ties on MedExQA, and trails by 0\.46 on MedMCQA\. The five\-dataset macro average is 60\.61% versus 60\.08% for Random\. Only AfriMed\-QA resolves a nonzero accuracy difference, so the evidence does not support a claim of consistent accuracy dominance\.

Table 2:Test accuracy \(%, mean±\\pmstandard deviation across three training seeds\)\. Random averages3×33\\times 3acquisition/training runs\. Bold marks the best token\-budgeted method; All rationales is an unbudgeted reference\.Macro\-F1 tells a similarly restrained story\. RMS\-RSP averages 55\.71, essentially matching Random at 55\.68\. Signed\-RSP reaches 56\.50, while answer margin is best at 56\.86\. Thus the canonical score’s main empirical advantage is not general discrimination performance\.

### 5\.2Boundary\-sensitive selection improves option\-order robustness

Table[3](https://arxiv.org/html/2609.09684#S5.T3)shifts from one locked operating point to full\-budget efficiency and controlled invariance\. LALP has the best accuracy Token\-AUBC \(0\.6045\); canonical RMS\-RSP is below Random \(0\.5960 versus 0\.6021\)\. In contrast, RMS\-RSP has the best macroReAcc\(0\.4260\) andReCon\(0\.5694\), improving over Random by 1\.91 and 2\.85 percentage points\. The robustness gains are directionally positive on every dataset \(Figure[1](https://arxiv.org/html/2609.09684#S5.F1)\)\. AfriMed\-QA resolves both differences; MedExQA’sReAccinterval touches zero at its lower endpoint\. The remaining intervals include zero\.

Table 3:Five\-dataset macro averages\. Token\-AUBC uses original\-order accuracy\. Bold marks the best budgeted method; All rationales is an unbudgeted reference and has no Token\-AUBC\.Figure 1:Canonical RMS\-RSP minus Random in percentage points\. Points are means; bars are paired 95% hierarchical bootstrap intervals over training seeds, Random acquisition seeds, and test items\. Robustness differences are positive on all five datasets, whereas original\-order accuracy is heterogeneous\.The metric separation is informative\. RMS magnitude rewards rationales whose representations are strongly coupled to the gold–distractor boundary, regardless of the direction of a particular perturbation\. That coupling is plausibly useful for learning a less position\-dependent answer rule, but it does not guarantee that every selected rationale improves clean accuracy\. We treat this as an empirical interpretation, not a causal proof\.

### 5\.3Signed versus RMS aggregation

The signed ablation is better on locked\-budget macro accuracy \(60\.87 versus 60\.61\) and macro\-F1 \(56\.50 versus 55\.71\), while canonical RMS is substantially better on robustness \(ReAcc42\.60 versus 40\.23;ReCon56\.94 versus 53\.15\)\. This tradeoff supports preserving perturbation magnitude when the goal is invariance, but also shows that the aggregation choice should be tied to the deployment objective\. A single RSP variant is not uniformly best\.

### 5\.4Full supervision improves average performance at much higher cost

The all\-rationales reference reaches 63\.74% macro accuracy and 59\.84 macro\-F1, exceeding RMS\-RSP by 3\.12 and 4\.13 points, respectively\. It also improves macroReAccby 2\.79 points andReConby 1\.67 points\. These averages conceal substantial heterogeneity \(Table[4](https://arxiv.org/html/2609.09684#S5.T4)\)\. Only MedMCQA resolves a nonzero all\-rationales gain over RMS\-RSP on all three metrics:\+6\.21\+6\.21accuracy points \(95%95\\%CI\[4\.93,7\.45\]\[4\.93,7\.45\]\),\+8\.41\+8\.41ReAccpoints \(\[5\.47,11\.03\]\[5\.47,11\.03\]\), and\+10\.31\+10\.31ReConpoints \(\[6\.37,13\.63\]\[6\.37,13\.63\]\)\. Conversely, on PubMedQA the full reference gains 1\.50 accuracy points but loses 4\.83ReAccand 6\.83ReConpoints; all three intervals include zero, and robustness varies strongly across seeds\.

Table 4:Unbudgeted all\-rationales reference\. Token multiple is relative to the locked RMS\-RSP budget; deltas are All rationales minus RMS\-RSP in percentage points\. Only the three MedMCQA intervals exclude zero\.

## 6Discussion and limitations

#### What the evidence supports\.

The strongest defensible claim is narrow: under a fixed rationale\-token budget, selecting examples whose rationale\-token representations exert a large local effect on the gold–distractor boundary can improve invariance to answer\-option reorderings\. The evidence does not support “RSP consistently improves medical QA accuracy\.” Accuracy gains are dataset\- and budget\-dependent, and only AfriMed\-QA has a clearly nonzero locked\-budget gain over Random\.

#### Practical interpretation\.

RSP is best viewed as a curation layer for candidate explanations that already exist—for example, archived dataset rationales or inexpensive machine\-generated drafts—before scarce expert validation or downstream training budget is spent\. The present experiment measures the value of selecting rationale tokens for training; it does not measure clinician annotation time or the quality improvement from expert rewriting\. A prospective study should compare total generation, review, and correction cost\.

#### Full\-supervision reference\.

Using every rationale improves average discrimination and robustness, showing that additional rationale supervision can be valuable when cost is unconstrained\. However, it requires 29–254 times the locked token budget, its advantage is concentrated in MedExpQA and MedMCQA, and it reduces the PubMedQA robustness point estimates\. The appropriate claim is therefore efficiency under scarcity: RMS\-RSP often approaches the high\-resource reference with far fewer rationale tokens, not that selection universally outperforms full supervision\.

#### Candidate\-rationale visibility\.

The selector reads each rationale before deciding whether to unlock it for downstream training\. This is appropriate for offline curation but not for classic active learning in which the annotation does not yet exist\. Calling the method pre\-annotation acquisition without this qualification would overstate its scope\. A future proxy that scores questions without candidate rationales, or uses cheap drafts before expert review, is needed for that setting\.

#### Experimental limits\.

We evaluate one 4B medical model, three training seeds, and relatively small acquisition pools\. The token budgets sometimes unlock very few rationales \(four on AfriMed\-QA at 512 tokens\), increasing sensitivity to individual traces\. The all\-rationales reference is neither token\- nor update\-matched: processing many more rationales also entails many more gradient updates, so it is a high\-resource comparison rather than a causal estimate of selection quality\. MedExQA and PubMedQA use derived splits, and MedMCQA is not a pristine confirmatory test\. Development selection explicitly favors the canonical method; Token\-AUBC mitigates but does not remove that concern\. We do not correct for multiple comparisons\. Option permutations probe a real MCQ failure mode but are narrower than clinical distribution shift, factuality, harm, or calibration under deployment\. Dataset rationales are treated as supervision without new clinician auditing, so we cannot attribute gains to clinical explanation quality\. Finally, these are benchmark experiments and do not validate the model for diagnosis or patient care\.

## 7Conclusion

We formulated budgeted rationale selection for answer\-labeled medical QA and evaluated a rationale\-local perturbation score against recent reasoning\-data selectors on five datasets\. Canonical RMS\-RSP does not dominate standard accuracy or Token\-AUBC\. Training on every rationale improves average performance but costs 29–254 times more rationale tokens and is not uniformly more robust\. Within the low\-budget comparison, RMS\-RSP’s reproducible advantage is a consistent increase in robustness and semantic consistency under answer\-option permutations\. This result suggests a promising, appropriately limited role for representation sensitivity: not as a universal difficulty score, but as a mechanism\-aligned signal for selecting rationale supervision when stable medical decisions matter\.

## References

- Alonsoet al\.\(2024\)I\. Alonso, M\. Oronoz, and R\. AgerriMedExpQA: multilingual benchmarking of large language models for medical question answering\.Artificial Intelligence in Medicine155,pp\. 102938\.External Links:[Document](https://dx.doi.org/10.1016/j.artmed.2024.102938)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p4.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Brier \(1950\)G\. W\. BrierVerification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.Cited by:[§4\.4](https://arxiv.org/html/2609.09684#S4.SS4.p1.1)\.
- Goncharovet al\.\(2026\)A\. Goncharov, D\. Vyazhev, P\. Sychev, E\. Khalafyan, and A\. ZaytsevComplexity\-aware fine\-tuning\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 682–696\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.34)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p2.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2609.09684#S4.I1.i1.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 1321–1330\.Cited by:[§4\.4](https://arxiv.org/html/2609.09684#S4.SS4.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§4\.2](https://arxiv.org/html/2609.09684#S4.SS2.p1.1)\.
- Jinet al\.\(2026\)H\. H\. Jin, W\. Yang, M\. Ghaffari, C\. Morato, and B\. MirzasoleimanReasoning quality emerges early: data curation for reasoning models\.InProceedings of the 43rd International Conference on Machine Learning,Note:arXiv:2606\.26797Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p2.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px2.p1.1),[5th item](https://arxiv.org/html/2609.09684#S4.I1.i5.p1.1)\.
- Jinet al\.\(2019\)Q\. Jin, B\. Dhingra, Z\. Liu, W\. W\. Cohen, and X\. LuPubMedQA: a dataset for biomedical research question answering\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2567–2577\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1259)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p4.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Justet al\.\(2026\)H\. A\. Just, M\. Ko, and R\. JiaThe signal is in the steps: local scoring for reasoning data selection\.arXiv preprint arXiv:2510\.03988\.Note:Version 2Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p2.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px2.p1.1),[3rd item](https://arxiv.org/html/2609.09684#S4.I1.i3.p1.1)\.
- Kimet al\.\(2024\)Y\. Kim, J\. Wu, Y\. Abdulle, and H\. WuMedExQA: medical question answering benchmark with multiple explanations\.InProceedings of the 23rd Workshop on Biomedical Natural Language Processing,pp\. 167–181\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.bionlp-1.14)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p4.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Nimoet al\.\(2025\)C\. Nimo, T\. Olatunji, A\. T\. Owodunni, T\. Abdullahi, E\. Ayodele, M\. Sanni, E\. C\. Aka, F\. Omofoye, F\. Yuehgoh, T\. Faniran,et al\.AfriMed\-QA: a pan\-african, multi\-specialty, medical question\-answering benchmark dataset\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1948–1973\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.96)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p4.1)\.
- Palet al\.\(2022\)A\. Pal, L\. K\. Umapathi, and M\. SankarasubbuMedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.InProceedings of the Conference on Health, Inference, and Learning,Proceedings of Machine Learning Research, Vol\.174,pp\. 248–260\.Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p4.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Pezeshkpour and Hruschka \(2024\)P\. Pezeshkpour and E\. HruschkaLarge language models sensitivity to the order of options in multiple\-choice questions\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 2006–2017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.130)Cited by:[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px3.p1.1)\.
- Sellergrenet al\.\(2025\)A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.MedGemma technical report\.arXiv preprint arXiv:2507\.05201\.Note:Revised April 2026Cited by:[§4\.2](https://arxiv.org/html/2609.09684#S4.SS2.p1.1)\.
- Settles \(2009\)B\. SettlesActive learning literature survey\.Computer Sciences Technical Report 1648, University of Wisconsin–Madison\.Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p1.1)\.
- Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 74952–74965\.Cited by:[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)B\. Wang, R\. Miao, C\. Shen, S\. Yan, K\. Liu, X\. Li, X\. Yuan, S\. Fan, J\. Zhang, and J\. YeOn the step length confounding in LLM reasoning data selection\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 18443–18457\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.918)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p2.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px2.p1.1),[4th item](https://arxiv.org/html/2609.09684#S4.I1.i4.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)Y\. Yang, M\. Lai, W\. Zhao, X\. Fan, Z\. Xi, M\. Wu, C\. Huang, J\. Zhao, H\. Lv, J\. Tong,et al\.Which reasoning trajectories teach students to reason better? a simple metric of informative alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 42123–42150\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1950)Cited by:[§1](https://arxiv.org/html/2609.09684#S1.p2.1),[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2609.09684#S4.I1.i2.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. GoodmanSTaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 15476–15488\.Cited by:[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.09684#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AAdditional implementation details

Rationale steps are segmented deterministically using newline boundaries followed by a sentence heuristic\. At most 384 rationale tokens are scored\. RSP uses one clean pass and 12 noisy passes per candidate \(three layers, four perturbations each\)\. A fixed suffix requests the final answer, and only single\-token option letters are scored\. The answer\-only and rationale branches share the same maximum sequence length \(1,024\), optimizer family \(AdamW\), cosine learning\-rate schedule, and 3% warmup\. Selected rationale branches contain three records per selected item: selected answer replay, rationale target, and one randomly sampled unselected answer replay\. Random acquisition seeds are 13, 23, and 37, crossed with downstream seeds 13, 23, and 37\.

## Appendix BComplete macro\-F1 results

Table 5:Test macro\-F1 \(%, mean across three downstream seeds; Random averages nine runs\)\. Bold marks the best budgeted method; All rationales is unbudgeted\.
## Appendix CDataset\-level Token\-AUBC and robustness

Table 6:Accuracy Token\-AUBC over 0/256/512/1,024 rationale tokens\.Table 7:Option\-order robust accuracy \(ReAcc\)\.Table 8:Semantic consistency across option orders \(ReCon\)\.
## Appendix DCanonical RMS\-RSP versus Random

Table 9:Paired differences from Random\. Values and intervals are percentage points\.
## Appendix EReproducibility and artifact scope

The experiment artifacts retain selected UID order, token cost, acquisition score, predictions, per\-run metrics, and command arguments\. The present anonymous draft omits identifying repository links\. Code and processed split manifests should be released with a non\-identifying archive at submission time\. Dataset licenses and model terms remain those of the original resources\. No new patient data or clinical records were collected\.