Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation

# Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation
Source: [https://arxiv.org/html/2605.21949](https://arxiv.org/html/2605.21949)
Shao Kan Jinglue Technology Development \(Nanjing\) Co\., Ltd\. Room 1215\-13, 12th Floor, Building A2, Huizhi Science and Technology Park, No\. 8 Hengtai Road, Nanjing Economic and Technological Development Zone, Nanjing, China shaokan1991@gmail\.com [https://orcid\.org/0009\-0003\-4872\-6193](https://orcid.org/0009-0003-4872-6193)

###### Abstract

Medical RAG systems in high\-risk QA settings are often evaluated through a single answer\-or\-abstain decision, but mixed evidence may support one claim, require conditions for another, and contradict a third\. We studyclaim\-selective certification: each response is decomposed into verifiable claims, scored against retrieved evidence, and mapped by an intent\-aware selector to \{full, partial, conflict, abstain\}\. On the primary weak\-label certificate protocol, whose real\-source\-only dev/test rows cover the naturally occurring non\-abstain actions, the full system records UCCR=0\.0000=0\.0000, PAU=1\.0000=1\.0000, PAU Precision=0\.9901=0\.9901, and action accuracy=0\.9204=0\.9204on dev \(n=314n=314\), and UCCR=0\.0000=0\.0000, PAU=0\.9967=0\.9967, PAU Precision=0\.9739=0\.9739, and action accuracy=0\.8997=0\.8997on test \(n=319n=319\)\. UCCR measures unsupported\-claim risk within the certificate definition, and a source\-missing counterfactual slice evaluatesabstainunder empty evidence\. Shortcut controls quantify the action\-label prior explained by source and intent metadata, while source/evidence\-novel slices characterize transfer boundaries\. The resulting interface separates action\-label prediction from evidence\-linked claim selection under mixed evidence\.

## 1Introduction

Retrieval\-augmented generation \(RAG\) grounds language models in external knowledge\[[17](https://arxiv.org/html/2605.21949#bib.bib1),[7](https://arxiv.org/html/2605.21949#bib.bib2)\], but high\-risk medical QA has an asymmetric answer contract: unsupported safety, dosing, or contraindication claims can be harmful, while blanket abstention can suppress usable evidence\. The same evidence set may support a dosing constraint, leave monitoring uncertain, and contradict a contraindication\. We studyclaim\-selective certification: decompose a question into verifiable claim units, score each claim against evidence, and choose whether to state it directly, state it with conditions, contest it, or withhold it\.

The implementation has three stages: template\-based claim decomposition, cue\-based relation scoring for support, conflict, and limitation signals, and anintent\-aware risk\-calibrated selector\. The selector receives aquestion\_intent: pregnancy, lactation, monitoring, and special\-population questions often require condition\-limited wording, whereas contraindication and interaction questions more often require certification or conflict handling\.

We evaluate this interface on 2,223 medical QA samples, including 2,103 examples \(94\.6%\) from publicly downloaded real sources and 120 synthetic examples \(5\.4%\)\. Primary results use a real\-source\-only split, with synthetic, source\-missing, and source/evidence\-novel slices characterizing behavior under synthetic examples, empty evidence, and source/evidence novelty\. We compare threshold\-only, binary\-form, NLI\-based, learned\-relation, and full claim\-selective systems under the same weak\-label certificate protocol\. The study contributes a claim\-level certification/action formulation, an intent\-aware fixed policy layer, and a certificate\-producing evaluation framework that reports UCCR, PAU, PAU Precision, action accuracy, risk–coverage, source\-overlap, shortcut, and source/evidence\-novel analyses for high\-risk medical RAG\.

## 2Related Work

Retrieval\-augmented generation grounds language models by retrieving external documents and conditioning generation on them\[[17](https://arxiv.org/html/2605.21949#bib.bib1),[10](https://arxiv.org/html/2605.21949#bib.bib3),[7](https://arxiv.org/html/2605.21949#bib.bib2)\]\. Much of the recent work improves evidence acquisition through dense retrieval\[[14](https://arxiv.org/html/2605.21949#bib.bib4)\], iterative retrieval and self\-reflection\[[2](https://arxiv.org/html/2605.21949#bib.bib5)\], query rewriting for RAG pipelines\[[18](https://arxiv.org/html/2605.21949#bib.bib6)\], or decomposition strategies for compositional questions\[[24](https://arxiv.org/html/2605.21949#bib.bib7)\]\. RAG methods determine which evidence is available to the model\. Our setting starts from the next decision point: once evidence is available, which claims should appear in the answer?

Selective prediction and abstention reduce risk by withholding low\-confidence predictions\[[8](https://arxiv.org/html/2605.21949#bib.bib8),[13](https://arxiv.org/html/2605.21949#bib.bib9)\]\. Related QA and dialogue work studies time\-sensitive answerability\[[3](https://arxiv.org/html/2605.21949#bib.bib10)\], calibrated belief\-state distributions\[[32](https://arxiv.org/html/2605.21949#bib.bib11)\], and uncertainty over meanings rather than surface strings\[[16](https://arxiv.org/html/2605.21949#bib.bib12)\]\. These approaches are effective when an instance is globally answerable, unanswerable, or uncertain\. They are less expressive when a medical question contains several claims with different evidence states\. We keep the risk\-control objective but move the decision unit from the whole response to individual claims and action types\.

Faithfulness and fact\-checking methods verify whether generated or proposed claims are supported by sources\[[19](https://arxiv.org/html/2605.21949#bib.bib13),[5](https://arxiv.org/html/2605.21949#bib.bib14),[25](https://arxiv.org/html/2605.21949#bib.bib15),[31](https://arxiv.org/html/2605.21949#bib.bib17),[27](https://arxiv.org/html/2605.21949#bib.bib18)\], while citation\-grounded generation work studies how models present attributed evidence in the output itself\[[6](https://arxiv.org/html/2605.21949#bib.bib16)\]\. More directly, long\-form factuality evaluators decompose responses into atomic facts and check evidence support\[[21](https://arxiv.org/html/2605.21949#bib.bib19),[4](https://arxiv.org/html/2605.21949#bib.bib20)\], while efficient grounded checkers such as MiniCheck train smaller models for document\-grounded fact verification\[[30](https://arxiv.org/html/2605.21949#bib.bib21)\]\. These works make factual support measurable at the claim or fact level\. Claim\-selective RAG uses the same kind of signal earlier in the pipeline, before the final response is formed, and maps each claim to an explicit action and certificate state rather than only scoring a completed answer\.

Diagnostic evaluation work shows that high aggregate accuracy can hide shortcut behavior, annotation artifacts, and distribution\-shift failures\. Prior studies expose such behavior with hypothesis\-only artifact analyses\[[9](https://arxiv.org/html/2605.21949#bib.bib22)\], controlled heuristic challenge sets\[[20](https://arxiv.org/html/2605.21949#bib.bib23)\], behavioral test suites\[[26](https://arxiv.org/html/2605.21949#bib.bib24)\], data maps\[[29](https://arxiv.org/html/2605.21949#bib.bib25)\], and in\-the\-wild distribution\-shift benchmarks\[[15](https://arxiv.org/html/2605.21949#bib.bib26)\]\. We bring the same perspective to claim\-selective RAG: shortcut controls, source/evidence\-novel slices, and source\-missing abstention evaluations separate certificate\-producing behavior from label\-space priors and transfer failures\.

Medical QA has been studied in retrieval, benchmark, and large\-model settings\[[1](https://arxiv.org/html/2605.21949#bib.bib27),[12](https://arxiv.org/html/2605.21949#bib.bib28),[11](https://arxiv.org/html/2605.21949#bib.bib29),[23](https://arxiv.org/html/2605.21949#bib.bib30),[28](https://arxiv.org/html/2605.21949#bib.bib31),[22](https://arxiv.org/html/2605.21949#bib.bib32)\]\. Many datasets evaluate multiple\-choice, yes/no, or complete\-answer behavior\. Practical medical questions, however, often need partial support, contraindication handling, or condition\-specific wording\. The present work targets that interface gap by evaluating claim\-level selection, explicit action calibration, and risk–utility metrics such as UCCR and PAU\.

## 3Problem Formulation

Letqqdenote a medical question and letℰ=\{e1,…,ek\}\\mathcal\{E\}=\\\{e\_\{1\},\\ldots,e\_\{k\}\\\}be the retrieved evidence available to the system\. A conventional RAG policy emits one responser=LM​\(q,ℰ\)r=\\mathrm\{LM\}\(q,\\mathcal\{E\}\)and is then evaluated as a whole\. This is too coarse for mixed evidence: the same evidence packet can support one claim, only conditionally support a second, contradict a third, and leave a fourth unsupported\. We instead treat the response as a decision over claims\.

The interface first produces a small set of verifiable claim skeletons𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}and metadatamim\_\{i\}for each claim, including question intent and claim type\. Each claim–evidence pair receives a relation vector

𝐬​\(ci,ej\)=\(ssup,sconf,slim\)∈\[0,1\]3,\\mathbf\{s\}\(c\_\{i\},e\_\{j\}\)=\\big\(s\_\{\\mathrm\{sup\}\},s\_\{\\mathrm\{conf\}\},s\_\{\\mathrm\{lim\}\}\\big\)\\in\[0,1\]^\{3\},\(1\)where the coordinates summarize support, conflict, and limitation/conditionality\. Scores are pooled across evidence with max aggregation:

Sℓ​\(ci\)=maxej∈ℰ⁡sℓ​\(ci,ej\),ℓ∈\{sup,conf,lim\}\.S\_\{\\ell\}\(c\_\{i\}\)=\\max\_\{e\_\{j\}\\in\\mathcal\{E\}\}s\_\{\\ell\}\(c\_\{i\},e\_\{j\}\),\\qquad\\ell\\in\\\{\\mathrm\{sup\},\\mathrm\{conf\},\\mathrm\{lim\}\\\}\.\(2\)The selector maps\(Ssup,Sconf,Slim,mi\)\(S\_\{\\mathrm\{sup\}\},S\_\{\\mathrm\{conf\}\},S\_\{\\mathrm\{lim\}\},m\_\{i\}\)to a claim status

zi∈\{certified,condition​\-​limited,conflicting,omitted\}\.z\_\{i\}\\in\\\{\\mathrm\{certified\},\\mathrm\{condition\\mbox\{\-\}limited\},\\mathrm\{conflicting\},\\mathrm\{omitted\}\\\}\.\(3\)Certified and condition\-limited claims may be expressed with evidence links; conflicting claims trigger conflict behavior; omitted claims are withheld\.

The final action is a coarse summary of the selected claim set:

a∈\{full,partial,conflict,abstain\}\.a\\in\\\{\\texttt\{full\},\\texttt\{partial\},\\texttt\{conflict\},\\texttt\{abstain\}\\\}\.\(4\)Intuitively,fullmeans the requested answer can be stated directly,partialmeans usable information exists but requires caveats or omissions,conflictmeans material evidence disagreement or risk should be surfaced, andabstainmeans no claim should be expressed\. The action is therefore not a free\-form answer label; it is a policy decision induced by claim\-level statuses and evidence certificates\.

Evaluation follows the same claim interface\. Let𝒫\\mathcal\{P\}be the expressed predicted claims and𝒢use\\mathcal\{G\}\_\{\\mathrm\{use\}\}the construction\-derived gold\-usable claims under the weak\-label protocol\. Unsupported Critical Claim Rate \(UCCR\) measures certificate\-level risk among expressed critical claims:

UCCR=\|\{c∈𝒫:c​is critical and​c∉𝒢use\}\|\|\{c∈𝒫:c​is critical\}\|,\\mathrm\{UCCR\}=\\frac\{\|\\\{c\\in\\mathcal\{P\}:c\\text\{ is critical and \}c\\notin\\mathcal\{G\}\_\{\\mathrm\{use\}\}\\\}\|\}\{\|\\\{c\\in\\mathcal\{P\}:c\\text\{ is critical\}\\\}\|\},\(5\)defined as zero when no critical claim is expressed\. The target is zero unsupported expressed critical claims\. Partial Answer Utility \(PAU\) measures retained usable information:

PAU=\|𝒫∩𝒢use\|\|𝒢use\|\.\\mathrm\{PAU\}=\\frac\{\|\\mathcal\{P\}\\cap\\mathcal\{G\}\_\{\\mathrm\{use\}\}\|\}\{\|\\mathcal\{G\}\_\{\\mathrm\{use\}\}\|\}\.\(6\)We also report PAU Precision and F1 to distinguish retaining usable claims from over\-expressing extra claims, and Action Accuracy to evaluate the four\-way action interface\. These are protocol metrics over weak labels, not expert clinical judgments\.

This formulation makes the central tradeoff explicit\. If the evidence supports𝒞sup⊂𝒞\\mathcal\{C\}\_\{\\mathrm\{sup\}\}\\subset\\mathcal\{C\}and does not support the remaining claims, a document\-level answer policy largely chooses between expressing everything, which can raise UCCR, or abstaining, which lowers PAU\. A claim\-selective policy can instead express𝒞sup\\mathcal\{C\}\_\{\\mathrm\{sup\}\}, omit unsupported claims, and returnpartialwhen appropriate\. The experiments ask whether this interface can meet the certificate target while preserving utility and whether the resulting actions remain meaningful beyond shortcut priors\.

## 4Method

The system follows a three\-stage pipeline: claim decomposition, relation scoring, and intent\-aware risk\-calibrated selection\. We study the post\-retrieval decision layer: given a fixed evidence set, the system decides which claims should be certified, condition\-limited, contested, or omitted\. Figure[1](https://arxiv.org/html/2605.21949#S4.F1)summarizes the pipeline\. The system exposes a small set of candidate claims, scores each claim against retrieved evidence, and applies an explicit action policy\. This keeps the main decisions inspectable under weak supervision and makes it possible to evaluate action calibration separately from open\-ended generation\.

![Refer to caption](https://arxiv.org/html/2605.21949v1/x1.png)Figure 1:System architecture\. The pipeline combines template\-based claim decomposition with explicit question intent, cue\-based relation scoring, and an intent\-aware risk\-calibrated selector\.### 4\.1Design Rationale

The design centers the post\-retrieval decision boundary: the claim interface fixes the decision unit, the relation scorer exposes support, conflict, and limitation signals, and the selector maps those signals to an action\. This structure lets us study action calibration under a common weak\-label protocol while keeping claim selection, certification, and audit tied to explicit intermediate variables\.

### 4\.2Claim Decomposition

Given a queryqq, the decomposition layer emits verifiable claim skeletons𝒞=\{c1,…,cn\}\\mathcal\{C\}=\\\{c\_\{1\},\\ldots,c\_\{n\}\\\}\. The templates produce a small set of high\-level critical claims rather than unrestricted fine\-grained extractions\. Each query also receives aquestion\_intent, covering indication, dosage, contraindication, interaction, pregnancy/lactation, monitoring, missed\-dose, and special\-population questions\. The resulting skeletons provide a stable interface for studying selective inclusion, condition\-limited wording, conflict handling, and omission under a shared weak\-label protocol\.

Table 1:Claim\-template and selector\-policy interface\. The rows summarize the operating families used by the selector\.
### 4\.3Relation Scoring

For each claim–evidence pair\(ci,ej\)\(c\_\{i\},e\_\{j\}\), the relation module estimates support, conflict, and limitation scores\. The scorer combines lexical overlap with a cue lexicon to identify whether evidence supports a claim, contradicts it, or supports it only with conditions\. The cue lexicon contains English phrases from labels and abstracts\. Scores are aggregated across retrieved evidence with max\-style pooling so that one strong passage can activate the relevant relation signal:

Ssupport​\(ci\)\\displaystyle S\_\{\\text\{support\}\}\(c\_\{i\}\)=maxej∈ℰ⁡ssupport​\(ci,ej\)\\displaystyle=\\max\_\{e\_\{j\}\\in\\mathcal\{E\}\}s\_\{\\text\{support\}\}\(c\_\{i\},e\_\{j\}\)\(7\)Sconflict​\(ci\)\\displaystyle S\_\{\\text\{conflict\}\}\(c\_\{i\}\)=maxej∈ℰ⁡sconflict​\(ci,ej\)\\displaystyle=\\max\_\{e\_\{j\}\\in\\mathcal\{E\}\}s\_\{\\text\{conflict\}\}\(c\_\{i\},e\_\{j\}\)\(8\)Slimitation​\(ci\)\\displaystyle S\_\{\\text\{limitation\}\}\(c\_\{i\}\)=maxej∈ℰ⁡slimitation​\(ci,ej\)\.\\displaystyle=\\max\_\{e\_\{j\}\\in\\mathcal\{E\}\}s\_\{\\text\{limitation\}\}\(c\_\{i\},e\_\{j\}\)\.\(9\)The scorer preserves the originatingquestion\_intent, so the selector can condition on both evidence relations and question type\.

### 4\.4Intent\-Aware Risk\-Calibrated Selection

The selector maps each scored claim to one of four statuses:certified, when evidence is strong enough for direct inclusion;condition\-limited, when the claim should be used only with caveats or partial wording;conflicting, when the evidence is contradictory or too risky for direct certification; andomitted, when support is too weak\. The final response action is chosen from \{full, partial, conflict, abstain\} based on the selected claim set\.

The mapping is intent\-aware\. A single global threshold is not adequate across medical question types: monitoring, pregnancy, lactation, and special\-population questions often require condition\-limited answers even when some support is present, whereas contraindication and interaction questions more often require certification or conflict handling\. The selector groups intents into coarse policy families—full\-certify oriented, partial\-oriented, conflict\-oriented, mixed, and dosage\-specific cases—and applies different mappings from\(Ssupport,Sconflict,Slimitation\)\(S\_\{\\text\{support\}\},S\_\{\\text\{conflict\}\},S\_\{\\text\{limitation\}\}\)to claim status\. The ablation compares this policy with a threshold\-only selector that uses generic score thresholds without intent\-specific rules\.

The implementation also contains two scoped source\-family priors\. First, explicit dosage\-instruction cues from openFDA label evidence can rescue dosage support in the relation scorer\. Second, PubMed Literature review\-level evidence can downgrade selected full answers to partial for specified claim families\. These priors are fixed implementation choices, not learned calibration parameters; the experiments therefore report source\-conditioned majority controls and source/evidence\-novel slices to expose how much action behavior is recoverable from source and intent metadata\.

## 5Experiments

We evaluate certificate risk, retained utility, intent\-aware action calibration, expressiveness beyond answer/abstain baselines, and behavior under source/evidence novelty\.

### 5\.1Data Provenance and Main Evaluation Split

We use a 2,223\-item medical QA collection: 2,103 public\-source examples \(94\.6%\) and 120 synthetic examples \(5\.4%\)\. Main results use a real\-source\-only split with trainn=1,470n=1\{,\}470, dev evaln=314n=314, and test evaln=319n=319; the synthetic subset contributes a separaten=20n=20stress slice\. We also derive a source/evidence\-novel holdout from primary dev/test rows whose normalizedsource\_urlandevidence\_textare both absent from train \(devn=82n=82, testn=78n=78\)\. This slice evaluates source/evidence novelty, while the primary split remains the main protocol\. A source\-missing counterfactual slice preserves questions and claim skeletons, removes evidence, and sets gold action toabstain, targeting deterministic retrieval\-failure abstention under empty evidence\. Tables[4](https://arxiv.org/html/2605.21949#A3.T4),[5](https://arxiv.org/html/2605.21949#A3.T5), and[15](https://arxiv.org/html/2605.21949#A6.T15)summarize counts\.

PubMedQA maps abstract\-levelyes/no/mayberesearch conclusions intofull/conflict/partial, so it serves as an abstract\-style interface\-transfer slice rather than a drug\-label QA proxy\.

### 5\.2Metrics

We report four metrics\.UCCRmeasures the fraction of expressed critical claims that are unsupported under the certificate definition\.PAUis recall\-style utility over retained gold\-usable claims and is paired with PAU Precision, F1, and risk–coverage curves to expose over\-expression\.F1uses sample\-scoped claim matching because claim identifiers are reused\.Action Accuracyevaluates \{full, partial, conflict, abstain\} actions and is interpreted together with certificate production and perturbation controls\.

### 5\.3Weak\-Label Certificate Protocol

Weak labels define a shared certificate protocol: each prediction records claim status, evidence identifiers, and relation scores\. UCCR and PAU are protocol measurements over expressed unsupported critical claims and retained usable claims\. Shortcut controls and risk–coverage curves are reported alongside them so that action behavior, certificate production, and label\-space priors are analyzed together\. To calibrate this protocol against external review, we separately re\-audited a 100\-item human\-validation subset using official\-source\-first evidence review with preserved screenshots, HTML snapshots, and text traces\. The final audit labels contain 49full\_support, 49conditional\_support, and 2conflictcases, agreeing with the weak labels on 73/100 items \(0\.7300; Cohen’sκ=0\.5027\\kappa=0\.5027\)\. This subset is not used for threshold tuning or benchmark replacement; it is reported only to bound how closely the weak\-label certificate protocol tracks pharmacist adjudication\.

### 5\.4Evaluation Protocol and Label–Policy Separation

All numbers are computed against construction\-derived weak labels\. We report dev and test results on the primary real\-source\-only split; appendix material provides split details, commands, intervals, and risk–coverage analysis\. The threshold\-only baseline is tuned ondev\_evalby a small grid search oversupport,conflict, andcondition\_limited; among UCCR=0=0candidates, we maximize PAU, then Action Accuracy, then F1\. The selectedsupport=0\.35,conflict=0\.55, andcondition\_limited=0\.30 thresholds transfer unchanged to test\. The full selector is a fixed policy specification with intent\-conditioned branches and global fallback thresholds, analyzed with shortcut controls, threshold perturbations, and a policy\-constant audit\. The speech\-act\-guided proxy tunes one globalanswer\_support=0\.34 gate on dev and transfers it unchanged to test\.

At inference, the full selector predicts from the question, claim skeleton and intent, retrieved evidence, relation scores, and documented source/context fields used by scoped source\-family priors\. Because source type, source\-level claim type, and intent can carry action priors, we report metadata\-only majority controls fit on train; Table[9](https://arxiv.org/html/2605.21949#A4.T9)lists label\-policy information access\.

The analysis covers retrieval\-only, threshold\-only, shortcut, binary\-form, NLI, learned\-relation, and full claim\-selective rows\. Binary\-form baselines collapse the final action space to answer/abstain; NLI and learned\-relation rows keep the native \{full, partial, conflict, abstain\} interface, with the learned row swapping only the relation module\.

## 6Results

### 6\.1Main Ablation Results

We first report primary real\-source\-only results, then use controls and transfer slices to separate certificate production from action\-label priors\. Table[2](https://arxiv.org/html/2605.21949#S6.T2)isolates the transition from evidence access to relation scoring and then to action\-level policy\.

Table 2:Main ablation results on the primary real\-source\-only split\. UCCR is defined by the weak\-label certificate protocol \(Section[5\.3](https://arxiv.org/html/2605.21949#S5.SS3)\), and PAU Prec is the fraction of expressed claims that are gold\-usable\.Retrieval\-only and relation\-only rows express available claims without action selection, producing nonzero UCCR \(0\.0860/0\.0878 and 0\.1943/0\.2069 on dev/test\) and low action accuracy\. The dev\-tuned threshold\-only selector restores UCCR to zero and reaches PAU 0\.9933/0\.9732 and F1 0\.9754/0\.9620, but action accuracy remains 0\.5223/0\.5517\. The full selector keeps UCCR=0\.0000=0\.0000while improving PAU, PAU Precision, F1, and action accuracy to 0\.9204/0\.8997\. Bootstrap intervals keep the full selector’s F1 and Action Accuracy gains over threshold\-only positive on both splits; risk–coverage analysis gives the same within\-system pattern\. The separate 100\-item human re\-audit shows that the weak\-label protocol is informative but not expert\-equivalent: weak and audited labels agree on 73/100 items withκ=0\.5027\\kappa=0\.5027, and the disagreement mass is concentrated infull/conditionalboundary cases rather than support\-versus\-conflict reversals\. Table[2](https://arxiv.org/html/2605.21949#S6.T2)should therefore be read as a protocol\-level certificate and action comparison, not as a substitute for expert clinical adjudication\.

![Refer to caption](https://arxiv.org/html/2605.21949v1/x2.png)Figure 2:Shortcut and perturbation controls on the primary split\. Metadata\-only majority rows are action\-only controls fit from training\-set weak labels, and certificate metrics apply only to evidence\-linked claim outputs\.Shortcut controls reveal a strong action\-label prior\. Intent\-majority reaches 0\.9013/0\.8934 action accuracy without evidence; source\+intent and source\+claim\-type action\-only rows reach 0\.9299/0\.9310 and 0\.9331/0\.9279, exceeding the full selector on action accuracy alone\. These rows do not select claims, assign evidence, expose relation scores, or produce certificates, so UCCR and PAU are not applicable\. Perturbations confirm that the full selector still uses policy inputs: removing intent gives 0\.5223/0\.5517, shuffling intent gives 0\.4936/0\.4671, and shuffling evidence gives 0\.8344/0\.8088\.

### 6\.2External\-Form Baselines

The ablation table tests the within\-system contribution chain\. We add response\-format baselines: direct answering, binary answer/abstain behavior, a speech\-act\-guided answer/abstain proxy, a standard NLI\-based semantic claim\-selective baseline, and a stronger learned\-relation claim\-selective variant\. Full numeric rows are reported in Table[11](https://arxiv.org/html/2605.21949#A4.T11)\.

![Refer to caption](https://arxiv.org/html/2605.21949v1/x3.png)Figure 3:Baseline operating map on the primary split\. Binary\-form baselines reduce unsupported\-claim risk by collapsing the action space, whereas claim\-selective baselines retain the native action interface\.Direct and citation\-only answers incur large UCCR penalties\. Binary answer/abstain and verifier\-only rows drive UCCR to zero but sacrifice PAU and action accuracy by collapsingpartialandconflict\. The speech\-act\-guided proxy is stronger, reaching PAU 0\.9732/0\.9599 and F1 0\.9652/0\.9551 with UCCR zero, but action accuracy remains 0\.4777/0\.5611\. The NLI\-based claim\-selective baseline improves action accuracy to 0\.7484/0\.6614\. The strongest external baseline swaps the relation module while retaining the same intent\-aware selector and native claim\-selective action space; it reaches 0\.8758/0\.8652 action accuracy\. A controlled relation\-module comparison shows action\-accuracy changes of−0\.0446\-0\.0446on dev and−0\.0345\-0\.0345on test, placing the main contribution at the claim\-selective action policy\.

### 6\.3Abstract\-Style Transfer Behavior and Failure Modes

Source slices localize the remaining errors \(Table[12](https://arxiv.org/html/2605.21949#A5.T12)\)\. OpenFDA and FDA FAERS are stable; PubMed Literature is mostly handled by the partial\-answer policy\. PubMedQA is the main abstract\-style transfer slice: itsyes/no/mayberesearch judgments do not map cleanly ontofull/conflict/partial, and test action accuracy is 0\.5161\.

![Refer to caption](https://arxiv.org/html/2605.21949v1/x4.png)Figure 4:Source/evidence\-novel boundary\. The full selector keeps the certificate target as overlap constraints tighten, but action accuracy drops on the strict source/evidence\-novel slice\.The primary split has no exact question overlap but substantial source reuse: train\-to\-dev/test source\-URL overlap is 0\.7389/0\.7555 and evidence\-text overlap is 0\.5541/0\.5298\. We therefore materialize the strictest still\-usable source/evidence\-novel holdout\. The full selector keeps UCCR=0\.0000=0\.0000and PAU=1\.0000=1\.0000, but action accuracy drops from 0\.9204/0\.8997 to 0\.7805 on dev \(n=82n=82\) and 0\.7692 on test \(n=78n=78\)\. It remains above unchanged threshold\-only \(0\.5854/0\.5256\) and learned\-relation swap \(0\.6341/0\.5769\), while source\-conditioned majority rows remain competitive or stronger on action accuracy alone\. Excluding PubMedQA, source/evidence\-novel action accuracy is 0\.9245 \(49/5349/53\) on dev and 0\.9362 \(44/4744/47\) on test; the aggregate drop is concentrated in PubMedQA \(0\.5172/0\.5161\)\. A source\-missing counterfactual slice evaluates theabstainaction under empty evidence\. Both threshold\-only and full selectors abstain for all dev/test examples \(n=314/319n=314/319\), giving action accuracy 1\.0000; retrieval\-only has action accuracy 0\.0000 and UCCR=1\.0000=1\.0000\. The matching threshold\-only result places this slice as action\-space coverage evidence for the fourth action\. Claim\-type and error slices in the appendix show the same pattern: remaining errors are mostlyfull/partial/conflictboundary decisions under ambiguous evidence\.

## 7Discussion

The results support the claim\-selective action interface\. Under a shared certificate protocol, it calibrates primaryfull,partial, andconflictbehavior while the source\-missing counterfactual slice exercisesabstainunder empty evidence\. A dev\-tuned threshold\-only selector reaches the same UCCR target and abstains under empty evidence, but remains much weaker on primary non\-abstain action accuracy\.

For high\-risk medical QA, the design implication is that answer selection and claim selection should be evaluated separately\. A system may have enough evidence to state a dosing constraint, insufficient evidence for monitoring advice, and contradictory evidence for a contraindication in the same retrieved packet\. A document\-level answer/abstain interface collapses those states, whereas claim\-selective certification records which claims are expressed, which evidence supports them, and which response action follows from the resulting certificate\.

Shortcut controls are part of the empirical result\. Intent\-majority reaches 0\.9013/0\.8934 action accuracy, and source\-conditioned action\-only rows can exceed the full selector\. The distinction is structural: metadata\-only controls predict an action label, while the proposed interface selects claims, assigns evidence, records relation scores, and produces an auditable certificate\. No\-intent, intent\-shuffled, and evidence\-shuffled rows show sensitivity to policy inputs\.

This distinction matters for interpreting the primary numbers\. Action accuracy alone would make the source\-conditioned majority rows a confound: they show that a large part of the weak\-label action space is recoverable from source and intent metadata\. The controls expose that structure and keep the empirical claim at the level of evidence\-linked certification\. The proposed interface is valuable because it attaches action decisions to claim\-level evidence states and makes the remaining dependence on metadata measurable\.

The external baselines locate the contribution\. Direct answering incurs unsupported\-claim risk; binary abstention reduces that risk by sacrificing utility; NLI and learned\-relation claim selection are the strongest model\-side comparisons\. PubMedQA and source/evidence\-novel cases show that remaining errors are mainlyfull/partial/conflictdecisions under abstract\-style evidence, rather than unsupported generation alone\. The resulting picture is a measured same\-source\-family protocol with explicit source/evidence\-novel and abstract\-style transfer boundaries\.

The source/evidence\-novel result separates certificate behavior from action generalization\. The full selector keeps UCCR at zero on the novelty slice, which means the expressed\-claim certificate target remains satisfied under the weak\-label metric\. At the same time, action accuracy drops to 0\.7805/0\.7692, and the drop is concentrated in PubMedQA\. The method preserves the certificate constraint on this derived boundary slice, while the intent/source policy still struggles when abstract\-levelyes/no/maybejudgments must be mapped to the drug\-QA action interface\. The abstention experiment completes action coverage: the source\-missing counterfactual evaluates policy behavior after complete evidence removal, selector\-based systems deterministically enter the abstain branch, and retrieval\-only behavior cannot\. The matching threshold\-only result identifies the slice as coverage evidence for the fourth action\. Overall, the empirical pattern supports a methodological claim\. Claim\-selective certification gives a structured way to expose mixed\-evidence decisions, compare answer/abstain systems with native multi\-action systems, and audit where source\-family priors enter the policy\. The results characterize a same\-source\-family weak\-label protocol with explicit failure\-boundary slices; deployment\-oriented validation would require independently sampled source\-disjoint data, expert adjudication, and naturally occurring abstention cases\.

## 8Limitations

Several limitations follow from the evaluation design\. Labels are construction\-derived weak labels, and UCCR is an expressed\-claim certificate metric within this protocol; the 100\-item human re\-audit reaches 73/100 agreement withκ=0\.5027\\kappa=0\.5027, so the paper provides weak\-label protocol evidence rather than clinical\-safety validation\. Shortcut controls reveal source\- and intent\-conditioned action priors, and majority rows provide action\-only comparisons without certificate production\. The primary split has no exact question duplicates but does reuse sources and evidence: train\-to\-dev/test source\-URL overlap is 0\.7389/0\.7555 and evidence\-text overlap is 0\.5541/0\.5298\. The derived source/evidence\-novel holdout drops action accuracy to 0\.7805/0\.7692, mainly from PubMedQA, and the source\-missing counterfactual tests only deterministic abstention under complete evidence removal\. Finally, the selector is a fixed policy with intent\-conditioned constants and scoped source\-family priors, and template decomposition plus cue\-based relation scoring still leave fine\-grained multi\-claim decomposition, implicit contraindications, semantic disagreement, and PubMedQA\-style abstract judgments as open extensions\.

## 9Conclusion

We study claim\-selective certification as an auditable alternative to answer\-or\-abstain behavior for mixed\-evidence medical QA\. On the real\-source\-only protocol, the full system records UCCR=0\.0000=0\.0000on dev/test, PAU=1\.0000/0\.9967=1\.0000/0\.9967, PAU Precision=0\.9901/0\.9739=0\.9901/0\.9739, and action accuracy=0\.9204/0\.8997=0\.9204/0\.8997; shortcut controls and novelty slices define both the value and the boundary of this weak\-label protocol\. Rather than claiming source\-disjoint clinical generalization, the paper isolates a reproducible certification interface, the shortcut structure of its weak\-label action space, and the transfer boundary exposed by abstract\-style evidence\.

## References

- \[1\]\(2019\)Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering\.InProceedings of the 18th BioNLP Workshop and Shared Task,pp\. 370–379\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[2\]A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. Hajishirzi\(2023\)Self\-rag: learning to retrieve, generate, and critique through self\-reflection\.arXiv preprint arXiv:2310\.11511\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[3\]W\. Chen, X\. Wang, and W\. Y\. Wang\(2021\)A dataset for answering time\-sensitive questions\.arXiv preprint arXiv:2108\.06314\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p2.1)\.
- \[4\]C\. Du, N\. Hu, D\. Huang, J\. Huang, Q\. Le, R\. Liu, Y\. Lu, D\. Peng, X\. Song, D\. Tran, J\. Wei, and C\. Yang\(2024\)Long\-form factuality in large language models\.InAdvances in Neural Information Processing Systems 37,pp\. 80756–80827\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[5\]N\. Dziri, E\. Kamalloo, S\. Milton, O\. Zaiane, M\. Yu, E\. M\. Ponti, and S\. Reddy\(2022\)FaithDial: a faithful benchmark for information\-seeking dialogue\.Transactions of the Association for Computational Linguistics10,pp\. 1473–1490\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[6\]T\. Gao, H\. Yen, J\. Yu, and D\. Chen\(2023\)Enabling large language models to generate text with citations\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 6465–6488\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.398)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[7\]Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, and H\. Wang\(2023\)Retrieval\-augmented generation for large language models: a survey\.arXiv preprint arXiv:2312\.10997\.Cited by:[§1](https://arxiv.org/html/2605.21949#S1.p1.1),[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[8\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.InAdvances in neural information processing systems,pp\. 4878–4887\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p2.1)\.
- \[9\]S\. Gururangan, S\. Swayamdipta, O\. Levy, R\. Schwartz, S\. R\. Bowman, and N\. A\. Smith\(2018\)Annotation artifacts in natural language inference data\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 107–112\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p4.1)\.
- \[10\]G\. Izacard and E\. Grave\(2021\)Leveraging passage retrieval with generative models for open domain question answering\.InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume,pp\. 874–880\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.eacl-main.74)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[11\]D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits\(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[12\]Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. Lu\(2019\)PubMedQA: a dataset for biomedical research question answering\.arXiv preprint arXiv:1909\.06146\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[13\]A\. Kamath, R\. Jia, and P\. Liang\(2020\)Selective question answering under domain shift\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 5684–5696\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.503)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p2.1)\.
- \[14\]V\. Karpukhin, B\. Oğuz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih\(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 6769–6781\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[15\]P\. W\. Koh, S\. Sagawa, H\. Marklund, S\. M\. Xie, M\. Zhang, A\. Balsubramani, W\. Hu, M\. Yasunaga, R\. L\. Phillips, I\. Gao, T\. Lee, E\. David, I\. Stavness, W\. Guo, B\. Earnshaw, I\. Haque, S\. Beery, J\. Leskovec, A\. Kundaje, E\. Pierson, S\. Levine, C\. Finn, and P\. Liang\(2021\)WILDS: a benchmark of in\-the\-wild distribution shifts\.InProceedings of the 38th International Conference on Machine Learning,pp\. 5637–5664\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p4.1)\.
- \[16\]L\. Kuhn, Y\. Gal, and S\. Farquhar\(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p2.1)\.
- \[17\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in Neural Information Processing Systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2605.21949#S1.p1.1),[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[18\]X\. Ma, Y\. Gong, P\. He, H\. Zhao, and N\. Duan\(2023\)Query rewriting in retrieval\-augmented large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 13679–13690\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.322)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[19\]J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald\(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 1906–1919\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.173)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[20\]R\. T\. McCoy, E\. Pavlick, and T\. Linzen\(2019\)Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 3428–3448\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p4.1)\.
- \[21\]S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi\(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12076–12100\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[22\]H\. Nori, N\. King, S\. M\. McKinney, D\. Carignan, and E\. Horvitz\(2023\)Capabilities of gpt\-4 on medical challenge problems\.arXiv preprint arXiv:2303\.13375\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[23\]A\. Pal, L\. K\. Umapathi, and M\. Sankarasubbu\(2022\)MedMCQA: a large\-scale multi\-subject multi\-choice dataset for medical domain question answering\.arXiv preprint arXiv:2203\.14371\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[24\]O\. Press, M\. Zhang, S\. Min, L\. Schmidt, N\. A\. Smith, and M\. Lewis\(2023\)Measuring and narrowing the compositionality gap in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5664–5687\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.378)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p1.1)\.
- \[25\]H\. Rashkin, V\. Nikolaev, M\. Lamm, L\. Aroyo, M\. Collins, D\. Das, S\. Petrov, G\. S\. Tomar, I\. Turc, and D\. Reitter\(2023\)Measuring attribution in natural language generation models\.Computational Linguistics,pp\. 1–64\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[26\]M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh\(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4902–4912\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p4.1)\.
- \[27\]T\. Schuster, A\. Fisch, and R\. Barzilay\(2021\)Get your vitamin c\! robust fact verification with contrastive evidence\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 624–640\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.52)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[28\]K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06291-2)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p5.1)\.
- \[29\]S\. Swayamdipta, R\. Schwartz, N\. Lourie, Y\. Wang, H\. Hajishirzi, N\. A\. Smith, and Y\. Choi\(2020\)Dataset cartography: mapping and diagnosing datasets with training dynamics\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 9275–9293\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p4.1)\.
- \[30\]L\. Tang, P\. Laban, and G\. Durrett\(2024\)MiniCheck: efficient fact\-checking of llms on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 8818–8847\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[31\]J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal\(2018\)FEVER: a large\-scale dataset for fact extraction and verification\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 809–819\.Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p3.1)\.
- \[32\]C\. van Niekerk, M\. Heck, C\. Geishauser, H\. Lin, N\. Lubis, M\. Moresi, and M\. Gasic\(2020\)Knowing what you know: calibrating dialogue belief state distributions via ensembles\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 3096–3102\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.277)Cited by:[§2](https://arxiv.org/html/2605.21949#S2.p2.1)\.

## Appendix AAppendix Overview

This appendix reports the claim summary, additional diagnostics, uncertainty estimates, reproducibility commands, and asset information used to support the main paper\.

Algorithm 1Schematic claim\-selective action interface\. The procedure summarizes the fixed decision flow from retrieved evidence to the final action\.1:Input:question

qq, retrieved evidence

ℰ\\mathcal\{E\}
2:Emit claim skeletons

𝒞\\mathcal\{C\}and question intent\.

3:forclaim

ci∈𝒞c\_\{i\}\\in\\mathcal\{C\}do

4:Score support, conflict, and limitation against each

ej∈ℰe\_\{j\}\\in\\mathcal\{E\}\.

5:Aggregate scores into

\(Ssup,Sconf,Slim\)\(S\_\{\\mathrm\{sup\}\},S\_\{\\mathrm\{conf\}\},S\_\{\\mathrm\{lim\}\}\)\.

6:Map relation scores and intent to a status in \{certified, condition\-limited, conflicting, omitted\}\.

7:endfor

8:Build an evidence\-linked certificate for expressed claims\.

9:ifno claim is expressiblethen

10:returnabstain

11:elseifmaterial conflict is selectedthen

12:returnconflict

13:elseifall required critical claims are directly certifiedthen

14:returnfull

15:else

16:returnpartial

17:endif

## Appendix BClaim Summary

Table 3:Summary of the main claims and their evaluation level\.
## Appendix CPrimary Split and Protocol Tables

Table 4:Primary real\-source\-only split used for the main results\.Table 5:Gold action coverage and abstention scope\. The primary dev/test protocol evaluates the non\-abstain actions, and source\-missing rows exercise abstention under counterfactual retrieval failure\.
## Appendix DHuman Re\-audit of Weak Labels

To estimate how closely the weak\-label certificate protocol tracks pharmacist adjudication, we separately re\-audited a 100\-item human\-validation subset using official\-source\-first evidence review\. The audit proceeded through an initial pass, a second\-pass whole\-set adjudication, and an item\-by\-item third\-pass web re\-audit of all remaining disagreement cases, with screenshots, HTML snapshots, and text snapshots preserved underdata/annotation/third\_pass\_web\_reaudit/\. The final audited labels were not used to tune thresholds or replace the primary benchmark; they are reported only as a calibration layer for interpreting the weak\-label protocol\.

Table 6:Summary of the 100\-item human re\-audit\. Agreement is measured against the original weak labels\.Table 7:Disagreement structure in the 100\-item human re\-audit\. Rows count weak\-label→\\rightarrowaudited\-label transitions among the 27 disagreement items\.Most disagreements are boundary cases betweenfull\_supportandconditional\_support, not reversals between support and conflict\. The third\-pass item\-by\-item web re\-audit made no further changes to the final audited set\. We therefore use this subset to calibrate scope, not to relabel the main benchmark: it supports the claim that the protocol carries nontrivial medical signal while still falling short of expert\-adjudicated clinical gold data\.

The source\-missing counterfactual slice completes action\-interface coverage\. It preserves the original primary dev/test questions and claim skeletons, setsevidence\_pool=\[\]and clearsevidence\_text, converts all gold claims toomitted, and sets the gold action toabstain\. Table[8](https://arxiv.org/html/2605.21949#A4.T8)reports the corresponding action behavior\. Because every stress item has empty evidence, these rows characterize deterministic abstention under retrieval failure; semantic evidence insufficiency, natural abstention frequency, and expert\-labeled correctness require separate evaluation data\.

Table 8:Source\-missing abstention results\. Selector rows abstain under empty evidence, whereas retrieval/relation\-only rows express claims without evidence and fail the action check\.Table 9:Information available to weak\-label construction and model\-side policies\.Table 10:Shortcut controls on the primary split\. Majority rows are action\-only controls fit from training\-set weak labels, and certificate metrics apply only to rows with evidence\-linked claim outputs\.Table 11:Baseline package on the primary split\. Binary\-form baselines use the mappinganswer→\\rightarrowfullandabstain→\\rightarrowabstainfor action accuracy, whereas claim\-selective rows retain the native \{full, partial, conflict, abstain\} action space\.
## Appendix EExtended Diagnostics

This section expands the main source\- and claim\-type analyses for the full system\. The tables and figures below report diagnostic slices under the same weak\-label protocol as the main results\.

Table 12:Source\-level diagnostics for the full system on the primary split\.![Refer to caption](https://arxiv.org/html/2605.21949v1/x5.png)Figure 5:Extended source\-level diagnostics for the full system\. OpenFDA has the highest action accuracy, whereas PubMedQA has the lowest\.Table 13:Hard claim\-type diagnostics for the full system\. Slices withn<20n<20are reported descriptively\.![Refer to caption](https://arxiv.org/html/2605.21949v1/x6.png)Figure 6:Action confusion on the strongest and weakest sources\. PubMedQA errors are dominated byfull/partialboundary mistakes rather than unsupported generation\.
## Appendix FSource\-Overlap Analysis

The primary split removes exact question duplicates and retains source\-family overlap\. We measure exact overlap over question text, source URL, and evidence text\. Table[14](https://arxiv.org/html/2605.21949#A6.T14)reports the main analysis\. Train\-to\-dev/test question overlap is zero, while source\-URL overlap is substantial: 232/314 dev examples and 241/319 test examples share a source URL with training\. Evidence\-text overlap is also nontrivial at 174/314 and 169/319\. The source pattern is uneven: OpenFDA examples have 100% train source\-URL overlap in both dev and test, FDA FAERS has partial overlap, and PubMed Literature/PubMedQA have no exact train source\-URL overlap\.

Table 14:Source\-overlap analysis for the primary split\. Novelty\-slice metrics use the unchanged full selector, and the source/evidence\-novel holdout requires both source URL and evidence text to be absent from train\.DiagnosticSplit / keynOverlapRateAction AccExact overlaptrain→\\rightarrowdev question31400\.0000–Exact overlaptrain→\\rightarrowdev source URL3142320\.7389–Exact overlaptrain→\\rightarrowdev evidence text3141740\.5541–Exact overlaptrain→\\rightarrowtest question31900\.0000–Exact overlaptrain→\\rightarrowtest source URL3192410\.7555–Exact overlaptrain→\\rightarrowtest evidence text3191690\.5298–Full selectordev all314––0\.9204Full selectordev source/evidence\-novel82––0\.7805Full selectortest all319––0\.8997Full selectortest source/evidence\-novel78––0\.7692The source/evidence\-novel holdout is smaller and distribution\-shifted\. Its lower action accuracy characterizes the primary split as a controlled same\-source\-family evaluation with a separate source/evidence\-novel transfer slice\. On this boundary slice, the full selector records UCCR=0\.0000=0\.0000under the weak\-label certificate metric, with PAU/F1 of 1\.0000/0\.9873 on dev and 1\.0000/0\.9583 on test\. Tables[15](https://arxiv.org/html/2605.21949#A6.T15)and[16](https://arxiv.org/html/2605.21949#A6.T16)provide the reproducible slice definition and the corresponding comparison rows\.

Table 15:Source/evidence\-novel holdout construction\. Rows are selected from the primary dev/test evaluation files when both normalizedsource\_urlandevidence\_textare absent from train\.Table 16:Source/evidence\-novel holdout results\. Selector and baseline rows use fixed primary\-split settings with no threshold retuning, and majority rows are action\-only controls that do not produce certificates\.Table 17:Source/evidence\-novel full\-selector accuracy by source\. The non\-PubMedQA aggregate shows that most of the action drop is localized to the abstract\-style PubMedQA transfer slice\.On the source/evidence\-novel holdout, the full selector makes 18 action errors on dev and 18 on test\. The source\-specific pattern is concentrated: FDA FAERS remains at 1\.0000 action accuracy on both slices, PubMed Literature remains high \(0\.8919/0\.9211\), and PubMedQA remains low \(0\.5172/0\.5161\)\. Excluding PubMedQA, source/evidence\-novel action accuracy is 0\.9245/0\.9362\. This supports interpreting the holdout as an abstract\-style and source\-shift boundary rather than a broad failure of the certificate metric\.

## Appendix GStatistical Uncertainty

We computed nonparametric bootstrap intervals on the primary real\-source\-only dev and test splits\. Table[18](https://arxiv.org/html/2605.21949#A7.T18)reports point estimates and 95% intervals for threshold\-only selection and the full intent\-aware selector\.

The threshold\-only comparison uses thedev\-selectedoperating point described in Section[5](https://arxiv.org/html/2605.21949#S5)\. A grid search oversupport,conflict, andcondition\_limitedthresholds selects the candidate with UCCR=0=0and then maximizes PAU, action accuracy, and F1 in that order\. The selected setting issupport=0\.35,conflict=0\.55, andcondition\_limited=0\.30, transferred unchanged to test\. The speech\-act\-guided answer/abstain proxy is tuned separately on dev by searching a small set of globalanswer\_supportgates while keeping its profile\-specific retrieval logic fixed; this selectsanswer\_support=0\.34\.

Table 18:Bootstrap 95% intervals for the main selector comparison\. Intervals are estimated from 1,000 sample\-level bootstrap resamples\.The paired bootstrap deltas localize the gain\. On dev, the full selector improves PAU by 0\.0067 with a 95% interval of \[0\.0000, 0\.0168\], F1 by 0\.0196 \[0\.0097, 0\.0299\], and action accuracy by 0\.3981 \[0\.3408, 0\.4522\]\. On test, the corresponding deltas are 0\.0234 \[0\.0068, 0\.0435\] for PAU, 0\.0231 \[0\.0099, 0\.0381\] for F1, and 0\.3480 \[0\.2915, 0\.4075\] for action accuracy\.

For UCCR, bootstrap intervals degenerate at zero because no unsupported expressed critical claim is observed under the reported weak\-label certificate metric on the primary split\. Interpreting the dev/test expressed critical claims as Bernoulli trials gives 0 observed events out of 302 expressed critical claims on dev and 0 out of 306 on test\. The corresponding upper ends of the two\-sided 95% Wilson score intervals are 0\.0126 and 0\.0124\.

## Appendix HPAU Precision

PAU is a recall\-style utility metric: it measures how many gold usable claims are retained\. To make the over\-expression boundary explicit, we also compute a precision counterpart,\|gold​usable∩pred​usable\|/\|pred​usable\|\|\\mathrm\{gold\\ usable\}\\cap\\mathrm\{pred\\ usable\}\|/\|\\mathrm\{pred\\ usable\}\|\. This precision counterpart distinguishes retained useful claims from over\-expression\.

Table 19:PAU precision on the primary split\. Higher values indicate fewer extra usable predictions beyond the weak\-label usable set\.Retrieval\-only keeps PAU at 1\.0000 because it expresses all gold usable claims, but its PAU precision is lower because it also expresses claims outside the weak\-label usable set\. The full selector has the strongest PAU precision among these rows, indicating that its high PAU is not obtained by broad over\-expression\.

## Appendix ISelector Threshold Sensitivity

We run one\-at\-a\-time perturbations around the reported selector operating points\. For the tuned threshold\-only selector, we perturbsupport,conflict,condition\_limited, andlimitationby±0\.05\\pm 0\.05from the dev\-selected point\. For the full selector, we apply the same perturbation to its global fallback thresholds while leaving the intent\-conditioned policy branches fixed\.

Table 20:Selector threshold sensitivity\. All rows have UCCR=0\.0000=0\.0000under the reported weak\-label certificate metric, and the full selector is unchanged because the high\-impact decisions are governed by intent\-conditioned policy branches\.The sensitivity analysis matches the main result table\. Threshold\-only behavior can keep UCCR at zero, but its utility and action behavior are brittle to a modest increase in the condition\-limited gate\. The full selector is dominated by intent\-conditioned branches, so these global fallback\-threshold perturbations leave its dev/test metrics unchanged\. The result localizes the reported decisions to the intent\-conditioned policy branches rather than to the global fallback gates\.

## Appendix JSelector Policy Specification

The full selector is a fixed policy specification with intent\-conditioned branches and global fallback thresholds\. The repository includes a policy\-constant audit atscripts/audit\_selector\_policy\_constants\.pyforsrc/selection/selector\.py\. The audit covers 445 lines across the fallback, intent\-aware, and final classification functions and finds 141 numeric constants in those functions\. These constants define the implemented policy for the current relation\-score distribution; shortcut controls, source slices, and threshold perturbations characterize the sensitivity of that policy\.

We additionally run a branch\-family sensitivity audit atscripts/analyze\_selector\_branch\_sensitivity\.py\. This audit perturbs the support, conflict, or limitation score entering one intent branch family at a time by±0\.05\\pm 0\.05and±0\.10\\pm 0\.10to characterize robustness of the fixed policy\. The largest overall action\-accuracy drops are−0\.0382\-0\.0382on dev and−0\.0408\-0\.0408on test, both under a−0\.10\-0\.10support perturbation to the indication branch\. Slice\-level changes are larger in small or boundary\-heavy families, including interaction on dev and dosage\-adjustment and research\-question slices on test\. In the largest\-change rows, UCCR remains0\.00000\.0000under the certificate metric\. This audit localizes sensitivity to specific intent\-conditioned branches rather than to the global fallback gates\.

## Appendix KSelective\-Prediction View

We also report a risk–coverage view at the critical\-claim level\. Each critical claim is treated as a selectable item\. Coverage is selected critical claims divided by total gold critical claims, and risk is selected critical claims not marked supportable by the gold weak labels divided by selected critical claims\. This view is supplementary because it ignores conflict disclosure and certificate semantics\.

For the threshold\-only selector, we sweep the support threshold and tie the condition\-limited threshold tosupport \- 0\.10while keeping the conflict thresholds fixed\. We also plot the dev\-tuned threshold\-only operating point \(support=0\.35,conflict=0\.55,condition\_limited=0\.30\)\. The full selector, learned claim\-selective baseline, and NLI baseline are reported as fixed operating points\. On dev, the tuned threshold\-only point reaches coverage 0\.9873 and risk 0\.0419, whereas the full selector reaches coverage 0\.9618 and risk 0\.0099\. On test, the corresponding comparison is 0\.9592 / 0\.0490 for the tuned threshold\-only point versus 0\.9592 / 0\.0261 for the full selector\. The learned claim\-selective baseline remains close but is slightly lower\-coverage and higher\-risk than the full selector on both splits \(0\.9427/0\.9561 coverage and 0\.0135/0\.0295 risk\)\.

![Refer to caption](https://arxiv.org/html/2605.21949v1/x7.png)Figure 7:Selective\-prediction view on the primary split\. The threshold\-only selector traces a tunable risk–coverage family, and the full selector keeps similar coverage but lower risk than the dev\-tuned threshold\-only operating point on both splits\.
## Appendix LCompute Resources

Experiments were reproduced on a Windows workstation with an Intel Core Ultra 9 185H CPU \(16 physical cores / 22 logical processors\), approximately 33\.95 GB of physical memory, Intel Arc integrated graphics, and an NVIDIA GeForce RTX 4060 Laptop GPU\. The main runtime costs come from relation\-model loading and repeated baseline sweeps rather than large\-scale training\.

Representative wall\-clock times are:

- •dev ablation rerun: 110\.93 seconds
- •baseline package rerun on the primary real\-source\-only split: 321\.94 seconds

## Appendix MExisting Assets and Terms

The reported evaluation chain depends on the following external assets\.

- •openFDA / FAERS APIs\.The official openFDA terms state that, unless otherwise noted, content and data are generally unrestricted and made available under a CC0 1\.0 dedication, while also warning that some third\-party content may be separately marked and that API use is subject to service limits and terms\.111[https://open\.fda\.gov/terms](https://open.fda.gov/terms)
- •DailyMed\.DailyMed is provided by the U\.S\. National Library of Medicine as the official public source of FDA label information, but the site also notes that NLM does not review SPL content before publication and that the “in use” labeling may differ from the most recent FDA\-approved labeling\.222[https://dailymed\.nlm\.nih\.gov/dailymed/about\-dailymed\.cfm](https://dailymed.nlm.nih.gov/dailymed/about-dailymed.cfm)
- •PubMed / NLM literature content\.NLM’s own copyright guidance states that some NLM data are U\.S\. government works while abstracts and other contributed materials may still be protected by copyright, leaving downstream users responsible for respecting those restrictions when redistributing content\.333[https://www\.nlm\.nih\.gov/databases/download\.html](https://www.nlm.nih.gov/databases/download.html)
- •PubMedQA\.The official PubMedQA repository distributes the benchmark under the MIT license and specifies the dataset download and evaluation process\.444[https://github\.com/pubmedqa/pubmedqa](https://github.com/pubmedqa/pubmedqa)
- •

The evaluation distinguishes between public\-domain or permissive API access, NLM\-hosted content that may still contain copyrighted abstracts, and benchmark/model assets distributed under repository\-specific licenses\.

## Appendix NReproducibility Note

The experiments use a single real\-source\-only split family:data/splits/primary\_real\_source/, which containstrain\.jsonl,dev\_eval\.jsonl, andtest\_eval\.jsonl\. The main reported results use only thedev\_eval\.jsonlandtest\_eval\.jsonlfiles from this real\-source\-only split\.

#### Main ablation reruns\.

The main ablation table can be regenerated with:

```
python scripts/run_ablation_study.py \
  data/splits/primary_real_source/dev_eval.jsonl \
  outputs/regression_checks/<dev_run>

python scripts/run_ablation_study.py \
  data/splits/primary_real_source/test_eval.jsonl \
  outputs/regression_checks/<test_run>
```

#### Baseline package reruns\.

The external\-form baseline package can be regenerated with:

```
python scripts/run_current_baseline_package.py \
  --split-dir data/splits/primary_real_source \
  --output-dir outputs/baselines/current_clean_mainline
```

The speech\-act\-guided proxy tuning run can be regenerated with:

```
python scripts/tune_pragaura_proxy_baseline.py
```

#### Slice diagnostics and consistency checks\.

Source\-level and claim\-type diagnostics are regenerated from the same split family, and the repository includes diagnostic scripts for the data and gapfill state:

```
python scripts/analyze_eval_slices.py --input <split_jsonl> ...
python scripts/analyze_shortcut_controls.py
python scripts/audit_source_overlap.py
python scripts/audit_selector_policy_constants.py
python scripts/audit_data_experiment_state.py
python scripts/audit_gapfill_state.py
```

#### Reproducibility package\.

The commands above rerun the reported implementation inside the project repository and define the split files, frozen outputs, and regeneration commands used by the experiments\. The anonymous supplemental package is a curated whitelist of the scripts, split files, stress slices, canonical outputs, and diagnostics needed to inspect the paper’s claims; non\-primary workflow holdouts, unfinished manual\-audit preparation folders, and historical diagnostic snapshots are intentionally excluded\. The package also includes a Croissant metadata file,artifact/croissant\_metadata\.json, with core dataset fields and Responsible AI notes for data collection, weak\-label annotation, known biases, limitations, and intended research use\.

## Appendix OSynthetic Stress Slice

The synthetic stress slice is reported separately from the main result table\. On the 20\-example synthetic stress set, the full system obtains UCCR=0\.0000=0\.0000, PAU=0\.0000=0\.0000, F1=0\.0000=0\.0000, and action accuracy=0\.6500=0\.6500\. The zero PAU and F1 values indicate a label\-interface mismatch, so the slice functions as a stress test while the real\-source\-only chain remains the primary result source\.