Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage

# Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage
Source: [https://arxiv.org/html/2605.30826](https://arxiv.org/html/2605.30826)
Shuheng Cao1,∗Ruiqi Chen2,∗Renjie Cao3,†Zhenhao Zhang4,†Siyu Zhang1,†Tingting Dan5,‡1University of California, San Diego2University of Michigan, Ann Arbor3The Hong Kong University of Science and Technology, Guangzhou4ShanghaiTech University5University of North Carolina, Chapel Hill∗Equal contribution as co\-first authors\.†Equal contribution\.‡Corresponding author\.

###### Abstract

Biomedical curation turns entity mentions into review decisions\. Modern LLM panels can surface plausible biomedical candidates, but multi\-model agreement signals salience and does not establish corpus convention correctness\. We define panel surfaced candidate verification as a curator\-facing layer after candidate generation, and introduce a candidate\-level panel output benchmark that aligns eight LLMs’ outputs over five public biomedical Named Entity Recognition\(NER\) datasets into candidate rows\. We instantiate this layer with BioConCal, a supervised scorer that uses inference\-time gold\-free panel and candidate features to rank a candidate stream\. In the in\-domain document split, BioConCal raises AUROC from 0\.753 for raw agreement to 0\.910\. At a validation selected 0\.95 precision target, it selects 1,340 candidate rows at empirical precision 0\.939, compared with 293 rows for raw agreement, withRcand=0\.592R\_\{\\mathrm\{cand\}\}=0\.592and row labelRcorpus=0\.523R\_\{\\mathrm\{corpus\}\}=0\.523against a panel row label ceiling of 0\.883\. These results show that calibrated panel evidence can turn a noisy candidate stream into a higher\-yield review queue, while preserving the workflow boundary that candidate sources set the recoverable universe\. Target domain validation, source expansion tradeoffs, span reconciliation, and human review remain downstream requirements\.

Beyond Agreement: Scoring Panel\-Surfaced Biomedical Entity Candidates for Curator Triage

Shuheng Cao1,∗Ruiqi Chen2,∗Renjie Cao3,†Zhenhao Zhang4,†Siyu Zhang1,†Tingting Dan5,‡1University of California, San Diego2University of Michigan, Ann Arbor3The Hong Kong University of Science and Technology, Guangzhou4ShanghaiTech University5University of North Carolina, Chapel Hill∗Equal contribution as co\-first authors\.†Equal contribution\.‡Corresponding author\.

## 1Introduction

Biomedical named entity recognition is often evaluated as span extraction\. In curation pipelines, however, extracted mentions become review decisions\. A curator must decide whether a candidate should be accepted, routed for review, or rejected\. False positives that reach a curated resource can be costly to trace and reverse\(Liet al\.,[2016](https://arxiv.org/html/2605.30826#bib.bib1); Doganet al\.,[2014](https://arxiv.org/html/2605.30826#bib.bib2); Smithet al\.,[2008](https://arxiv.org/html/2605.30826#bib.bib3); Collier and Kim,[2004](https://arxiv.org/html/2605.30826#bib.bib4); Krallingeret al\.,[2015](https://arxiv.org/html/2605.30826#bib.bib5)\)\.

Modern LLMs sharpen this decision point\. They can surface plausible biomedical candidates under new entity definitions without task\-specific training\(Singhalet al\.,[2023](https://arxiv.org/html/2605.30826#bib.bib13); Luoet al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib8); Nagaret al\.,[2024](https://arxiv.org/html/2605.30826#bib.bib23)\)\. Their outputs also include plausible non gold candidates, boundary variants, type confusions, and convention mismatches\. The central difficulty is not only whether a phrase looks biomedical\. It is whether the candidate follows the target corpus convention\. We use corpus convention correctness to mean compatibility with the target dataset boundary, entity type, granularity, normalization, occurrence, and annotation rules\. It does not mean independent biomedical truth verification\.

Multi\-model agreement is an obvious triage signal\. Candidates emitted by several models are often more salient\. Yet salience is not correctness under a corpus convention\. The question for curation is whether a stream of candidates surfaced by a model panel can be calibrated and prioritized for curator review\.

We call this problem panel surfaced candidate verification\. It is a layer after candidate generation and before human curation\. The input is a fixed candidate stream from a defined model panel\. The output is a candidate correctness score for the curator facing triage\. The unit is an aligned candidate row, not an individual LLM extractor output\. The panel defines the recoverable candidate universe, and scoring determines review yield within that universe\. We therefore separate candidate\-level triage from corpus\-level coverage\.

![Refer to caption](https://arxiv.org/html/2605.30826v1/x1.png)Figure 1:BioConCal overview\. A multi\-model panel first surfaces biomedical entity candidates, which are aligned into candidate rows for benchmark construction\. BioConCal scores this fixed candidate stream using inference\-time gold\-free panel and candidate features, then ranks candidates for curator triage\. The workflow separates candidate generation, candidate\-level verification, validation\-selected operating points, span reconciliation, and final human review\.We instantiate this setting with a candidate\-level panel output benchmark and BioConCal\. The benchmark aligns eight LLMs’ outputs over five public biomedical NER datasets into candidate rows with agreement structure, inference time gold\-free candidate features, and multiple correctness labels\. BioConCal is an in\-domain supervised scorer learned from labelled candidate rows\. At inference time, it uses panel\-derived signals and candidate metadata to prioritize a fixed candidate stream for review\. In domain, BioConCal raises AUROC from 0\.753 for raw agreement to 0\.910 and selects 1,340 candidate rows at empirical precision 0\.939 under the validation selected 0\.95 target, compared with 293 rows for raw agreement\.

We make four contributions\.

1. 1\.We define panel surfaced candidate verification, a curator facing problem that starts after candidate generation and asks which surfaced candidates should be prioritized for review\.
2. 2\.We construct a candidate\-level panel output benchmark by aligning eight LLMs’ outputs over five public biomedical NER datasets into candidate rows with agreement structure, candidate metadata, correctness labels, and candidate universe ceilings\.
3. 3\.We introduceBioConCal, an in\-domain supervised scorer that uses inference time gold free panel and candidate features to calibrate fixed stream candidates for curator triage\.
4. 4\.We analyze the workflow boundary, showing how agreement errors, validation\-selected operating points, source expansion, deterministic span reconciliation, and human review shape deployment\.

## 2Benchmark and Candidate Construction

This benchmark evaluates candidate verification after generation\. Its unit is an aligned candidate row surfaced by an explicitly defined panel\. Each row represents a possible curator decision rather than a standalone extractor output\.

### 2\.1Datasets

We use five biomedical NER datasets covering eight unified entity types \(chemical, disease, gene, protein, DNA, RNA, cell line, cell type\)\. Table[1](https://arxiv.org/html/2605.30826#S2.T1)lists the datasets\. Four datasets form the main benchmark\. CHEMDNER is treated as robustness only for offset\-sensitive analyses because the public mirror does not provide reliable character offsets\. For each dataset, we sample 500 documents with a fixed random seed\. This fixed\-size design balances entity schemas and keeps the eight model panel evaluation tractable\. The benchmark is not intended to replace full corpus extractor leaderboards\. It is designed to study candidate verification under a shared panel protocol\. The resulting benchmark contains 2,500 documents, and each model is evaluated on the same document identifiers, giving 20,000 model\-document extraction calls\.

DatasetEntity typesRoleBC5CDRchemical, diseaseMainNCBI DiseasediseaseMainBC2GMgeneMainJNLPBAprotein, DNA, RNA, cell line, cell typeMainCHEMDNERchemicalRobustness onlyTable 1:Five biomedical NER datasets\. Each contributes 500 documents to the benchmark\.
### 2\.2Model panel

The panel has eight models\. Two are closed frontier systems \(OpenAI GPT\-5\.5 and Claude Opus 4\.7\), and six are local open weight models covering dense, MoE, instruction\-tuned, and reasoning families\(Meta AI,[2024](https://arxiv.org/html/2605.30826#bib.bib17); Yanget al\.,[2025](https://arxiv.org/html/2605.30826#bib.bib18); DeepSeek\-AI,[2025](https://arxiv.org/html/2605.30826#bib.bib19); MiniMax AI,[2025](https://arxiv.org/html/2605.30826#bib.bib20); Mistral AI,[2025](https://arxiv.org/html/2605.30826#bib.bib21); Abdinet al\.,[2024](https://arxiv.org/html/2605.30826#bib.bib22)\)\. The panel is used as a heterogeneous candidate source rather than a leaderboard of individual LLMs\. Diversity across providers and model families creates the agreement and disagreement structure used by the downstream scorer\. Local serving uses vLLM with an OpenAI\-compatible endpoint\(Kwonet al\.,[2023](https://arxiv.org/html/2605.30826#bib.bib16)\)\. The full per model panel table is in Appendix Table[7](https://arxiv.org/html/2605.30826#A2.T7)\.

### 2\.3Candidate master table

For every document, we align model predictions into occurrence\-level candidate rows under\(normalized\_text, type\)keys\. Outputs assigned to the same occurrence slot are merged into one row with a vote bitmap, provider list, and surface variants\.

Repeated mentions are retained as separate rows\. If a document contains four mentions of the same chemical and the panel emits two surface variants, the candidate master keeps occurrence level rows rather than collapsing all evidence into one normalized string\. This design matches the curator\-facing unit of review\.

Surface variants and provider offsets are retained for downstream analysis\. Exact span labels use the first available provider offset\. Theoffset\_available\_countandoffset\_ambiguity\_countfields flag missing or many to many offset evidence for deterministic span alignment\.

Multiset coverage caps repeated candidate keys at the corresponding gold occurrence count\. Extra repeated candidates are overflow and cannot increase corpus recall\. Gold annotations define training and evaluation labels\. They are not used as input features at inference time\. The resulting table has 23,700 rows and is the unit of analysis for all candidate scoring experiments\.

#### Candidate universe\.

The panel defines the recoverable candidate universe\. All candidate\-level scoring, coverage, and selection metrics in this paper are computed over that universe\. Gold mentions missed by all panel models remain outside it\. A post hoc scorer can prioritize surfaced candidates, but it cannot select candidates no source produced\.

#### Two recall spaces\.

BioConCal ranks only surfaced candidates\. We therefore reportRcandR\_\{\\mathrm\{cand\}\}over positive candidate rows\. We also reportRcorpusR\_\{\\mathrm\{corpus\}\}over all corpus gold mentions\. This second metric includes the candidate generator ceiling because gold mentions that no panel model surfaced cannot be selected by a scorer\.

Two ceiling conventions distinguish triage from source comparison\. The row label panel ceiling is the maximumRcorpusR\_\{\\mathrm\{corpus\}\}under the paper’s candidate row labels, with 2,124 positive candidate rows divided by 2,404 gold mentions, or 0\.883\. The stricter multiset source ceiling measures represented gold occurrences after capping repeated\(document, type, normalized\_text\)keys by gold occurrence counts, giving 2,057/2,404 = 0\.856 for LLM\-8\. We use the stricter ceiling when comparing candidate generators such as BERN2\. Sampled documents absent from the candidate master have zero gold mentions in the released samples, so the corpus denominator includes all gold mentions in the sampled test documents and is unchanged by excluding zero\-candidate, zero\-gold documents from candidate\-row scoring\.

### 2\.4Matching variants

We use four matching variants to separate candidate identity from localization sensitivity\.Exact span typedeclares a prediction correct when document, entity type, surface text, start, and end agree\.Exact surface typedrops the offset constraint\.Normalized text typeadditionally tolerates case, whitespace, and punctuation variants and is the primary candidate label\.Relaxed overlap typeadditionally accepts substring containment of normalized text or overlapping offsets of the same type\. BioConCal predicts the normalized text type label\. Exact surface and exact span labels are used to characterize downstream localization sensitivity in Section[6\.1](https://arxiv.org/html/2605.30826#S6.SS1)\.

#### Candidate keying\.

The keying analyses test whether the benchmark conclusions depend on row granularity or label strictness\. The main benchmark uses occurrence\-aware rows and normalized text\-type labels\. Appendix Table[21](https://arxiv.org/html/2605.30826#A9.T21)varies row aggregation, while Appendix Table[20](https://arxiv.org/html/2605.30826#A9.T20)varies the correctness label with the candidate set held fixed\.

## 3BioConCal for Candidate Triage

BioConCal is a supervised scorer for fixed panel surfaced candidate rows\. For each row, it estimates correctness under the normalized text type label defined in Section[2\.4](https://arxiv.org/html/2605.30826#S2.SS4)\. Scores rank the candidate queue for curator review within the candidate universe defined by the panel\. BioConCal therefore changes review priority, not candidate generation or exact offset localization\.

The input representation uses only features available at inference time\. The features fall into four groups\.Agreement structure: total, closed\-model, and open\-model agreement counts, plus a model\-family diversity score over six families\.Mention features: surface length, token count, document occurrence count, and binary indicators for dash, slash, parenthesis, digit, Greek letter, and capitalized abbreviation\.Surface availability: exact and normalized surface presence in the document, plus the count and ambiguity of provider\-supplied offsets\.Document features: document length and one\-hot entity type\. Gold annotations define the supervised training label\. Gold\-derived signals such as gold entity count, hardness score, and difficulty bucket are excluded from the inputs to prevent leakage\. The complete feature list is in Appendix Table[14](https://arxiv.org/html/2605.30826#A4.T14)\.

We instantiate BioConCal with two standard supervised scorers\. Logistic regression gives a linear treatment of standardized features\. A gradient boosted tree \(GBT\) scorer captures nonlinear interactions usingHistGradientBoostingClassifierwith 300 iterations, learning rate 0\.06, and L2 regularization 0\.5\. We also include isotonic regression on the raw agreement count as a single feature calibration baseline\. Expected calibration error \(ECE\) is computed with 10 equal\-width bins\. Calibration refers to empirical probability scoring and validation\-selected operating points\. It is not a distribution\-free precision guarantee\.

### 3\.1Training and evaluation

All train, validation, and test splits are at document level\. The primary in\-domain protocol uses a 60/20/20 split stratified by dataset, seed 42\. We assert split integrity by checking that no\(dataset,doc\_id\)\(\\text\{dataset\},\\text\{doc\\\_id\}\)pair appears in more than one split\. The check is saved intables/split\_integrity\_check\.json\.

We use a leave one dataset out \(LODO\) protocol to separate ranking transfer from operating point transfer\. In each fold, BioConCal trains on three of the four main datasets and tests on the held\-out dataset\. Within the training pool, documents are split 75/25 into inner training and inner validation folds\. CHEMDNER remains a separate robustness setting\.

Operating points are selected only on validation data\. For the 0\.90, 0\.95, and 0\.97 precision targets, thresholds are selected on the validation fold and then*frozen*before scoring the test fold\. We never select thresholds on test data\. We report AUROC, AUPRC, Brier score, and ECE, plus empirical test precision, test recall, selected count, and false positive count at each validation selected threshold\.

## 4Experimental Setup

The extraction protocol defines the candidate stream before BioConCal scoring\. All panel models receive the same task instruction for a given dataset\. The prompt asks each model to extract only explicitly mentioned biomedical entities from the dataset allowed type set and to return a JSON object with anentitiesarray\. Raw outputs are parsed into a common schema\. For MiniMax\-M2, Mistral\-Small 3\.2, and Phi\-4 we use vLLM guided decoding with a JSON schema, which prevents long prose reasoning and constrains output to schema\-valid JSON\. Missing or parse\-failed rows count as zero predictions\. This keeps the candidate universe tied to surfaced model outputs rather than post hoc repair\.

## 5Main Results

### 5\.1Candidate sources and agreement baselines

We report single\-model scores only to characterize the panel as a heterogeneous candidate source\. The objective is not to rank LLM extractors, but to study the correctness of the candidates they surface\. Among single models, OpenAI gives the highest normalized mention F1 \(0\.800\) with the highest recall, while Claude gives higher precision \(0\.827\) at lower recall\. The open\-weight models occupy distinct points in the precision\-recall plane, which is the diversity the agreement analysis needs\. The obvious way to convert a panel into a review queue is to threshold agreement count, and this section tests that heuristic directly\. Voting baselines follow the standard ensemble pattern\. Union maximizes recall but admits many false positives, higher agreement thresholds raise precision and cut recall, and unanimous voting reaches the highest precision \(0\.940\) while recovering only 11\.5 percent of gold mentions\. The full single\-model and voting tables are in Appendix Tables[8](https://arxiv.org/html/2605.30826#A2.T8)and[9](https://arxiv.org/html/2605.30826#A2.T9)\.

### 5\.2Agreement is informative but incomplete

Per\-mention precision rises monotonically with agreement count, from 0\.266 atk=1k\{=\}1to 0\.940 atk=8k\{=\}8, and the largest single\-step gain is betweenk=1k\{=\}1andk=2k\{=\}2\. Even unanimous agreement does not eliminate error\. The unanimous set contains 1,496 predictions of which 90 are false positives, a residual rate of 6\.0 percent\. This is the empirical form of the deceptive\-simplicity problem\. A candidate can be salient enough for every model to emit, yet still violate corpus\-specific correctness through boundary, type, multiplicity, or annotation\-convention mismatch\. We inspect these high\-agreement errors in the curation\-oriented audit analysis\. The per\-kkcalibration curve is in Appendix Figure[6](https://arxiv.org/html/2605.30826#A16.F6)\.

### 5\.3Candidate scoring beyond agreement count

Learned candidate scoring converts panel agreement into a higher\-yield review queue within the fixed candidate universe\. Under the in\-domain document\-level split, BioConCal\-GBT raises ranking AUROC from 0\.753 for the one\-feature agreement\-count scorer to 0\.910\. It also lowers Brier score from 0\.205 to 0\.119\. Because BioConCal scores only surfaced candidates, these gains are triage gains within the candidate universe rather than new candidate\-generation recall\. The full scorer comparison is in Table[3](https://arxiv.org/html/2605.30826#S5.T3), and the operating\-point comparison is in Table[2](https://arxiv.org/html/2605.30826#S5.T2)\.

![Refer to caption](https://arxiv.org/html/2605.30826v1/x2.png)Figure 2:Why learned candidate scoring improves over agreement count, on the document\-level 60/20/20 test fold\. \(A\) BioConCal improves the raw\-agreement precision\-coverage tradeoff and reaches far higher coverage at the validation\-frozen 0\.95 precision target\. \(B\) Agreement carries the largest signal, while candidate metadata helps convert agreement into high\-yield review selection\. Agreement\-only and non\-agreement\-only scoring each recover little precision\-targeted candidate recall, while the main BioConCal model reaches 0\.592\. \(C\) Grouped permutation importance\. Agreement features account for 67\.5 percent of grouped importance, and mention features contribute 26\.5 percent\. Source:scripts/main\_candidate\_scoring\_evidence\.py\.
### 5\.4Validation\-targeted high\-precision selection

MethodSel\.FPTest PRcandRcorpusCeil\.*Fixed voting rules \(no threshold tuning\)*Agreement≥\\geq22,5877970\.6920\.8430\.7450\.883Agreement≥\\geq41,2752600\.7960\.4780\.4220\.883Agreement≥\\geq6715930\.8700\.2930\.2590\.883Unanimous \(k=8k=8\)293140\.9520\.1310\.1160\.883OpenAI∪\\cupClaude \(closed≥1\\geq 1\)2,7437430\.7290\.9420\.8320\.883*Validation\-frozen P95 threshold*Raw agreement / 8293140\.9520\.1310\.1160\.883Isotonic on agreement293140\.9520\.1310\.1160\.883BioConCal \(logistic\)1,229850\.9310\.5390\.4760\.883BioConCal \(GBT\)1,340820\.9390\.5920\.5230\.883Table 2:Validation\-targeted high\-precision selection on the document\-level 60/20/20 test fold\. Threshold\-tuned methods use a 0\.95 precision threshold selected on validation and applied frozen at test\.RcandR\_\{\\mathrm\{cand\}\}is over 2,124 gold\-positive candidate rows\.RcorpusR\_\{\\mathrm\{corpus\}\}is over 2,404 corpus gold mentions in the row\-label recall space\. “Ceil\.” is the within\-panel row\-label ceiling 2,124/2,404=0\.883, distinct from the stricter multiset generator\-source ceiling in Appendix Table[12](https://arxiv.org/html/2605.30826#A3.T12)\. Source:tables/risk\_control\_selection\_docsplit\.csvandtables/end\_to\_end\_interpretation\.csv\.The ranking gain translates into a larger high\-precision review queue\. At a 0\.95 precision threshold selected on the validation fold and applied frozen at test, BioConCal\-GBT selects 1,340 candidates with 82 false positives and empirical test precision 0\.939\. Raw agreement selects 293 candidates with 14 false positives at empirical precision 0\.952\. Within the candidate universe, BioConCal increases recall from 0\.131 to 0\.592 at comparable empirical precision\.

Against all corpus gold mentions, the same operating point reaches 0\.523 recall in the row\-label recall space, below the panel\-union ceiling of 0\.883\. Under the stricter source\-comparison accounting used for generator and hybrid analyses, the same LLM\-8 operating point is reported as 0\.522 in Appendix Table[27](https://arxiv.org/html/2605.30826#A14.T27)\. The gain is therefore a curator\-triage gain over surfaced candidates, not an end\-to\-end extraction recall gain\. Compared with unanimity, BioConCal exposes 979 additional true candidates at the cost of 68 additional false positives\. This is validation\-targeted selection rather than a precision guarantee, so we report empirical test precision directly\.

Monotonic transforms of the agreement count, including Platt scaling and isotonic regression, yield the same selected set as the raw agreement threshold\. Split\-conformal\-inspired score cutoffs over the validation\-fold false\-positive quantile recover more candidates than raw thresholding but fall slightly below the 0\.95 nominal precision on test\. None of these baselines closes the precision\-coverage gap achieved by BioConCal\. The full table is in Appendix Table[28](https://arxiv.org/html/2605.30826#A17.T28)\.

### 5\.5Stronger candidate\-scoring baselines

To test whether the gain comes from agreement count alone, vote identity alone, or candidate metadata alone, Table[3](https://arxiv.org/html/2605.30826#S5.T3)compares progressively richer inference\-time gold\-free scorers under the same document\-level split and validation\-frozen P95 protocol\.

ScorerAUROCBrierECEP@95Rcand@95Rcorpus@95Raw agreement count0\.7530\.2050\.0770\.9520\.1310\.116Vote\-vector logistic0\.8640\.1480\.0590\.9410\.1510\.134Aggregate\-agreement0\.8610\.1480\.0560\.9520\.1310\.116Non\-agreement only0\.8650\.1510\.0430\.9520\.0280\.025BioConCal \(logistic\)0\.8910\.1350\.0370\.9310\.5390\.476BioConCal \(GBT\)0\.9100\.1190\.0320\.9390\.5920\.523Table 3:Stronger candidate\-scoring baselines on the document\-level 60/20/20 test fold\. Vote\-vector logistic uses the 8 per\-model vote indicators\. Aggregate\-agreement uses the 4 agreement\-structure features\. Non\-agreement only uses mention, surface, and document features\. P@95, Rcand@95, and Rcorpus@95 are at the validation\-frozen P95 threshold\. Source:tables/stronger\_candidate\_scoring\_baselines\_p95\.csv\. The complete logistic\-and\-GBT grid is in Appendix Table[16](https://arxiv.org/html/2605.30826#A7.T16)\.Three controls clarify the source of the gain\. Scorers built only on agreement structure, whether the 8\-way vote vector or the 4 aggregate\-agreement features, reach AUROC near 0\.86 but recover little precision\-targeted recall\. The agreement pattern alone is therefore informative but insufficient for high\-yield review\. On the primary split, a non\-agreement scorer using mention, surface, and document features reaches AUROC 0\.865 but selects only 63 rows at the validation\-frozen P95 threshold\. Candidate metadata alone therefore does not explain the main review\-yield result\.

BioConCal combines agreement and candidate metadata and reaches Rcand0\.592 at empirical precision 0\.939\. Permutation importance on the validation fold attributes about 67 percent of grouped importance to agreement features and 27 percent to mention features, withclosed\_agreement\_countthe strongest individual feature\. The GBT and logistic forms of BioConCal are close in AUROC, 0\.910 against 0\.891\. BioConCal is therefore best read as a panel\-surfaced candidate verification framework rather than a single fixed classifier architecture\.

A feature\-removal ablation shows AUROC within±0\.004\\pm 0\.004of the main BioConCal model across variants that keep both agreement and non\-agreement features\. Only agreement\-only and vote\-pattern\-only variants lose the high\-yield selection behavior\. Full ablations are in Appendix[T](https://arxiv.org/html/2605.30826#A20)\.

### 5\.6Robustness across splits

Across 10 document\-level splits, BioConCal\-GBT remains stable and consistently outperforms raw agreement, with AUROC 0\.907±\\pm0\.010\. A document\-level bootstrap on seed 42 gives BioConCal\-GBT AUROC and Rcand95% intervals that do not overlap raw agreement\. The full seed\-wise and bootstrap results are in Appendix[H](https://arxiv.org/html/2605.30826#A8)\.

### 5\.7Cheaper\-panel and single\-model candidate baselines

Panel size changes both scoring information and candidate coverage\. We train candidate classifiers under restricted candidate universes on the same document\-level 60/20/20 split and the same inference\-time gold\-free feature set\. Each panel defines its own candidate universe by recomputing agreement features over the subset and dropping rows with zero in\-panel votes\. For comparability, Rfull@95 is measured over the full eight\-panel positive candidate set\. Smaller panels are therefore penalized for candidates they cannot surface\.

Panel\|M\|\|M\|Cand\.AUROCRfull@95OpenAI112,6280\.8150\.170Claude19,3240\.8140\.276Best open17,9790\.8800\.262Closed\-2213,6980\.8060\.286Diverse\-3315,1560\.8800\.446Open\-6614,4740\.9080\.335Full\-8818,8120\.9100\.592

Table 4:Cheap\-panel baselines on the document\-level test split\. Cheaper panels recover much of AUROC, while the full panel gives higher high\-precision recall by expanding the candidate universe\. Rfull@95 is empirical test recall over the full eight\-panel positive candidate set at the validation\-selected P95 operating point\. Best open is MiniMax\-M2\. Diverse\-3 is OpenAI plus MiniMax\-M2 plus DeepSeek\-R1\. The full table with precision, FP, and coverage is in Appendix Table[25](https://arxiv.org/html/2605.30826#A13.T25)\.Cheaper panels recover much of the in\-domain ranking AUROC, but they do not recover the same high\-precision recall\. Open\-6 reaches AUROC 0\.908 and Diverse\-3 reaches 0\.880, compared with 0\.910 for Full\-8\. At the validation\-frozen 0\.95 precision target, single\-model and closed\-only pipelines reach at most 0\.286 Rfull@95\. Diverse\-3 reaches 0\.446, while Full\-8 reaches 0\.592 by expanding the candidate universe\. These results support cost\-aware panel selection rather than a single universal panel\. The appropriate panel depends on whether deployment prioritizes cost, reproducibility, or maximum recoverable review yield\. The gain decomposition in Appendix Table[L](https://arxiv.org/html/2605.30826#A12)provides the corresponding feature\-level comparison\.

### 5\.8LODO and OOD deployment conditions

MethodAUROCBrierP@95Rcand@95Raw agreement / 80\.7910\.1990\.8930\.230Isotonic on agreement0\.7910\.2260\.8930\.230BioConCal \(logistic\)0\.8560\.1510\.8980\.170BioConCal \(GBT\)0\.8510\.1820\.9100\.170

Table 5:LODO macro mean across the four main datasets, with the threshold selected on the inner\-validation fold and frozen at test\. AUROC measures ranking transfer, while P@95 and Rcand@95 depend on transferring a source\-selected operating point\. Values are the macro means fromtables/bioconcal\_lodo\_nested\_thresholds\.csv\. Per\-fold values are in Appendix Table[22](https://arxiv.org/html/2605.30826#A10.T22)\.LODO separates ranking transfer from operating\-point transfer\. BioConCal improves mean AUROC over raw agreement, from 0\.791 to 0\.851 for GBT and 0\.856 for logistic\. At the validation\-frozen P95 operating point, however, source\-selected thresholds fall below the nominal precision target on average\. These results argue for target\-domain threshold validation before deployment\. They do not imply that a small target calibration set always restores a high\-yield operating point under entity\-type shift\. Per\-fold transfer and target\-domain recalibration results are in Appendix Tables[22](https://arxiv.org/html/2605.30826#A10.T22)and[23](https://arxiv.org/html/2605.30826#A11.T23)\.

#### Deployment\.

Deployment should select thresholds on target\-domain validation data, report empirical precision after transfer, and use scores for triage rather than automatic insertion\. The full deployment checklist is in Appendix[J](https://arxiv.org/html/2605.30826#A10.SS0.SSS0.Px1)\.

The reliability diagram in Appendix Figure[5](https://arxiv.org/html/2605.30826#A15.F5)shows that BioConCal predicted probabilities track empirical correctness closely on the in\-domain fold\.

### 5\.9An external extractor as a candidate source

External candidate sources raise the recoverable ceiling, but they also change the verification burden\. We evaluate BERN2\(Sunget al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib35)\)as an external candidate source on the same documents\. Under strict multiset accounting, the ceiling is 0\.856 for LLM\-8, 0\.737 for BERN2, and 0\.901 for LLM\-8∪\\cupBERN2\.

BioConCal\-Hybrid addresses scorer comparability with a learned GBT over a merged LLM\-8∪\\cupBERN2 universe\. It reaches AUROC 0\.935 and AUPRC 0\.948, and selects 1,847 candidate rows, including 1,716 row\-label positives and 131 false positives\. Under multiset corpus accounting, 1,705 selected gold occurrences over 2,404 corpus gold mentions giveRcorpus=0\.709R\_\{\\mathrm\{corpus\}\}=0\.709at empirical precision 0\.929\. This recovers more corpus gold than the primary LLM\-8 BioConCal workflow, but at lower empirical precision and with more false positives\. The comparison supports the workflow decomposition\. Generators set the recoverable universe, while learned verification determines review yield and false\-positive burden within that universe\. Full comparisons are in Appendix Tables[12](https://arxiv.org/html/2605.30826#A3.T12)and[13](https://arxiv.org/html/2605.30826#A3.T13)\.

## 6Curation\-Oriented Analysis

### 6\.1Span reconciliation after candidate triage

Candidate triage resolves normalized entity identity, but curation still requires exact character localization\. We therefore treat span reconciliation as a downstream step rather than as part of the scorer\. Deterministic realignment searches exact, normalized, and supporting\-span fallbacks, then chooses the match nearest the model offset\. After alignment, exact\-span F1 rises from 0\.076 to 0\.777 for OpenAI and from 0\.162 to 0\.708 for Claude\. These gains show that candidate verification and localization are separable stages\. The scorer prioritizes which normalized candidates should be reviewed, and alignment prepares selected candidates for offset\-level inspection\. Per\-rule results are in Appendix Table[W](https://arxiv.org/html/2605.30826#A23)\.

### 6\.2Auditing agreement errors and review boundaries

Audit sheets make the residual review burden explicit\. We release the 90 unanimous false positives and three disagreement audit sheets covering 758 further candidates \(Appendix[E](https://arxiv.org/html/2605.30826#A5)\)\. These sheets surface cases where agreement, source identity, or normalization cues may diverge from corpus convention\. Their category labels are auto\-suggested and are intended for follow\-up annotation review, not as manually verified error labels\.

Stratified calibration preserves the same broad increasing trend, with expected deviations in small dataset and entity type strata\. The normalization ablation preserves the central pattern across matching variants \(Appendix Figure[9](https://arxiv.org/html/2605.30826#A26.F9)and Appendix[Z](https://arxiv.org/html/2605.30826#A26)\)\. These checks define the boundary of the review workflow rather than a new automatic decision rule\. BioConCal produces the ranked candidate queue, span reconciliation prepares selected normalized candidates for offset inspection, audit sheets expose cases requiring human adjudication, and source expansion changes the precision coverage tradeoff\. Appendix Table[27](https://arxiv.org/html/2605.30826#A14.T27)makes this boundary quantitative\. The primary LLM\-8 BioConCal workflow gives the largest high precision review yield in the LLM\-8 queue, while BioConCal\-Hybrid recovers more corpus gold at lower empirical precision\.

## 7Conclusion

We introduced a candidate\-level panel output benchmark for panel surfaced candidate verification, a curator triage layer between biomedical entity extraction and human review\. In the in\-domain document split, BioConCal raised AUROC from 0\.753 for raw agreement to 0\.910\. At the validation selected 0\.95 precision target, it selected 1,340 candidate rows at empirical precision 0\.939, compared with 293 rows for raw agreement\. This operating point givesRcand=0\.592R\_\{\\mathrm\{cand\}\}=0\.592and row labelRcorpus=0\.523R\_\{\\mathrm\{corpus\}\}=0\.523within a panel universe whose row label ceiling is 0\.883\. The results support a workflow decomposition in which candidate sources set the recoverable universe, learned scoring determines review yield within that universe, and source expansion through BERN2 trades coverage for lower empirical precision\. Target domain validation, deterministic span reconciliation, and human review remain required before selected candidates can support curated resource updates\.

## Limitations

#### Candidate universe and verifier scope\.

BioConCal scores candidates that have already been surfaced by the panel\. It does not generate new mentions and cannot recover gold mentions missed by every panel model\. Within panel corpus recall is therefore bounded by the row label ceiling of 0\.883 on the test fold\. BioConCal is also not an upper bound on context\-aware verification\. Richer context\-aware verifiers can be used after BioConCal when a candidate needs textual evidence beyond panel and candidate metadata\.

#### Portability and operating points\.

BioConCal is tied to the candidate sources, prompts, decoding constraints, corpus convention, and entity type mix used to train and validate it\. If any of these change, the classifier and its operating threshold require target domain validation before deployment\. The LODO and target recalibration analyses show why this validation matters under entity type shift, but they do not guarantee that 25 to 100 labeled documents will recover the same yield in every target corpus\. Prompt sensitivity is evaluated on an open weight subset in Appendix[X](https://arxiv.org/html/2605.30826#A24)\.

#### Human review and reproducibility\.

BioConCal scores define a review priority, not automatic acceptance or rejection\. False positives can contaminate curated biomedical resources if accepted without review, and false negatives can leave useful candidates outside the high\-priority queue\. High scores, low scores, and audit categories should therefore be treated as triage signals for human adjudication\. The released audit categories are auto\-suggested and are intended to guide follow\-up review rather than serve as verified error labels\. Closed API providers introduce cost and reproducibility asymmetry, while the open weight six model variant is the reproducible alternative\. CHEMDNER is kept as a robustness dataset because its public mirror lacks reliable character offsets\. We report empirical test precision and do not claim distribution\-free precision guarantees\.

## Ethics and Reproducibility

All datasets are public biomedical NER resources, and the work introduces no new human\-subject data\. We release all analysis scripts, and the open\-weight six\-model variant is locally reproducible\. The full eight\-model benchmark depends on closed API access\. Reproduction commands are in Appendix[A](https://arxiv.org/html/2605.30826#A1)\.

## References

- M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.\(2024\)Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- A gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.InarXiv preprint,Note:Preprint[https://arxiv\.org/abs/2107\.07511](https://arxiv.org/abs/2107.07511)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- N\. Collier and J\. Kim \(2004\)Introduction to the bio\-entity recognition task at jnlpba\.InProceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications,pp\. 73–78\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p1.1)\.
- U\. Das, K\. B\. Atmakuri, D\. Ho, C\. Lee, and Y\. Lee \(2026\)Clinical knowledge graph construction and evaluation with multi\-llms via retrieval\-augmented generation\.Note:arXiv preprint arXiv:2601\.01844Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- R\. de Oliveira, M\. Garber, J\. M\. Gwinnutt, E\. Rashidi, J\. Hwang, W\. Gilmour, J\. Nanavati, K\. Zine El Abidine, and C\. D\. Mack \(2025\)A study of calibration as a measurement of trustworthiness of large language models in biomedical natural language processing\.JAMIA Open8\(4\),pp\. ooaf058\.External Links:[Document](https://dx.doi.org/10.1093/jamiaopen/ooaf058)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- T\. G\. Dietterich \(2000\)Ensemble methods in machine learning\.InMultiple Classifier Systems,pp\. 1–15\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- R\. I\. Dogan, R\. Leaman, and Z\. Lu \(2014\)NCBI disease corpus: a resource for disease name recognition and concept normalization\.Journal of Biomedical Informatics47,pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2013.12.006)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p1.1)\.
- J\. Fries, N\. Seelam, G\. Altay, L\. Weber, M\. Kang, D\. Datta, R\. Su, S\. Garda, B\. Wang, S\. Ott, M\. Samwald, and W\. Kusa \(2022\)Dataset debt in biomedical language modeling\.InProceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models,virtual\+Dublin,pp\. 137–145\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.bigscience-1.10)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- Y\. Gu, R\. Tinn, H\. Cheng, M\. Lucas, N\. Usuyama, X\. Liu, T\. Naumann, J\. Gao, and H\. Poon \(2021\)Domain\-specific language model pretraining for biomedical natural language processing\.ACM Transactions on Computing for Healthcare3\(1\),pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.1145/3458754)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[Table 6](https://arxiv.org/html/2605.30826#A2.T6.1.3.1.1.1)\.
- A\. Khandelwal, A\. Kar, V\. R\. Chikka, and K\. Karlapalem \(2022\)Biomedical NER using novel schema and distant supervision\.InProceedings of the 21st Workshop on Biomedical Language Processing,Dublin, Ireland,pp\. 155–160\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.bionlp-1.15)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- M\. Krallinger, O\. Rabal, F\. Leitner, M\. Vazquez, D\. Salgado, Z\. Lu, R\. Leaman,et al\.\(2015\)The chemdner corpus of chemicals and drugs and its annotation principles\.Journal of Cheminformatics7\(Suppl 1\),pp\. S2\.External Links:[Document](https://dx.doi.org/10.1186/1758-2946-7-S1-S2)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- J\. Lee, W\. Yoon, S\. Kim, D\. Kim, S\. Kim, C\. H\. So, and J\. Kang \(2020\)BioBERT: a pre\-trained biomedical language representation model for biomedical text mining\.Bioinformatics36\(4\),pp\. 1234–1240\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btz682)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[Table 6](https://arxiv.org/html/2605.30826#A2.T6.1.3.1.1.1)\.
- J\. Li, Y\. Sun, R\. J\. Johnson, D\. Sciaky, C\. Wei, R\. Leaman, A\. P\. Davis, C\. J\. Mattingly, T\. C\. Wiegers, and Z\. Lu \(2016\)BioCreative v cdr task corpus: a resource for chemical disease relation extraction\.Database2016,pp\. baw068\.External Links:[Document](https://dx.doi.org/10.1093/database/baw068)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p1.1)\.
- Z\. Lin, S\. Trivedi, and J\. Sun \(2023\)Generating with confidence: uncertainty quantification for black\-box large language models\.Note:arXiv preprint arXiv:2305\.19187Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- R\. Luo, L\. Sun, Y\. Xia, T\. Qin, S\. Zhang, H\. Poon, and T\. Liu \(2022\)BioGPT: generative pre\-trained transformer for biomedical text generation and mining\.Briefings in Bioinformatics23\(6\),pp\. bbac409\.External Links:[Document](https://dx.doi.org/10.1093/bib/bbac409)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p2.1)\.
- T\. Lv, L\. Luo, J\. Li, Y\. Wang, Y\. Pan, C\. Liu, Y\. Wang, Y\. Jiang, H\. Lv, Y\. Sun, J\. Wang, and H\. Lin \(2025\)A unified biomedical named entity recognition framework with large language models\.Note:arXiv preprint arXiv:2510\.08902Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- Meta AI \(2024\)Llama 3\.3 70b instruct\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- MiniMax AI \(2025\)MiniMax\-m2\.Note:GitHub repositoryExternal Links:[Link](https://github.com/MiniMax-AI/MiniMax-M2)Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- Mistral AI \(2025\)Mistral small 3\.2\.Note:Model cardExternal Links:[Link](https://docs.mistral.ai/models/model-cards/mistral-small-3-2-25-06)Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- A\. Nagar, V\. Schlegel, T\. Nguyen, H\. Li, Y\. Wu, K\. Binici, and S\. Winkler \(2024\)LLMs are not zero\-shot reasoners for biomedical information extraction\.Note:arXiv preprint arXiv:2408\.12249Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p2.1)\.
- A\. Niculescu\-Mizil and R\. Caruana \(2005\)Predicting good probabilities with supervised learning\.Proceedings of the 22nd International Conference on Machine Learning \(ICML\),pp\. 625–632\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- C\. G\. Northcutt, L\. Jiang, and I\. L\. Chuang \(2021\)Confident learning: estimating uncertainty in dataset labels\.InJournal of Artificial Intelligence Research,Vol\.70,pp\. 1373–1411\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- M\. S\. Obeidat, M\. S\. Al Nahian, and R\. Kavuluru \(2025\)Do LLMs surpass encoders for biomedical NER?\.In2025 IEEE International Conference on Healthcare Informatics \(ICHI\),pp\. 352–358\.External Links:[Document](https://dx.doi.org/10.1109/ICHI64645.2025.00048)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- J\. Platt \(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.Advances in Large Margin Classifiers10\(3\),pp\. 61–74\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620,pp\. 172–180\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06291-2)Cited by:[§1](https://arxiv.org/html/2605.30826#S1.p2.1)\.
- L\. Smith, L\. K\. Tanabe, R\. J\. Ando, C\. Kuo, I\. Chung, C\. Hsu, Y\. Lin, R\. Klinger, C\. M\. Friedrich, K\. Ganchev,et al\.\(2008\)Overview of biocreative ii gene mention recognition\.Genome Biology9\(Suppl 2\),pp\. S2\.External Links:[Document](https://dx.doi.org/10.1186/gb-2008-9-s2-s2)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.30826#S1.p1.1)\.
- M\. Sung, M\. Jeong, Y\. Choi, D\. Kim, J\. Lee, and J\. Kang \(2022\)BERN2: an advanced neural biomedical named entity recognition and normalization tool\.Bioinformatics38\(20\),pp\. 4837–4839\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/btac598)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1),[Table 6](https://arxiv.org/html/2605.30826#A2.T6.1.2.1.1.1),[§5\.9](https://arxiv.org/html/2605.30826#S5.SS9.p1.1)\.
- S\. Swayamdipta, R\. Schwartz, N\. Lourie, Y\. Wang, H\. Hajishirzi, N\. A\. Smith, and Y\. Choi \(2020\)Dataset cartography: mapping and diagnosing datasets with training dynamics\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.Note:arXiv preprint arXiv:2305\.14975Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.
- T\. Watanabe, T\. Ichikawa, A\. Tamura, T\. Iwakura, C\. Ma, and T\. Kato \(2022\)Auxiliary learning for named entity recognition with multiple auxiliary biomedical training data\.InProceedings of the 21st Workshop on Biomedical Language Processing,Dublin, Ireland,pp\. 130–139\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.bionlp-1.13)Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§2\.2](https://arxiv.org/html/2605.30826#S2.SS2.p1.1)\.
- M\. Yuan, J\. Chen, Z\. Xing, G\. Mohammadi, and A\. Quigley \(2025\)A case study of scalable content annotation using multi\-llm consensus and human review\.Note:GenAICHI Workshop at CHI 2025; arXiv:2503\.17620Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px2.p1.1),[Table 6](https://arxiv.org/html/2605.30826#A2.T6.1.5.1.1.1)\.
- Y\. Yuan, W\. Wang, Q\. Guo, Y\. Xiong, C\. Shen, and P\. He \(2024\)Does chatgpt know that it does not know? evaluating the black\-box calibration of chatgpt\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 5191–5201\.Cited by:[Appendix B](https://arxiv.org/html/2605.30826#A2.SS0.SSS0.Px3.p1.1)\.

## Appendix AReproducibility commands

The candidate master and downstream analysis tables can be regenerated from the released sample documents, parsed provider outputs, consensus candidate artifacts, and table sources\. Full extraction regeneration additionally requires rerunning the provider calls; the downstream scripts below are run from the project root\.

build\_candidate\_master\.py: candidate\-level master\.

train\_bioconcal\_v3\.py: doc\-level 60/20/20 \+ LODO with inner\-val thresholds\.

selective\_baselines\.py: selective and conformal baselines\.

panel\_composition\_ablation\.py: panel ablations\.

feature\_importance\.py: permutation importance \+ figure\.

span\_alignment\_audit\_sample\.py: 100\-row span audit sample\.

extended\_span\_alignment\.py: span alignment metrics\.

audit\_unanimous\_fps\.py: unanimous FP audit sheet \(multiset\)\.

finalize\_unanimous\_fp\_audit\.py: refined audit with subcategories\.

disagreement\_audit\.py: candidate audit sheets\.

agreement\_calibration\_by\_strata\.py: per\-stratum\.

normalization\_ablation\.py: normalization deltas\.

stronger\_candidate\_scoring\_baselines\.py: consolidated stronger\-baseline table\.

main\_candidate\_scoring\_evidence\.py: 3\-panel main evidence figure\.

train\_hybrid\_candidate\_scorer\.py: learned LLM\-8∪\\cupBERN2 candidate scorer\.

Script locations are mixed by design\. The candidate\-master, training, ablation, and audit\-generation scripts live underpaper\_experiment\_materials/strong\_accept/scripts/\. The unanimous\-audit finalization, stratum calibration, main evidence, stronger\-baseline, and hybrid\-scoring scripts live under top\-levelscripts/\.

## Appendix BRelated work, model panel, and baseline tables

#### Biomedical NER and LLM candidate generation\.

Public biomedical NER benchmarks cover chemical, disease, gene, protein, nucleic acid, cell line, and cell type mentions\(Liet al\.,[2016](https://arxiv.org/html/2605.30826#bib.bib1); Doganet al\.,[2014](https://arxiv.org/html/2605.30826#bib.bib2); Smithet al\.,[2008](https://arxiv.org/html/2605.30826#bib.bib3); Collier and Kim,[2004](https://arxiv.org/html/2605.30826#bib.bib4); Krallingeret al\.,[2015](https://arxiv.org/html/2605.30826#bib.bib5)\)\. Biomedical encoders and generative biomedical language models include BioBERT, PubMedBERT, and BioGPT\(Leeet al\.,[2020](https://arxiv.org/html/2605.30826#bib.bib6); Guet al\.,[2021](https://arxiv.org/html/2605.30826#bib.bib7); Luoet al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib8)\)\. Supervised biomedical NER work studies multi\-dataset auxiliary learning, discontinuous and overlapping entities, and dataset availability debt\(Watanabeet al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib9); Khandelwalet al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib10); Frieset al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib11)\)\. Established systems combine neural recognition with entity normalization, and BERN2\(Sunget al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib35)\)is a widely used biomedical NER and normalization reference point\. LLM\-based biomedical information extraction has been studied for zero\-shot biomedical reasoning\(Nagaret al\.,[2024](https://arxiv.org/html/2605.30826#bib.bib23)\), LLM versus encoder biomedical NER tradeoffs\(Obeidatet al\.,[2025](https://arxiv.org/html/2605.30826#bib.bib12)\), flat and nested NER under generative frameworks\(Lvet al\.,[2025](https://arxiv.org/html/2605.30826#bib.bib24)\), and multi\-LLM workflows for clinical knowledge graph construction\(Daset al\.,[2026](https://arxiv.org/html/2605.30826#bib.bib25)\)\. These works ask whether an extractor or generator can produce entity predictions under a task schema\. We start after candidate sources have produced candidates and evaluate aligned candidate rows as review decisions\.

#### Panel\-surfaced verification for curator review\.

Multi LLM consensus paired with selective human review has been studied as an annotation pattern\(Yuanet al\.,[2025](https://arxiv.org/html/2605.30826#bib.bib26)\)\. BioConCal studies a different workflow layer\. Given a fixed stream of candidates surfaced by a multi\-model panel, the question is whether those candidates can be ranked for curator review using agreement structure and inference time gold\-free candidate features\. External extractors such as BERN2 can expand the candidate universe, but they do not remove the need for source\-specific verification\. Context\-aware verification uses richer document evidence to adjudicate individual candidates\. It is complementary to the panel\-derived triage layer studied here, which first ranks a large candidate stream for review\. Appendix Table[6](https://arxiv.org/html/2605.30826#A2.T6)contrasts evaluation units across related settings\.

#### Agreement, calibration, and LLM uncertainty\.

Ensembles and self\-consistency are widely used to improve reliability\(Dietterich,[2000](https://arxiv.org/html/2605.30826#bib.bib15); Wanget al\.,[2023](https://arxiv.org/html/2605.30826#bib.bib14)\)\. Probabilistic calibration of classifier outputs is standard\(Platt,[1999](https://arxiv.org/html/2605.30826#bib.bib29); Niculescu\-Mizil and Caruana,[2005](https://arxiv.org/html/2605.30826#bib.bib30)\)\. Selective classification allows abstention on low confidence inputs\(Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2605.30826#bib.bib31)\), conformal prediction provides distribution\-free coverage guarantees\(Angelopoulos and Bates,[2021](https://arxiv.org/html/2605.30826#bib.bib32)\), and confident learning and dataset cartography use disagreement to identify possibly mislabeled examples\(Northcuttet al\.,[2021](https://arxiv.org/html/2605.30826#bib.bib34); Swayamdiptaet al\.,[2020](https://arxiv.org/html/2605.30826#bib.bib33)\)\. For single\-LLM uncertainty,Linet al\.\([2023](https://arxiv.org/html/2605.30826#bib.bib27)\)quantifies black\-box uncertainty via the semantic dispersion of sampled generations, andTianet al\.\([2023](https://arxiv.org/html/2605.30826#bib.bib28)\)elicits verbalized confidence from RLHF\-fine\-tuned LLMs\. In biomedical NLP,de Oliveiraet al\.\([2025](https://arxiv.org/html/2605.30826#bib.bib36)\)reports that out\-of\-the\-box LLM calibration is poor while post hoc calibration helps, andYuanet al\.\([2024](https://arxiv.org/html/2605.30826#bib.bib37)\)documents systematic overconfidence in the black\-box calibration of a closed LLM\. Our work treats multi\-LLM agreement as panel evidence of salience and learns a downstream scorer from agreement and candidate metadata\. It targets validation\-selected operating points for review triage rather than distribution\-free guarantees or automatic curation\.

Line of workUnit of predictionMain objectivePanel agr\.Triage scoreRelation to BioConCalBERN2\(Sunget al\.,[2022](https://arxiv.org/html/2605.30826#bib.bib35)\)Mention spanEnd\-to\-end NER and normalizationNoNoCandidate sourceBiomedical BERT and PubMedBERT NER\(Leeet al\.,[2020](https://arxiv.org/html/2605.30826#bib.bib6); Guet al\.,[2021](https://arxiv.org/html/2605.30826#bib.bib7)\)Token and spanSupervised biomedical NERNoNoCandidate sourceExtractor\-level NER evaluationModel or spanRank or score standalone extractorsNoNoDifferent layerMulti\-LLM consensus annotation\(Yuanet al\.,[2025](https://arxiv.org/html/2605.30826#bib.bib26)\)Item labelConsensus plus human reviewYes \(count\)NoRelated, count onlyBioConCal \(this work\)Panel\-surfaced candidateValidation\-targeted candidate triageYesYesThis workTable 6:Evaluation units across related settings\. BioConCal conditions on surfaced candidates and evaluates candidate\-level triage rather than end\-to\-end extraction\. “Panel agr\.” is whether the method uses multi\-LLM panel agreement structure, and “Triage score” is whether it emits a validation\-targeted candidate\-correctness score\.ModelAccessType and sizeGPT\-5\.5Closed APIfrontier, undisclosedClaude Opus 4\.7Closed APIfrontier, undisclosedLlama 3\.3 70BLocalopen\-weight dense, 70BQwen3\-30B\-A3B\-InstructLocalopen\-weight MoE, 30\.5B/3\.3B activeDeepSeek\-R1 32BLocalopen\-weight reasoning, 32BMiniMax\-M2Localopen\-weight MoE, 230B/10B activeMistral\-Small 3\.2 24BLocalopen\-weight dense, 24BPhi\-4Localopen\-weight dense, 14BTable 7:Eight\-model panel\. Six models are served locally with vLLM\.ModelPRF1OpenAI GPT\-5\.50\.7880\.8120\.800Claude Opus 4\.70\.8270\.6290\.715MiniMax\-M20\.6650\.4330\.525Mistral\-Small 3\.20\.6590\.3860\.487Qwen3\-30B\-A3B0\.6120\.3610\.454DeepSeek\-R1 32B0\.7430\.3190\.446Phi\-40\.6660\.3070\.420Llama 3\.3 70B0\.7310\.2830\.408Table 8:Single\-model normalized mention F1, overall micro on the four main datasets \(excludes CHEMDNER\)\.RulePRF1Union, any model0\.5730\.8800\.694OpenAI∪\\cupClaude0\.7630\.8540\.806Agreement≥2\\geq 20\.7110\.7530\.731Agreement≥4\\geq 40\.8050\.4280\.559Agreement≥6\\geq 60\.8780\.2630\.405Unanimous \(k=8k=8\)0\.9400\.1150\.205Table 9:Voting baselines on the 8\-model panel\. The closed\-only union is the best F1 voting rule, but unanimous voting trades almost all recall for precision\.

## Appendix CBERN2 NER reference point

We run the BERN2 NER model \(the recognition component of BERN2, without entity normalization\) on the same 500\-document\-per\-dataset samples, using the officialmulti\_nermodule with the releaseddmis\-lab/bern2\-nerweights\. BERN2 heads map to our schema as disease, drug to chemical, gene, DNA, RNA, cell line, and cell type, while species and mutation are ignored\. BERN2 merges gene and protein into one head, so BC5CDR, NCBI Disease, and BC2GM use clean mappings while JNLPBA protein is approximate and reported separately\. Table[10](https://arxiv.org/html/2605.30826#A3.T10)reports normalized mention P/R/F1 and Table[11](https://arxiv.org/html/2605.30826#A3.T11)the candidate\-source overlap\. Run details are inbern2\_run\_report\.md\.

DatasetPRF1BC5CDR0\.8740\.7720\.820NCBI Disease0\.8690\.8750\.872BC2GM0\.8640\.8680\.866Overall \(clean\)0\.8730\.7820\.825JNLPBA \(approx\.\)0\.5090\.7400\.603Table 10:BERN2 NER model, normalized mention matching on the sampled documents\. Overall is over BC5CDR, NCBI Disease, and BC2GM with clean type mappings\. JNLPBA is approximate because BERN2 merges gene and protein\. Source:tables/bern2\_reference\_results\.csv\.DatasetGoldPanel coveredBERN2 coveredPanel ceilingPanel\+BERN2 ceilingBC5CDR9,9659,1178,3070\.9150\.939NCBI Disease5064604460\.9090\.953BC2GM6364995560\.7850\.936JNLPBA1,1401,0468480\.9180\.930Table 11:Candidate\-source overlap\. “Panel covered” and “BERN2 covered” are gold mentions whose normalized key is surfaced by each source\. Adding BERN2 to the panel raises the candidate\-universe ceiling, most on BC2GM genes\. Source:tables/bern2\_candidate\_source\_overlap\.csv\.Table[12](https://arxiv.org/html/2605.30826#A3.T12)decomposes the pipeline into candidate\-generation ceiling and triage yield on the same document\-level test fold\. The eight\-LLM panel reaches strict multiset ceiling 0\.856, BERN2 alone reaches 0\.737, and the additive generator union reaches 0\.901 by surfacing gold mentions the panel misses\. The union row in Table[12](https://arxiv.org/html/2605.30826#A3.T12)is an exploratory merge rule, not a learned scorer over a shared representation\. Table[13](https://arxiv.org/html/2605.30826#A3.T13)therefore adds BioConCal\-Hybrid, a learned GBT over the merged LLM\-8∪\\cupBERN2 universe with LLM agreement features where available, BERN2 source indicators, and BERN2 confidence scores\.

Candidate sourceCandsRepr\.CeilingSrcScorerAUROCP@0\.95RcandRcorpusgoldF1LLM\-8 only386620570\.856–BioConCal0\.9100\.9390\.5920\.522BERN2 only219917720\.7370\.770take\-alln/a0\.806n/a0\.737LLM\-8∪\\cupBERN2606521660\.901–mergen/a0\.836n/a0\.550Diverse\-3 only296812070\.502–BioConCal0\.8800\.9440\.4900\.269Diverse\-3∪\\cupBERN2516719740\.821–mergen/a0\.680n/a0\.567Table 12:Candidate\-generator comparison on the document\-level test fold \(seed 42\)\. The generator fixes the candidate\-universe*ceiling*\(Repr\. gold==multiset gold occurrences covered; ceiling==Repr\./2404/2404\); BioConCal determines*triage yield*\(AUROC, P@0\.95, Rcand, Rcorpus\) within that universe\. Scorers are not forced to match: a single extractor has no agreement features, so BERN2\-only is take\-all \(no triage\) and union rows use an exploratory merge rule\. “Src F1” is take\-all extractor F1 on the test fold \(single extractors only; “–” for panels, which have no single operating point\)\. Rcorpusis comparable across all rows; Rcandis within\-universe and reported only for BioConCal\-scored panels\. JNLPBA gene/protein mapping is approximate\. Source:scripts/main\_candidate\_generator\_comparison\.py,tables/main\_candidate\_generator\_comparison\.csv\.WorkflowCand\. sourceCandsRepr\.CeilingAUROCAUPRCTest PSel\.Row TPFPRcorpusFitLLM\-8 BioConCalLLM\-83,8662,0570\.8560\.910n/a0\.9391,3401,258820\.522PrimaryBERN2 take\-allBERN22,1991,7720\.737n/an/a0\.8062,1991,7724270\.737UnfilteredLLM\-8∪\\cupBERN2 mergeadditive union6,0652,1660\.901n/an/a0\.8361,5831,3232600\.550UnfilteredBioConCal\-Hybridmerged union4,1402,1500\.8940\.9350\.9480\.9291,8471,7161310\.709Tradeoff

Table 13:Learned hybrid candidate scoring on the document\-level test fold\. BioConCal\-Hybrid trains a GBT over a merged LLM\-8∪\\cupBERN2 candidate universe; BERN2\-only rows have zero LLM votes and source indicators mark them as BERN2\-only, while overlap rows retain LLM features and receive BERN2 indicators\. Row TP counts selected candidate rows labeled positive\. Rcorpususes multiset corpus accounting; for the hybrid row, 1,705 selected multiset gold mentions over 2,404 corpus gold mentions gives R=corpus0\.709\{\}\_\{\\mathrm\{corpus\}\}=0\.709\. The hybrid reduces the false\-positive burden relative to the exploratory merge and recovers more true candidate rows than LLM\-only BioConCal, but its empirical precision 0\.929 is lower than the primary LLM\-8 BioConCal precision of 0\.939\. This is a candidate\-triage comparison, not a claim that BioConCal is a better extractor than BERN2\. Source:scripts/train\_hybrid\_candidate\_scorer\.py,tables/hybrid\_candidate\_scorer\_results\.csv\.
## Appendix DFeature definitions

FeatureGroupDefinitionagreement\_countAgreementNumber of providers \(of 8\) emitting the same\(normalized text,type\)\(\\text\{normalized text\},\\text\{type\}\)key\.closed\_agreement\_countAgreementSubcount over closed\-API providers \(OpenAI, Claude\)\.open\_agreement\_countAgreementSubcount over six open\-weight providers\.model\_family\_diversity\_scoreAgreementDistinct families among agreeing providers \(six families\)\.mention\_length\_charsMentionCharacter length of predicted surface\.mention\_token\_countMentionWhitespace\-token count of predicted surface\.mention\_occurrences\_in\_docMentionLiteral occurrences of predicted surface in the source document\.has\_\*indicatorsMentionDash, slash, paren, digit, Greek letter, or capitalized abbreviation in the surface\.exact\_surface\_in\_docSurface availability1 if predicted surface is a literal substring of the document\.normalized\_in\_docSurface availability1 if normalized surface is a substring of normalized document\.offset\_available\_countSurface availabilityProviders that supplied a numeric offset\.offset\_ambiguity\_countSurface availabilityDistinct\(start,end\)\(\\text\{start\},\\text\{end\}\)tuples across providers\.doc\_text\_lengthDocumentCharacter length of source document\.entity\_typeEntityUnified eight\-type schema, one\-hot encoded\.Table 14:BioConCal gold\-free input\-feature definitions\. Training labels come from gold annotations and are not listed here\. Per\-feature permutation importance is in Figure[4](https://arxiv.org/html/2605.30826#A7.F4)\.
## Appendix EAuto\-suggested audit sheets

We release four candidate audit sheets\. Set A is the unanimous false\-positive list \(90 candidates under multiset normalized matching\)\. Set B is high\-agreement false positives withk∈\{6,7\}k\\in\\\{6,7\\\}\(358 candidates\)\. Set C is minority true positives withk≤2k\\leq 2\(200 sampled\)\. Set D is closed\-versus\-open disagreement \(200 sampled\)\. All sheets are CSV with auto\-suggested category labels and are released to support follow\-up annotation review; the categories have not been manually verified\. The final Set A source is the revision fileunanimous\_fp\_audit\_final\.csv; older combined audit artifacts are not used for the final 90\-candidate count\. A 100\-row span\-alignment audit sample stratified across repeated\-surface, multi\-candidate\-span, long\-or\-nested, overlap, and random cases is inaudit/span\_alignment\_manual\_audit\_sample\.csv\.

## Appendix FSpan\-alignment method success rates

Alignment methodTotalWith goldPost=GoldRateExact surface match12,62812,4029,6370\.777Table 15:Span\-alignment method success rates on OpenAI predictions across the four main datasets\. Source:tables/span\_alignment\_error\_modes\_v2\.csv\.
## Appendix GFull feature ablation

FeaturesClf\.\#feat\.AUROCBrierECEP@95Rcand@95SelA\. agreement\_count onlylogistic10\.7530\.2050\.0770\.9520\.131293A\. agreement\_count onlyGBT10\.7510\.1990\.0500\.9520\.131293B\. vote\-pattern only \(8 indicators\)logistic80\.8640\.1480\.0590\.9410\.151341B\. vote\-pattern only \(8 indicators\)GBT80\.8590\.1510\.0530\.9530\.143318C\. aggregate agreement only \(4 feats\)logistic40\.8610\.1480\.0560\.9520\.131293C\. aggregate agreement only \(4 feats\)GBT40\.8600\.1490\.0540\.0000\.0000D\. non\-agreement only \(15 feats\)logistic150\.8290\.1690\.0540\.9130\.158367D\. non\-agreement only \(15 feats\)GBT150\.8650\.1510\.0430\.9520\.02863E\. agg\. agree\. \+ non\-agree\.logistic190\.8910\.1350\.0370\.9310\.5391,229E\. agg\. agree\. \+ non\-agree\.GBT190\.9100\.1190\.0320\.9390\.5921,340F\. vote\-pattern \+ non\-agreementlogistic230\.8920\.1340\.0360\.9300\.5451,245F\. vote\-pattern \+ non\-agreementGBT230\.9080\.1200\.0370\.9390\.5541,253G\. all\-feature diagnosticlogistic270\.8920\.1340\.0370\.9280\.5401,237G\. all\-feature diagnosticGBT270\.9100\.1190\.0260\.9460\.5571,251Table 16:Complete feature\-ablation grid \(logistic \+ GBT, all seven feature sets\)\. Row E is the main 19\-feature BioConCal definition used in the paper; row G is an all\-feature diagnostic that additionally includes the 8 per\-model vote indicators\. Row C\-GBT \(aggregate\-agreement\-only, GBT\) selects zero candidates at the val\-frozen P95 threshold because the GBT predicted\-probability distribution on aggregate\-agreement\-only features has no validation point above the 0\.95 precision target; this is a quirk of finite\-sample threshold selection on a low\-dimensional GBT and is included for completeness\. Source:tables/feature\_ablation\_baselines\.csv\.![Refer to caption](https://arxiv.org/html/2605.30826v1/x3.png)Figure 3:Feature\-ablation AUROC \(left\) and recall at the validation\-frozen P95 threshold \(right\) for GBT and logistic classifiers\.![Refer to caption](https://arxiv.org/html/2605.30826v1/x4.png)Figure 4:Permutation feature importance for BioConCal\-GBT, doc\-level validation fold \(mean drop in negative log\-loss, five shuffles\)\. Left: top\-20 individual features\. Right: grouped importance by feature family\.
## Appendix HRobustness: splits, bootstrap, and candidate keying

### H\.1Per\-seed and bootstrap detail

Table[18](https://arxiv.org/html/2605.30826#A8.T18)gives the per\-seed results for the headline scorer behind the summary in Table[17](https://arxiv.org/html/2605.30826#A8.T17)\. Table[19](https://arxiv.org/html/2605.30826#A8.T19)gives the document\-level bootstrap 95% confidence intervals on the seed\-42 test fold \(2,000 resamples of documents, stratified by dataset; documents rather than candidates are resampled because within\-document candidates are correlated\)\. Both are produced without rerunning extraction, fromcandidate\_master\.csv\.

ScorerAUROCP@0\.95Rcand@0\.95Sel\.FPRaw agreement count0\.742±\\pm0\.0100\.931±\\pm0\.0340\.149±\\pm0\.044356±\\pm11428±\\pm23Isotonic on agreement0\.741±\\pm0\.0100\.931±\\pm0\.0340\.149±\\pm0\.044356±\\pm11428±\\pm23Vote\-vector logistic0\.852±\\pm0\.0110\.898±\\pm0\.0770\.043±\\pm0\.076102±\\pm1769±\\pm16Aggregate\-agreement scorer0\.846±\\pm0\.0100\.935±\\pm0\.0290\.144±\\pm0\.035340±\\pm8824±\\pm17Non\-agreement scorer0\.875±\\pm0\.0100\.953±\\pm0\.0140\.373±\\pm0\.144870±\\pm34942±\\pm25BioConCal logistic0\.895±\\pm0\.0100\.948±\\pm0\.0170\.471±\\pm0\.0491096±\\pm11659±\\pm24BioConCal GBT0\.907±\\pm0\.0100\.944±\\pm0\.0130\.565±\\pm0\.0641321±\\pm16175±\\pm25Table 17:Robustness over 10 document\-level splits \(seeds 13, 21, 42, 87, 101, 7, 123, 256, 512, 1000\), mean±\\pmsd\. Same protocol, gold\-free features, and validation\-targeted 0\.95 precision as Table[3](https://arxiv.org/html/2605.30826#S5.T3); each seed’s recall denominators are recomputed from its own test fold\. Source:scripts/repeated\_docsplit\_bioconcal\.py,tables/repeated\_docsplit\_summary\.csv\.SeedAUROCP@0\.95RcandRcorpusSel\.FP70\.9150\.9350\.6350\.579147096130\.9260\.9730\.5210\.479113431210\.9070\.9480\.5080\.459124465420\.9100\.9390\.5920\.523134082870\.9140\.9380\.6470\.5861514941010\.9030\.9530\.5190\.4641204571230\.9060\.9530\.4750\.4381158552560\.9000\.9280\.6540\.59215781135120\.8900\.9440\.5280\.47511776610000\.9000\.9320\.5740\.515139294mean±\\pmsd0\.907±\\pm0\.0100\.944±\\pm0\.0130\.565±\\pm0\.0640\.511±\\pm0\.0571321±\\pm16175±\\pm25Table 18:Per\-seed BioConCal\-GBT results over the 10 document\-level splits\. Source:scripts/repeated\_docsplit\_bioconcal\.py,tables/repeated\_docsplit\_all\_seeds\.csv\.ScorerAUROC \[95% CI\]P@0\.95 \[95% CI\]Rcand\[95% CI\]Raw agreement count0\.753 \[0\.732, 0\.776\]0\.952 \[0\.928, 0\.974\]0\.131 \[0\.112, 0\.151\]Vote\-vector logistic0\.864 \[0\.844, 0\.884\]0\.941 \[0\.914, 0\.965\]0\.151 \[0\.131, 0\.174\]Aggregate\-agreement scorer0\.861 \[0\.840, 0\.880\]0\.952 \[0\.928, 0\.973\]0\.131 \[0\.111, 0\.153\]Non\-agreement scorer0\.865 \[0\.844, 0\.885\]0\.952 \[0\.849, 1\.000\]0\.028 \[0\.014, 0\.044\]BioConCal logistic0\.891 \[0\.871, 0\.910\]0\.931 \[0\.895, 0\.961\]0\.539 \[0\.499, 0\.578\]BioConCal GBT0\.910 \[0\.892, 0\.928\]0\.939 \[0\.908, 0\.966\]0\.592 \[0\.559, 0\.626\]Table 19:Document\-level bootstrap 95% confidence intervals on the seed\-42 test fold \(2,000 resamples\)\. The BioConCal\-GBT AUROC interval does not overlap the raw\-agreement interval\. Source:scripts/bootstrap\_doclevel\_ci\.py,tables/bootstrap\_doclevel\_ci\.csv\.

## Appendix ICandidate keying ablation

Correctness keyCand\.Prev\.CeilingAUROCP@95RcandRcorpusSel\. \(FP\)Normalized text \+ type3,8660\.5490\.8830\.9100\.9390\.5920\.5231,340 \(82\)Exact surface \+ type3,8660\.5480\.8820\.9120\.9370\.6050\.5341,369 \(86\)Exact span \+ type3,8660\.0980\.1580\.8581\.0000\.0320\.00512 \(0\)Table 20:Candidate keying ablation on the document\-level 60/20/20 test fold\. The correctness key that labels a candidate as positive is varied while the candidate set and gold\-free features are held fixed, and BioConCal\-GBT is retrained for each key with a validation\-selected 0\.95 precision threshold\. The occurrence\-aware normalized row reproduces the canonical Full\-8 result\. Exact\-surface keying is nearly identical, while exact\-span keying collapses the positive set and the ceiling because LLM\-emitted offsets are unreliable, which is why exact localization is handled by deterministic alignment rather than as a scoring target\. Source:tables/candidate\_keying\_ablation\.csv\.### I\.1Candidate keying / granularity ablation

The candidate master is occurrence\-level: 18,812 rows over 15,611 distinct\(doc, type, normalized\_text\)keys on the four main datasets\. The table is a multiset, so repeated mentions and surface variants are kept as distinct rows and not silently merged\. If a document mentions*misoprostol*four times and the panel emits the surfaces*misoprostol*and*Misoprostol*, these become separate candidate rows that share the key\(misoprostol, chemical\)but differ in surface form and occurrence index; each carries its own surface features while agreement is computed per occurrence\. Boundary and granularity variants such as*IL\-2*versus*IL\-2 receptor*normalize to different keys and remain distinct candidates\. Across the four main datasets, roughly one row in six is a repeated\-mention or surface\-variant occurrence rather than a new normalized identity\.

Table[21](https://arxiv.org/html/2605.30826#A9.T21)varies the candidate key on the document\-level test fold \(seed 42\) and re\-trains BioConCal\-GBT\. Variants A and B collapse to one row per normalized key and per surface key respectively \(representative occurrence with the highest agreement count; gold\-positive if any occurrence is\); C is the occurrence\-aware table actually used; D scores the same rows against the relaxed\-overlap label\. Coverage is occurrence\-level against the same corpus gold \(Gcorpus=2,404G\_\{\\mathrm\{corpus\}\}=2\{,\}404\)\. Variant C reproduces the canonical Full\-8 numbers \(AUROC 0\.910, selected 1,340\), a no\-fabrication check\. AUROC and the lift over raw agreement are stable across keys, so the triage conclusion does not depend on the keying scheme; what changes is the occurrence\-level ceiling \(collapsing repeated mentions lowers recoverable gold occurrences from 0\.856 to 0\.470\), a reporting\-granularity effect, not a scorer failure\. Source:scripts/candidate\_keying\_ablation\.py,tables/candidate\_keying\_ablation\.csv\.

Candidate keyCandsPrev\.CeilingAUROCRaw AUROCP@0\.95A\. normalized\-collapsed121230\.4710\.4700\.9080\.8360\.958B\. surface key125800\.4830\.5050\.9110\.8330\.945C\. occurrence\-aware \(current\)188120\.5930\.8560\.9100\.7530\.939D\. relaxed\-overlap label188120\.7580\.8560\.8620\.7450\.943

Table 21:Candidate keying / granularity ablation\. Variant C is the occurrence\-aware canonical setting\.

## Appendix JFull LODO failure analysis

Held\-outTypes in testUnseen\-in\-trainPrev\.RawGBTLog\.P@95Rcand@95AUROCAUROCAUROCtesttestBC5CDRchem\., diseasechem\.0\.6530\.7370\.7810\.8461\.0000\.001NCBI Diseasedisease—0\.5060\.8690\.9100\.9190\.9790\.100BC2GMgenegene0\.4060\.7660\.8570\.8280\.7710\.415JNLPBAprotein, DNA, RNA, cell line, cell typeDNA, RNA, cell line, cell type0\.3990\.7940\.8530\.8370\.8890\.165Table 22:Full LODO per\-fold table, separating ranking transfer \(raw and BioConCal AUROC\) from operating\-point transfer \(P@95 and Rcand@95 from a source\-selected threshold\)\. “Unseen\-in\-train” lists entity types in the held\-out fold but not in training\. Sources:tables/lodo\_ranking\_vs\_threshold\_transfer\.csv,tables/lodo\_failure\_analysis\.csv\.#### Deployment checklist\.

Before applying BioConCal to a new corpus we recommend the following steps\. \(1\) Verify that the target entity types are represented in the validation data\. \(2\) Estimate the target\-domain candidate positive prevalence\. \(3\) Select operating thresholds on target\-domain validation documents\. \(4\) Report empirical precision on a held\-out target audit set before use\. \(5\) Use scores for triage only, not for automatic database insertion\. \(6\) If target entity types are unseen in training or calibration, treat a source\-selected 0\.95\-precision threshold as invalid\.

## Appendix KLODO target\-domain recalibration

Held\-outnc​a​ln\_\{cal\}Test P@95Test Rcand@95Sel\.NotesBC5CDR01\.0000\.0015source\-val \(canonical v3 LODO\)BC5CDR250\.859±\\pm0\.1000\.004±\\pm0\.00345target\-cal, 3 seedsBC5CDR500\.828±\\pm0\.0630\.005±\\pm0\.00346target\-cal, 3 seedsBC5CDR1000\.862±\\pm0\.0900\.003±\\pm0\.00230target\-cal, 3 seedsNCBI Disease00\.9790\.10047source\-val \(canonical v3 LODO\)NCBI Disease250\.924±\\pm0\.0070\.682±\\pm0\.033319target\-cal, 3 seedsNCBI Disease500\.904±\\pm0\.0300\.719±\\pm0\.086323target\-cal, 3 seedsNCBI Disease1000\.890±\\pm0\.0370\.737±\\pm0\.105291target\-cal, 3 seedsBC2GM00\.7710\.415280source\-val \(canonical v3 LODO\)BC2GM250\.830±\\pm0\.0230\.079±\\pm0\.08047target\-cal, 3 seedsBC2GM500\.555±\\pm0\.3930\.019±\\pm0\.01410target\-cal, 3 seedsBC2GM1000\.544±\\pm0\.3840\.019±\\pm0\.0149target\-cal, 3 seedsJNLPBA00\.8890\.165199source\-val \(canonical v3 LODO\)JNLPBA250\.875±\\pm0\.0080\.296±\\pm0\.099345target\-cal, 3 seedsJNLPBA500\.867±\\pm0\.0090\.240±\\pm0\.132268target\-cal, 3 seedsJNLPBA1000\.903±\\pm0\.0280\.066±\\pm0\.04564target\-cal, 3 seedsTable 23:LODO with target\-domain recalibration\. Thenc​a​l=0n\_\{cal\}\{=\}0rows are canonical source\-validation LODO results;nc​a​l≥25n\_\{cal\}\\geq 25rows reuse the same source\-trained model, select the P95 threshold on target\-domain calibration documents, and evaluate on the remaining target documents\. Mean±\\pmsd over three seeds\. Source:tables/lodo\_target\_recalibration\_summary\.csv\.
## Appendix LGain decomposition

PanelFeaturesAUROCP@95Rfull@95Best openNA0\.8800\.9330\.262Closed\-2NA0\.7940\.9550\.160Closed\-2NA\+AG0\.8060\.9430\.286Open\-6NA0\.8990\.9440\.331Open\-6NA\+AG0\.9080\.9580\.335Full\-8NA0\.8650\.9520\.028Full\-8AG0\.8730\.9510\.387Full\-8NA\+AG0\.9100\.9390\.592Table 24:Gain decomposition on the document\-level 60/20/20 test fold\. NA is the non\-agreement feature block; AG is the agreement\-structure block\. Source:tables/gain\_decomposition\.csv\.
## Appendix MFull cheap\-panel baseline table

Panel\|M\|\|M\|Cand\.Prev\.AUROCAUPRCBrierECESel\.FPTest PCand\. RCand\. CovOpenAI112,6280\.7930\.8150\.9370\.1340\.05736860\.9840\.1770\.141Claude19,3240\.8310\.8140\.9450\.1160\.065604180\.9700\.3730\.317Best open17,9790\.6740\.8800\.9160\.1340\.062596400\.9330\.5160\.352Closed\-2213,6980\.7690\.8060\.9150\.1400\.063644370\.9430\.2780\.224Diverse\-3315,1560\.6870\.8800\.9260\.1310\.0461,003560\.9440\.4900\.338Open\-6614,4740\.5640\.9080\.9140\.1250\.055742310\.9580\.4580\.247Full\-8818,8120\.5930\.9100\.9230\.1190\.0321,340820\.9390\.5920\.347Table 25:Full cheap\-panel baseline results on the document\-level 60/20/20 test fold\. Prev\. is positive prevalence in the panel candidate universe\. Sel\., FP, and Test P are at the validation\-selected P95 operating point\. Test P is empirical test precision\. Cand\. R and Cand\. Cov are candidate recall and candidate coverage over the panel’s own universe\. The main\-text Table[4](https://arxiv.org/html/2605.30826#S5.T4)reports Rfull@95, which is recall over the full eight\-panel positive candidate set\. Source:tables/single\_model\_candidate\_baselines\_p95\.csv\.
## Appendix NEnd\-to\-end interpretation

MethodSel\.FPC\-PC\-RCorp\-RUnion \(any model\)3,8661,7420\.5491\.0000\.883OpenAI∪\\cupClaude2,7437430\.7290\.9420\.832Agree≥\\geq22,5877970\.6920\.8430\.745Agree≥\\geq41,2752600\.7960\.4780\.422Agree≥\\geq6715930\.8700\.2930\.259Unanimous \(k=8k\{=\}8\)293140\.9520\.1310\.116BioConCal P951,340820\.9390\.5920\.523Table 26:End\-to\-end interpretation on the doc\-level 60/20/20 test fold\. C\-P is candidate precision\. C\-R is candidate recall \(selected TP over 2,124 gold\-positive candidates\)\. Corp\-R is corpus\-level recall \(selected TP over 2,404 gold mentions in the test corpus\)\. The empirical union candidate recall is 0\.883, which caps Corp\-R for any candidate\-level method\. Source:tables/end\_to\_end\_interpretation\.csv\.WorkflowCand\. sourceTest PTPFPSel\.RcorpusTriage fitMain interpretationUnanimous / raw agreement P95LLM\-80\.952279142930\.116Low yieldHigh precision, but too conservative for review yieldAgreement≥\\geq2LLM\-80\.6921,7907972,5870\.745Low precisionHigh raw recall, but 797 FPs make the queue noisyBERN2 take\-allBERN20\.8061,7724272,1990\.737UnfilteredHigher coverage, but no learned candidate filterLLM\-8∪\\cupBERN2 exploratory mergeadditive union0\.8361,3232601,5830\.550UnfilteredRaises ceiling, but unfiltered additions add FPsBioConCal GBTLLM\-80\.9391,258821,3400\.522PrimaryLargest primary high precision review yieldBioConCal\-Hybridmerged union0\.9291,7161311,8470\.709TradeoffHigher coverage, but lower precision than LLM\-8 BioConCalTable 27:Curator triage utility under high precision review constraints on the document level test fold\. Rows compare fixed voting, unfiltered source expansion, and learned scoring workflows by empirical precision and selected yield\. The primary BioConCal operating point is selected on validation at the 0\.95 precision target\. TP denotes selected candidate row positives\. Hybrid Rcorpususes multiset corpus accounting, with 1,705 selected gold occurrences over 2,404 corpus gold mentions\. BioConCal GBT is the primary LLM\-8 review queue\. BioConCal\-Hybrid scores the expanded LLM\-8∪\\cupBERN2 universe and trades precision for coverage\. This is a curator triage comparison, not an end\-to\-end extraction\. Data sources aretables/risk\_control\_selection\_docsplit\.csv,tables/end\_to\_end\_interpretation\.csv,tables/main\_candidate\_generator\_comparison\.csv, andtables/hybrid\_curator\_utility\.csv\.
## Appendix OReliability diagram

![Refer to caption](https://arxiv.org/html/2605.30826v1/x5.png)Figure 5:Reliability diagram on the document\-level 60/20/20 test fold\.
## Appendix PAgreement\-count calibration curve

![Refer to caption](https://arxiv.org/html/2605.30826v1/x6.png)Figure 6:Per\-mentionP​\(correct∣k\)P\(\\text\{correct\}\\mid k\)as a function of agreement countkkon the 8\-model panel\. Precision rises monotonically from 0\.266 atk=1k\{=\}1to 0\.940 atk=8k\{=\}8\.
## Appendix QFull selective and conformal baselines

MethodSel\.FPTest PTest RP gapRaw agreement \(val P95\)293140\.9520\.131\+0\.002\+0\.002Conformal\-style raw cutoff \(α=0\.05\\alpha\{=\}0\.05\)715930\.8700\.293−0\.080\-0\.080Platt scaling onkk293140\.9520\.131\+0\.002\+0\.002Isotonic onkk293140\.9520\.131\+0\.002\+0\.002BioConCal GBT \(val P95\)1,340820\.9390\.592−0\.011\-0\.011Conformal\-style BioConCal cutoff \(α=0\.05\\alpha\{=\}0\.05\)1,362830\.9390\.602−0\.011\-0\.011Table 28:Selective and split\-conformal\-inspired baselines on the document\-level 60/20/20 split\. “P gap” is the empirical test precision minus the 0\.95 nominal target\. Source:tables/selective\_baselines\.csv\. These are comparison baselines, not formal finite\-sample precision guarantees\.
## Appendix RPanel composition ablation

PanelNcand\{\}\_\{\\text\{cand\}\}AUROCBrierECEP@95Rcand@95All 8 models18,8120\.9100\.1190\.0320\.9390\.592Open\-weight 614,4740\.9080\.1420\.0690\.9580\.458Closed 2 \(OpenAI\+Claude\)13,6980\.8060\.2110\.1170\.9430\.278Drop OpenAI16,9770\.9110\.1210\.0380\.9370\.632Drop Claude18,1740\.9090\.1240\.0390\.9330\.574Table 29:Panel\-composition ablation of BioConCal\-GBT under the document\-level 60/20/20 split\. Agreement features are recomputed from each panel subset; candidates with zero in\-panel votes are removed\. The closed\-only two\-model panel drops AUROC to 0\.806, consistent with the importance of model diversity\. The full leave\-one\-model\-out grid is stable in the 0\.899–0\.912 AUROC range and is intables/panel\_composition\_ablation\.csv\.
## Appendix SUnanimous false\-positive audit \(auto\-suggested\)

Auto\-suggested categoryCount%Confirmed false positive00\.0Boundary mismatch6673\.3Type confusion77\.8Alias / synonym not in gold1314\.4Broader / narrower concept00\.0Possible gold omission33\.3Multiset overflow11\.1Total90100\.0Table 30:Auto\-suggested category distribution for the 90 unanimous \(8\-model\) false positives\. Categories are assigned by a rule\-based heuristic and have not been manually verified\. “Multiset overflow” is the case where the panel emitted the same\(normalized text,type\)\(\\text\{normalized text\},\\text\{type\}\)key more times than gold contained\. We also release a 50\-case stratified manual\-audit sample \(audit/unanimous\_fp\_manual\_sample\_50\.csv\) with blankmanual\_categoryandmanual\_notescolumns\.
## Appendix TFeature robustness ablation

Variant \(GBT\)\#feat\.AUROCΔ\\DeltaAUROCBrierP@95Rcand@95SelA\. BioConCal main model190\.910\+0\.000\+0\.0000\.1190\.9390\.5921,340B\. remove entity\_type180\.906−0\.004\-0\.0040\.1220\.9400\.5591,264C\. remove doc\_text\_length180\.909−0\.001\-0\.0010\.1190\.9420\.5491,238D\. remove surface availability150\.910\+0\.000\+0\.0000\.1200\.9350\.5991,361E\. remove offset only170\.910\+0\.000\+0\.0000\.1200\.9350\.5991,361F\. remove doc length \+ surface avail\.140\.912\+0\.002\+0\.0020\.1170\.9340\.6001,364G\. agreement \+ mention only130\.909−0\.001\-0\.0010\.1200\.9410\.5281,193H\. agreement only40\.860−0\.050\-0\.0500\.149—0\.0000I\. vote\-pattern only80\.859−0\.051\-0\.0510\.1510\.9530\.143318Table 31:Feature\-removal robustness ablation under the doc\-level 60/20/20 split and val\-frozen P95 protocol\.Δ\\DeltaAUROC is relative to variant A, the main 19\-feature BioConCal model\. AUROC and Rcand@95 are stable across A–G \(max\|Δ​AUROC\|=0\.004\|\\Delta\\text\{AUROC\}\|=0\.004\); only stripping to agreement\-only \(H\) or vote\-pattern\-only \(I\) collapses precision\-targeted selection\. The non\-agreement features \(mention/surface/document\) are useful in\-domain but not the dominant signal, and the scorer is not driven by corpus\-specific formatting shortcuts\. Source:tables/feature\_robustness\_ablation\.csv\.
## Appendix UCurator workload trade\-off

MethodSel\.TPFPΔ\\DeltaTPΔ\\DeltaFPΔ\\DeltaFP/Δ\\DeltaTPReviewed/Δ\\DeltaTPUnanimous \(k=8k\{=\}8\)293279140000BioConCal GBT \(val P95\)1,3401,25882979680\.0691\.069Table 32:Curator workload trade\-off between Unanimous and BioConCal P95 on the doc\-level test fold\. “Δ\\DeltaTP” and “Δ\\DeltaFP” are gains over Unanimous; “Δ\\DeltaFP/Δ\\DeltaTP” is the marginal false\-positive cost per additional true candidate; “Reviewed/Δ\\DeltaTP” is the curator\-side review burden per recovered true positive\. Source:tables/curator\_workload\_tradeoff\.csv\.
## Appendix VLearned\-selector baselines

SelectorAUROCBrierP@95Rcand@95SelFPRaw agreement threshold0\.7530\.2370\.9520\.13129314Agreement\-count logistic0\.7530\.2050\.9520\.13129314Agreement\-count GBT0\.7510\.1990\.9520\.13129314Vote\-pattern logistic0\.8640\.1480\.9410\.15134120Vote\-pattern GBT0\.8590\.1510\.9530\.14331815Agreement\-family GBT0\.8600\.149——00Agreement \+ mention GBT0\.9090\.1200\.9410\.5281,19371BioConCal GBT \(main 19\-feature\)0\.9100\.1190\.9390\.5921,34082Table 33:Learned\-selector baselines under the same doc\-level 60/20/20 split and val\-targeted P95 protocol\. Vote\-pattern \(8 per\-model indicators\) and aggregate\-agreement \(4\-feature\) GBT classifiers both reach AUROC≈0\.86\\approx 0\.86but recover little precision\-targeted recall; the largest Rcand@95 jump comes from adding mention features\. The agreement\-family GBT row selects zero candidates because no val\-fold operating point of that classifier reaches the 0\.95 precision target\. Source:tables/learned\_selectors\_baseline\.csv\.![Refer to caption](https://arxiv.org/html/2605.30826v1/x7.png)Figure 7:Precision\-coverage curves on the document\-level 60/20/20 test fold\. BioConCal improves on raw agreement count across the coverage range\.
## Appendix WSpan alignment per selection rule

Selection ruleStagePRF1OpenAI alonebefore0\.0750\.0770\.076OpenAI aloneafter0\.7650\.7890\.777Claude alonebefore0\.1880\.1430\.162Claude aloneafter0\.8190\.6230\.708MiniMax\-M2 alonebefore0\.0190\.0130\.015MiniMax\-M2 aloneafter0\.5090\.3320\.402Unanimous \(k=8k=8\)after0\.9240\.1130\.201BioConCal @ P=0\.95after0\.6660\.1750\.277Table 34:Exact\-span F1 before and after deterministic span realignment\. Source:tables/span\_alignment\_results\.csv\.
## Appendix XOpen\-weight prompt sensitivity

We evaluated prompt sensitivity on a fixed 50\-document\-per\-dataset subset \(seed 42\) of the four main datasets\. The complete three\-model open\-weight panel used for the cross\-prompt agreement\-calibration analysis is Phi\-4 14B, Mistral\-Small 3\.2 24B, and MiniMax\-M2 230B; these three models ran all three prompts to completion\. Llama 3\.3 70B \(served via Ollama\) finished only the original prompt and is therefore partial; it contributes to the single\-model table for the original prompt but is excluded from the cross\-prompt agreement calibration to keep the panel size constant across prompts\. CHEMDNER is excluded from this analysis because the main paper already treats it as robustness\-only \(its public mirror does not provide reliable character offsets\)\. These results are not claimed as closed\-model robustness or as full eight\-model prompt sensitivity; closed\-API models \(OpenAI GPT\-5\.5 and Claude Opus 4\.7\) were not re\-run with prompt variants\. Three prompts are compared:*original\_zero\_shot*\(the main paper’s template\),*strict\_entity\_only*\(instructs the model to extract only explicit surface mentions and not infer entities\), and*few\_shot\_2\_examples*\(two synthetic illustrative examples that do not overlap the evaluation subset\)\.

PromptCandsP@1P@2P@maxAgr\. AUCoriginal\_zero\_shot7970\.2510\.5850\.8760\.778strict\_entity\_only6490\.3920\.7040\.9190\.742few\_shot\_2\_examples9130\.2410\.5890\.7760\.753Table 35:Open\-weight prompt sensitivity on the three\-model panel \(Phi\-4, Mistral\-Small 3\.2, MiniMax\-M2; max agreementk=3k\{=\}3\)\.P​\(correct∣k\)P\(\\text\{correct\}\\mid k\)is monotonic inkkfor all three prompts\. Strict prompting raises high\-agreement precision at the cost of recall; few\-shot prompting raises recall at the cost of P@maxkk\. Source:tables/prompt\_sensitivity\_summary\.csv\.![Refer to caption](https://arxiv.org/html/2605.30826v1/x8.png)Figure 8:Agreement calibration across the three prompt variants on the open\-weight 50\-doc\-per\-dataset subset\. Marker size scales with the number of candidates at each agreement count\.
## Appendix YPer\-dataset candidate coverage

DatasetGoldRepresentedCeilingUnion candsBC SelBC Corp\-RBC5CDR1,9391,6430\.8473,3981,2280\.594NCBI Disease108950\.880175470\.426BC2GM1301050\.808184130\.069JNLPBA2272140\.943558520\.225Test total2,4042,0570\.8563,8661,3400\.523Table 36:Per\-dataset candidate coverage on the doc\-level 60/20/20 test fold\. “Gold” is the total gold mentions in the test docs; “Represented” counts gold mentions whose\(normalized text,type\)\(\\text\{normalized text\},\\text\{type\}\)key appears in the candidate universe\. Two numbers can be reported here: the*stricter multiset gold\-key coverage*\(0\.856 total, where a gold mention only counts as represented if a candidate has the same key under multiset bookkeeping\), and the*empirical union candidate recall under candidate\-level matching*used in the main text \(0\.883 total\)\. BC \(BioConCal\) Corp\-R is selected TP / dataset gold mentions\. Source:tables/candidate\_universe\_ceiling\.csv\.
## Appendix ZPer\-dataset agreement calibration and normalization

![Refer to caption](https://arxiv.org/html/2605.30826v1/x9.png)Figure 9:Agreement calibration broken down by dataset\. BC5CDR and NCBI Disease reach 0\.96 and 0\.97 atk=8k\{=\}8\. BC2GM reaches 0\.83, JNLPBA reaches 0\.90\. Source:tables/agreement\_calibration\_by\_type\.csvandfigures/figure\_calibration\_by\_dataset\.pdf\.Per\-unified\-type agreement calibration is released astables/agreement\_calibration\_by\_type\.csv\. The normalization ablation comparing exact\-surface, normalized, and relaxed\-overlap matching is intables/normalization\_ablation\.csv\. The mean F1 lift from exact\-surface to normalized matching is small across all eight models\. The lift from normalized to relaxed\-overlap is larger but does not change the model ranking, so the central findings hold under either matching variant\.