From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

# From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
Source: [https://arxiv.org/html/2607.06452](https://arxiv.org/html/2607.06452)
\\copyrightclause

Copyright for this paper by its authors\. Use permitted under Creative Commons License Attribution 4\.0 International \(CC BY 4\.0\)\.

\\conference

CLEF 2026 Working Notes, 21 – 24 September 2026, Jena, Germany

\[orcid=0009\-0009\-7096\-6410, email=nrbsld@korea\.ac\.kr, \]\\fnmark\[1\]

\[orcid=0009\-0008\-3165\-6973, email=eunhalee@korea\.ac\.kr, \]\\fnmark\[1\]

\[orcid=0009\-0005\-6216\-3096, email=dnjswnswkd03@mju\.ac\.kr, \]\\fnmark\[1\]

\[orcid=0009\-0005\-8354\-666X, email=sohyunjung@korea\.ac\.kr, \]\\fnmark\[1\]

\[orcid=0009\-0003\-1675\-8414, email=goodjungjun@korea\.ac\.kr, \]\\fnmark\[1\]

\[email=kangj@korea\.ac\.kr, \]\\cormark\[1\]

\\fntext

\[1\]These authors contributed equally to this work\.\\cortext\[1\]Corresponding author\.

Eunha LeeWonjune JangSohyun ChungJunha JungJaewoo KangDepartment of Computer Science and Engineering, Korea University, Seoul, 02841, Republic of KoreaDepartment of Mathematics, Myongji University, Yongin, 17058, Republic of KoreaAIGEN Sciences, Seoul, 04778, Republic of Korea

\(2026\)

###### Abstract

Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents\. This study presents a question\-type\-specific large language model \(LLM\) framework for BioASQ 14b Task B, designed to improve answer robustness and evidence grounding in biomedical question answering\. Rather than applying a single prompting strategy to all questions, the framework selects different inference procedures for yes/no, factoid, and list questions according to their distinct reasoning and evaluation requirements\. For yes/no questions, snippet shuffling and self\-reflection are used to reduce sensitivity to evidence ordering and improve decision stability\. For factoid questions, full\-snippet input is combined with chain\-of\-thought\-based in\-context learning to support accurate biomedical entity identification\. For list questions, a multi\-agent architecture is employed, in which evidence extraction, candidate generation, answer verification, and final aggregation are handled collaboratively\. Preliminary experiments on BioASQ 13b were used to identify effective inference strategies for each question type, and the resulting framework was subsequently evaluated in the official BioASQ 14b Task B challenge\. In the official evaluation, our framework showed competitive performance across multiple batches and achieved first place in the factoid subtask of Batch 4\. These results demonstrate the effectiveness of combining question\-type\-specific inference, ensemble prediction, and agent\-based verification for reliable biomedical question answering\.

###### keywords:

BioASQ 14b\\sepLLM\\sepMulti\-Agent System\\sepChain\-of\-Thought\\sepPrompt Engineering\\sep

## 1Introduction

Biomedical literature is expanding at a pace that makes it increasingly difficult for researchers and healthcare professionals to identify precise and reliable answers to specialized questions\. Important evidence is often distributed across multiple articles, expressed using heterogeneous terminology, or reported in partially complementary and sometimes conflicting contexts\. As a result, biomedical QA systems cannot rely on document retrieval alone\. They must read evidence in context, connect findings scattered across snippets, and return answers that are concise enough for evaluation while still being grounded in the source literature\. The BioASQ challenge has served as a major benchmark for advancing such capabilities by evaluating systems on biomedical semantic indexing and question answering tasks\[Tsatsaronis2015BioASQ,nentidis2026bioasq14\]\. In particular, BioASQ Task B requires systems to generate exact and ideal answers from evidence snippets extracted from biomedical literature, making it a suitable testbed for evaluating evidence\-grounded reasoning in large language models \(LLMs\)\[nentidis2026bioasq14,Krithara2023BioASQQA\]\.

BioASQ Task B includes several answer formats, among which yes/no, factoid, and list questions pose distinct reasoning challenges\. Yes/no questions require a stable binary decision based on potentially dispersed or conflicting evidence\. Factoid questions demand accurate identification of a specific biomedical entity, such as a drug, gene, disease, or molecular target, where minor variations in terminology can directly affect exact\-match evaluation\. List questions are more challenging in terms of coverage and verification because systems must retrieve multiple correct entities while avoiding unsupported or redundant predictions\. Although all three types are grounded in the same collection of snippets, they differ substantially in their reasoning requirements and evaluation criteria\[Krithara2023BioASQQA\]\. This suggests that a single inference strategy may not be equally effective across all question types\.

Recent LLMs have demonstrated strong capabilities in medical question answering and reasoning, particularly in retrieval\-augmented settings where relevant contextual evidence is provided\[lievin2024medical,singhal2025toward,zhang2025leveraging,jung2026breaking\]\. However, their application to biomedical QA remains constrained by several practical issues\. First, model outputs can be sensitive to the order and composition of retrieved snippets, producing unstable predictions even when the underlying evidence is unchanged\[kim2025prompting,liu2023lostmiddlelanguagemodels,zhang2025leveraging\]\. Second, when multiple snippets contain overlapping or conflicting information, an LLM may fail to identify which evidence should dominate the final answer\[Zong\_2025,lewis2021retrievalaugmentedgenerationknowledgeintensivenlp\]\. Third, entity\-oriented questions require careful output normalization and evidence grounding, since plausible but unsupported biomedical terms may be generated\[Merker2024MiBiAB,nentidis2025overviewbioasq2025thirteenth\]\. Finally, hallucinated answers are especially problematic in the biomedical domain, where reliability and traceability are essential\[zhu\-etal\-2025\-trust,asgari2025framework\]\. Prior work on retrieval\-augmented biomedical LLMs has shown that domain\-specific retrieval and self\-reflection can improve answer reliability and evidence use\[jeong2024improvingmedicalreasoningretrieval,ateia2024bioragentretrievalaugmentedgenerationshowcasing\]\. These limitations motivate inference procedures that explicitly account for evidence organization, answer verification, and question\-specific characteristics\.

In this work, we present a type\-specific LLM\-based ensemble and agentic QA framework developed for BioASQ 14b Task B\. Rather than applying a uniform inference procedure across all questions, our framework assigns specialized strategies according to question type\. The strategies were selected through preliminary experiments on BioASQ 13b data and subsequently deployed in the official BioASQ 14b evaluation\. Previous studies have independently demonstrated the robustness of both LLM\-based ensembles and agentic QA frameworks\[panou2024farming,roh2026clag\], but our contribution lies in unifying these two approaches\. For yes/no questions, we apply snippet shuffling together with self\-reflection\[shinn2023reflexionlanguageagentsverbal\]to reduce sensitivity to evidence ordering and improve the consistency of binary decisions\. For factoid questions, we combine full\-snippet input with chain\-of\-thought\-based in\-context learning\[wei2023chainofthoughtpromptingelicitsreasoning,wang2023selfconsistencyimproveschainthought\]to support precise identification of biomedical entities\. For list questions, we employ a multi\-agent collaboration architecture in which evidence extraction, candidate generation, verification, and final answer aggregation are performed in separate but coordinated stages\[tang2024medagentslargelanguagemodels,ateia2024bioragentretrievalaugmentedgenerationshowcasing\]\. Similar agent\-oriented architectures have recently been explored for biomedical question answering in BioASQ\[angulo2025aqams\]\.

To further improve robustness, we apply the selected type\-specific strategies to multiple LLMs and integrate their outputs using task\-appropriate ensemble procedures\. This design aims to mitigate model\-specific bias while preserving the distinct requirements of each answer format\. In the official BioASQ 14b Task B evaluation,ku\_dmisdemonstrated competitive performance across the evaluated batches and question types\. Notably, it ranked first in the factoid subtask of Batch 4 according to MRR\[BioASQ14bResults\]\. These results support the effectiveness of combining question\-type\-specific inference, ensemble prediction, and explicit verification mechanisms for reliable LLM\-based biomedical question answering\.

## 2Task setting

The BioASQ initiative aims to advance research on automated systems for answering biomedical questions\. BioASQ 14b uses English biomedical questions and benchmark data prepared by biomedical experts\. In Phase B of Task 14b, the organizer provides the same questions released in Phase A, together with gold relevant articles and snippets\. Systems may return an ideal paragraph answer for any question type and exact answers foryes/no, factoid, andlistquestions\[BioASQ14bGuidelines\]\. The 2026 test set was released in four batches, with Phase B answers due after gold snippets became available for each batch\[BioASQ14bGuidelines\]\.

Yes/No\.Given the supplied snippets, the system must determine whether the appropriate response is "yes" or "no\." A representative example is:"Can tumor cells be reprogrammed into immune cells?"In many cases, arriving at a correct yes/no decision requires extracting and aggregating information from several snippets rather than depending on any single passage\. Performance is measured using eitheraccuracyor themacro\-averaged F1\-score, in which "yes" and "no" are handled as independent classes\. For yes/no questions, we followed the official BioASQ evaluation protocol for exact answers introduced in BioASQ6\[malakasiotis2020evaluation\]\. The F1\-score is computed independently for the “yes” and “no” classes, yieldingF​1yF1\_\{y\}andF​1nF1\_\{n\}, respectively\. ForF​1yF1\_\{y\}, "yes" is treated as the positive class, whereas forF​1nF1\_\{n\}, "no" is treated as the positive class\. The official score is the macro\-averaged F1 score:

m​a​F​1=F​1y\+F​1n2,maF1=\\frac\{F1\_\{y\}\+F1\_\{n\}\}\{2\},
which assigns equal importance to performance on both answer classes\.

Factoid\.Questions in this category call for a concise response that conveys the specific piece of factual content being asked about\. For instance,"What is the most common genotoxic factor that humans are exposed to?"The system is required to return a list up to 5 entity names, ordered by decreasing confidence\. Gold\-standard responses are represented as a nested list, with each inner list grouping together synonymous or equivalent surface forms of the same correct answer\.

Factoid questions are evaluated usingstrict accuracy \(SAcc\), lenient accuracy \(LAcc\), and mean reciprocal rank \(MRR\)\. SAcc considers an answer correct only when the gold entity name or one of its synonyms appears as the first item in the system’s returned list, whereas LAcc considers it correct if the gold entity or synonym appears anywhere in the returned list\. Although SAcc and LAcc are reported for completeness, the official metric for factoid exact answers is MRR\[malakasiotis2020evaluation\]\. MRR evaluates the rank position of the first correct answer in the returned list and gives higher scores when the gold answer or its synonym appears closer to the top\. In the definition below, for each factoid questionqiq\_\{i\}the topmost position that contains the golden entity name is searched\. If the topmost position is thejj\-th one, thenr​\(i\)=jr\(i\)=j\.

MRR=1n⋅∑i=1n1r​\(i\),\\mathrm\{MRR\}=\\frac\{1\}\{n\}\\cdot\\sum\_\{i=1\}^\{n\}\\frac\{1\}\{r\(i\)\},
wherennis the number of factoid questions\.

List\.List\-type questions require systems to return multiple correct entity names\. A representative example is: “List signaling molecules \(ligands\) that interact with the receptor EGFR?” Unlike factoid questions, which expect a single best answer or a small ranked list of aliases, list questions evaluate whether the system retrieves as many distinct gold\-standard entities as possible\. Each gold entity may include synonyms, and if the system returns a synonym of a gold entity, it is counted as a correct match\. Duplicate mentions of the same entity are counted only once\. For each list question, the system’s returned list is compared with the gold\-standard list using precision, recall, and F1\-score\. The order of returned answers does not affect the score; only the overlap between the predicted and gold answer sets is considered\. Scores are averaged across all list questions, and the official BioASQ evaluation measure for list\-type questions is themean F1\-score\[malakasiotis2020evaluation\]\.

## 3System overview

![Refer to caption](https://arxiv.org/html/2607.06452v1/x1.png)Figure 1:Overall architecture for exact\-answer generation pipeline in BioASQ 14b\.Figure[1](https://arxiv.org/html/2607.06452#S3.F1)summarizes the overall architecture\. The system first routes each question according to its BioASQ type\. Each branch receives the question body and the gold snippets\. We did not use a single universal prompt; instead, each branch uses a prompt schema and aggregation strategy tailored to its expected answer space\. The final stage validates the output format before submission, enforcing yes/no labels for binary questions, ranked lists of up to five entries for factoid questions, and list\-style entity outputs for list questions\.

The system is inference\-time and prompt\-driven\. Multiple LLMs are used as complementary biomedical reasoners, and their outputs are aggregated to reduce sensitivity to a single model’s prompt interpretation\. The main aggregation logic differs by question type: majority voting for yes/no, consensus filtering for factoid answers, and agentic evidence verification for list answers\.

## 4Methods

During the BioASQ\-14b challenge period, official test results for the submitted batches were not available\. Therefore, we used the BioASQ\-13b test set as a development benchmark to compare candidate prompting strategies, model backbones, snippet\-processing methods, and agent\-based variants\. Tables[1](https://arxiv.org/html/2607.06452#S4.T1),[2](https://arxiv.org/html/2607.06452#S4.T2), and[3](https://arxiv.org/html/2607.06452#S4.T3)report these preliminary experiments for yes/no, factoid, and list questions, respectively\. Based on these development results, we selected the most robust strategy for each question type and applied the system to the official BioASQ\-14b evaluation\.

### 4\.1Yes/No questions

The yes/no subsystem is designed to produce stable binary decisions from conflicting evidence\. As shown in Table[1](https://arxiv.org/html/2607.06452#S4.T1), several methods achieved strong performance on the BioASQ\-13b development benchmark\. The previous baseline denotes the system that achieved state\-of\-the\-art performance in the BioASQ\-13b yes/no subtask\[kim2025prompting\]\. We extended this baseline with self\-reflection, repeated inference, and a verification agent to improve the robustness of yes/no prediction\.

Based on these findings, the final yes/no subsystem employs an ensemble of four LLM configurations: GPT\-5 Thinking, GPT\-5 Pro\[openai\_gpt5\], GPT\-4o\[openai\_gpt4o\]and Gemini 3\.0 Thinking\[google\_gemini3\]\. GPT\-4o was included because it remained competitive with GPT\-5\-family models and occasionally outperformed them in the BioASQ\-13b development experiments\.

For each question, snippets are shuffled to reduce sensitivity to snippet order\. Each model is queried four times with different snippet orders, and each run is constrained to return a binary answer, eitheryesorno\. The initial answer is then determined by majority vote over the sampled outputs\. Because conflicting outputs indicate that the evidence may support more than one interpretation, such cases are further examined by a verification agent rather than being resolved solely by majority voting\.

The verification agent is applied selectively\. If all model outputs are consistent, the majority decision is submitted directly\. If at least one model output differs, the case is passed to an agentic verification pipeline\. The first component, an evidence\-structuring agent, groups the snippets into positive evidence, negative evidence, and uncertainty evidence with respect to the question\. The second component, a final\-decision agent, receives the structured evidence and all model outputs, then produces a finalyes,no, ornone\. Ifnoneremains, the evidence is rechecked and the prompt is tightened to force a binary decision; this re\-adjudication is allowed up to two times\.

Table 1:Evaluation on the BioASQ\-13b test set for yes/no questions\. The best score in each column is highlighted in bold\. ICL2 denotes in\-context learning with two demonstration examples\.ModelMethodAcc\.F1 YesF1 NoMacro F1GPT\-4oFull snippets0\.94121\.00000\.88890\.9444Full snippets \+ ICL20\.94121\.00000\.88890\.9444Verifier \+ Reviser \+ ICL20\.94121\.00000\.88890\.9444Snippet highlight0\.94121\.00000\.88890\.9444Snippet highlight \+ ICL20\.94121\.00000\.88890\.9444Previous baseline1\.00001\.00001\.00001\.0000Previous baseline \+ self\-reflection1\.00001\.00001\.00001\.0000GPT\-5\.2Full snippets0\.94120\.96000\.88890\.9244Full snippets \+ CoT ICL20\.94120\.96000\.88890\.9244Snippet filtering top\-100\.94120\.96000\.88890\.9244Agent0\.94120\.96000\.88890\.9244GPT\-5\.4Full snippets0\.94120\.96000\.88890\.9244Snippet filtering top\-100\.94120\.96000\.88890\.9244Agent0\.94120\.96000\.88890\.9244ClaudeSonnet\-4\.6 Agent0\.94120\.96000\.88890\.9244Opus\-4\.6 Agent0\.94120\.96000\.88890\.9244

### 4\.2Factoid questions

The factoid subsystem is designed to produce a ranked list of concise biomedical entities or phrases that answer a given question\. Since BioASQ factoid evaluation is sensitive to exact answer forms, the subsystem must recover both the correct entity and its expected surface form\. Table[2](https://arxiv.org/html/2607.06452#S4.T2)shows that in\-context learning and full\-snippet input were effective in the BioASQ\-13b development experiments\. GPT\-4o with ICL\-based prompting achieved strong strict accuracy and MRR, while GPT\-5\-family models showed competitive lenient accuracy\. These findings motivated a factoid pipeline that combines full\-snippet reasoning, retrieved in\-context examples, and multi\-model consensus\.

Because factoid prediction depends heavily on answer form, the demonstrations used for in\-context learning must provide more than output\-format guidance\. They should also expose the model to similar question types, evidence patterns, and entity forms\. We therefore use retrieval\-based ICL, where semantically similar training questions are selected as demonstrations\. This design follows prior findings that the choice of in\-context examples can strongly affect downstream performance\[liu2021makesgoodincontextexamples\]\.

For each test question, we retrieve semantically similar training questions using BioBERT\-based question embeddings\[lee2020biobert\]\. The retrieved examples are stored in a precomputed ICL index, and two examples are inserted into the prompt at inference time\. This provides question\-specific demonstrations that are closer to the target biomedical context than manually selected generic examples\.

We further augment the retrieved demonstrations with verified chain\-of\-thought exemplars\. Rather than encouraging verbose final outputs, these exemplars demonstrate the evidence\-to\-answer process required for factoid QA: identifying the answer\-bearing snippet, excluding related but incorrect entities, and producing a concise canonical answer\. For training examples referenced by the ICL index, we use GPT\-5\.2\[openai\_gpt52\]to generate a short reasoning trace and an exact answer, retaining only examples whose generated answer matches the gold answer\. When available, the verified exemplar replaces the corresponding plain ICL example during inference\.

For each target question, all golden snippets are provided to the model to avoid missing answer mentions or disambiguating evidence\. For ICL examples, we use at most three snippets per example to control prompt length\.

The same prompting strategy is applied to a heterogeneous pool of LLMs, including Claude Opus 4\.6\[anthropic\_claude\_opus46\], Claude Sonnet 4\.6\[anthropic\_claude\_sonnet46\], GPT\-5\.2\[openai\_gpt52\], GPT\-5\.4\[openai\_gpt54\], Gemini 3\.1 Pro Preview\[google\_gemini31pro\], and Gemini 3 Flash Preview\[google\_gemini3flash\]\. Each model outputs a JSON array of exact\-answer candidates\. After normalizing superficial formatting differences, candidates are aggregated by voting\. Answers appearing in at least three model outputs are retained, ranked by vote count and evidence consistency, and truncated to at most five answers following the BioASQ factoid format\.

Table 2:Evaluation on the BioASQ\-13b test set for factoid questions\. The best score in each column is highlighted in bold\.ModelMethodStrict Acc\.Lenient Acc\.MRRGPT\-4oFull snippets0\.42300\.46200\.4420Full snippets \+ ICL20\.53800\.57700\.5580Verifier \+ Reviser \+ ICL20\.53800\.57700\.5580Snippet highlight0\.46200\.50000\.4810Snippet highlight \+ ICL20\.53800\.57700\.5580Previous baseline0\.42310\.50000\.4615Previous baseline \+ self\-reflection0\.50000\.50000\.5000GPT\-5\.2Full snippets0\.46200\.61500\.5320Full snippets \+ CoT ICL20\.38500\.46200\.4230Snippet filtering top\-100\.46200\.57700\.5060Agent0\.34600\.34600\.3460GPT\-5\.4Full snippets0\.50000\.57700\.5290Snippet filtering top\-100\.50000\.61500\.5510Agent0\.38500\.38500\.3850ClaudeSonnet\-4\.6 Agent0\.46200\.50000\.4810Opus\-4\.6 Agent0\.46200\.50000\.4810

### 4\.3List questions

List questions are challenging because they require both high recall and high precision: the system must retrieve all valid entities while avoiding unsupported ones\. As shown in Table[3](https://arxiv.org/html/2607.06452#S4.T3), agent\-based variants achieved the strongest performance on the BioASQ\-13b development benchmark, with GPT\-5\.2 obtaining the best mean precision, recall, and F\-measure\. These results motivated a collaborative list\-agent pipeline for BioASQ\-14b\.

Figure[2](https://arxiv.org/html/2607.06452#S4.F2)illustrates the workflow\. The pipeline consists of four sequential agents: an Evidence Analyst Agent, a Reasoning Agent, a Verification Agent, and a Supervisor Agent\.

The Evidence Analyst first scans all provided snippets, scores their relevance, extracts candidate entities, and produces a structured evidence report without committing to a final answer\. The Reasoning Agent then uses this report, together with the original snippets, to generate a candidate list while independently re\-reading the evidence to avoid omissions\. The candidate list is passed to the Verification Agent, which audits it for grounding, completeness, formatting, and duplicate entities\. Finally, the Supervisor Agent reviews the intermediate reports and verification feedback, then either adopts the candidate list or revises it using the original snippets\. Representative prompt templates for these agents are provided in Appendix[A](https://arxiv.org/html/2607.06452#A1)\.

This separation of roles targets two common failure modes of single\-pass LLM inference on list questions\. Models may over\-generate biomedical entities that are topically related but not directly requested, or under\-generate by missing valid entities that appear only in low\-salience snippets\. By separating evidence extraction, answer generation, and entity\-level verification, the list subsystem aims to improve recall and precision simultaneously\.

![Refer to caption](https://arxiv.org/html/2607.06452v1/x2.png)Figure 2:List\-agent workflow for BioASQ list questions, combining Evidence Agent, Reasoning Agent, and Verification Agent\. Verified outputs are reviewed by a supervisor to produce a final evidence\-supported list answer\.Table 3:Evaluation on the BioASQ\-13b test set for list questions\. The best score in each column is highlighted in bold\.ModelMethodMean Prec\.RecallF\-measureGPT\-4oFull snippets0\.53770\.50180\.4923Full snippets \+ ICL20\.49380\.49190\.4613Verifier \+ Reviser \+ ICL20\.52290\.48120\.4765Snippet highlight0\.54870\.49950\.4975Previous baseline0\.53630\.49470\.4920Previous baseline \+ self\-reflection0\.50000\.47600\.4643GPT\-5\.2Full snippets0\.43610\.40960\.4076Full snippets \+ CoT ICL20\.57680\.54950\.5398Snippet filtering top\-100\.46010\.42820\.4289Agent0\.59180\.58540\.5560GPT\-5\.4Full snippets0\.45970\.44630\.4375Snippet filtering top\-100\.44580\.43840\.4275Agent0\.52260\.57300\.5221ClaudeSonnet\-4\.6 Agent0\.50230\.58290\.5117Opus\-4\.6 Agent0\.52410\.56550\.5185

### 4\.4Submission variants

Across batches, submission variants were used to test small changes in prompting, answer normalization, and aggregation thresholds\. For example, list\-question variants differ in how strongly the verification feedback constrains the final list and in the precision\-recall balance of the aggregation agent\. Factoid variants differ mainly in in\-context example selection and consensus thresholds\. The official result page reports these variants separately asku\_dmis1,ku\_dmis2,ku\_dmis3,ku\_dmis4, andku\_dmis5when submitted\.

## 5Results

### 5\.1Overall Performance

Table 4:Best exact\-answer scores on BioASQ\-14b by batch, computed by taking the maximum score across submissions for each metric\. “Avg\.” denotes the arithmetic mean across the four batches\.BatchYes/NoFactoidListAcc\.F1 YesF1 NoMacro F1StrictLenientMRRMean Prec\.RecallF\-measure10\.94120\.95240\.92310\.93770\.43480\.47830\.45650\.30960\.26180\.273420\.95240\.96550\.92310\.94430\.25000\.50000\.34750\.46720\.46720\.462830\.90910\.93330\.85710\.89520\.41180\.52940\.47060\.47500\.55350\.474340\.87500\.90000\.83330\.86670\.45450\.72730\.57580\.66770\.70620\.5652Avg\.0\.91940\.93780\.88420\.91100\.38780\.55880\.46260\.47990\.49720\.4439

Table[4](https://arxiv.org/html/2607.06452#S5.T4)summarizes the best exact\-answer performance on BioASQ\-14b across the four official batches\. For each metric, the table reports the maximum score obtained among the submitted variants, with the final row showing the arithmetic mean across batches\.

Overall, the yes/no subsystem produced the most stable results\. Performance remained high across all batches, indicating that the snippet\-shuffling ensemble and selective verification strategy were effective for binary decision making even under varying evidence conditions\. Although later batches were more challenging, the system maintained strong macro\-level performance, suggesting that it handled both positive and negative answers without relying heavily on one class\.

Factoid performance showed greater variation across batches\. The difference between strict and lenient accuracy suggests that the system often recovered the correct entity but did not always rank the exact expected answer first\. This pattern reflects the difficulty of BioASQ factoid questions, where success depends not only on identifying the relevant biomedical entity but also on selecting the appropriate surface form and ranking it correctly\. The strongest factoid results were observed in the final batch, suggesting that the combination of full\-snippet reasoning, retrieved demonstrations, and multi\-model consensus was particularly effective when answer candidates could be recovered from the provided evidence\.

List questions exhibited the clearest improvement across batches\. Both recall\-oriented and precision\-oriented metrics increased in later batches, with the final batch showing the strongest overall list performance\. This trend supports the motivation behind the collaborative list\-agent pipeline: separating evidence extraction, answer generation, verification, and supervision helps reduce both over\-generation of unsupported entities and under\-generation of valid but less salient entities\.

### 5\.2Variant Analysis

Table 5:Submission variants that produced the best values in selected metric groups\. Ties are shown when multiple variants obtained the same value\.BatchBest yes/no metricsBest factoid strictBest factoid MRRBest list precisionBest list F\-measure1ku\_dmis1ku\_dmis1ku\_dmis1ku\_dmis1ku\_dmis12ku\_dmis1all variantsall variantsku\_dmis1ku\_dmis43all variantsall variantsku\_dmis2\-\-5ku\_dmis1ku\_dmis4,54all variantsku\_dmis1,2,3,5ku\_dmis3,5ku\_dmis1ku\_dmis4Table[5](https://arxiv.org/html/2607.06452#S5.T5)provides a more detailed view of how different submission variants contributed to the final results across question types and metrics\. For yes/no questions, the best\-performing variant was largely unchanged across submissions:ku\_dmis1achieved the best scores in Batches 1 and 2, while all variants tied in Batches 3 and 4\. This pattern suggests that the yes/no branch was relatively insensitive to the small prompt changes introduced in later submissions\.

In contrast, factoid results show greater sensitivity to variant design, particularly for MRR\. While strict accuracy was often shared by multiple variants, the best MRR in later batches came from more specific variants, such asku\_dmis2–5in Batch 3 andku\_dmis3,5in Batch 4\. This indicates that different in\-context example selections and consensus thresholds did not always change whether the correct entity appeared in the output, but they could affect its rank within the returned answer list\. Since MRR rewards placing the correct entity higher, these differences suggest that factoid performance depends not only on entity detection but also on answer ordering and confidence calibration\.

List\-question performance also depended on submission variants\. Theku\_dmis1variant consistently produced the best precision, whereasku\_dmis4orku\_dmis5produced the best list F\-measure in later batches\. This reflects the intended precision–recall trade\-off among list\-question variants: a more conservative aggregation strategy can suppress unsupported entities and improve precision, while a less restrictive or more recall\-oriented strategy can recover additional valid entities and improve F\-measure\. Batches 2 and 4 illustrate this clearly, whereku\_dmis1achieved the best precision,ku\_dmis4achieved the best F\-measure\.

Taken together, the results support the use of answer\-type\-aware inference strategies\. The yes/no branch benefited from repeated inference and majority voting, the factoid branch benefited from full\-snippet reasoning and multi\-model consensus, and the list branch benefited from explicit evidence verification and aggregation\. The remaining variability, especially in factoid and list questions, indicates that further improvements may require stronger biomedical synonym normalization, better ranking of factoid candidates, and more adaptive thresholding for list\-answer aggregation\.

### 5\.3Question\-type Analysis

The results support the decision to use different inference strategies for different question types\. The yes/no branch achieved high and stable macro F1 in the first two batches\. The snippet\-shuffling ensemble likely helped reduce ordering effects, while selective verification limited additional agentic processing to cases where the model pool disagreed\. This design avoids applying a costly verification step to every question while still targeting uncertain samples\.

Factoid performance was more variable\. Batch 4 produced the highest MRR, but Batch 2 had lower strict accuracy and MRR\. We attribute this behavior to two factors\. First, factoid questions are highly sensitive to surface\-form matching, so equivalent biomedical names may not be credited if the submitted form is not aligned with the official answer\. Second, the consensus threshold of three model outputs improves precision but can remove rare correct answers mentioned by only a subset of models\. The use of full snippets and chain\-of\-thought\-based in\-context learning helped stabilize entity extraction, but further improvements require stronger biomedical synonym normalization and abbreviation handling\.

List\-answer results improved substantially in later batches\. The best F\-measure increased from 0\.2734 in Batch 1 to 0\.5652 in Batch 4\. This trend is consistent with the addition and refinement of the collaborative list\-answer pipeline\. The verification agent was particularly useful for removing unsupported or overly broad entities, while the aggregation agent allowed the system to recover valid entities that were initially omitted by the reasoning agent\. The precision\-recall trade\-off across Batch 4 variants indicates that list answering remains sensitive to thresholding: the most precise variant was not the highest\-recall variant, and the best F\-measure came from a middle setting\.

## 6Limitations

This work has several limitations\. First, the system relies on proprietary LLMs, so exact reproduction depends on access to the same models and versions\. Second, the result summary in Table[4](https://arxiv.org/html/2607.06452#S5.T4)is a per\-metric maximum across submitted variants rather than the score of a single fixed system\. This is useful for analyzing the best achieved metric values, but it should not be interpreted as one official system row\. Third, the system focuses on exact answers and does not address ideal\-answer generation; the official ideal\-answer rows forku\_dmissubmissions were not scored in the result tables\. Finally, because the method is prompt\-driven, small prompt changes can affect output formatting, especially for factoid and list questions\.

## 7Conclusion

We presented theku\_dmissystem for BioASQ 14b Phase B exact\-answer generation\. The system uses question\-type\-specific LLM ensembles and verification mechanisms: snippet\-shuffled majority voting for yes/no questions, full\-snippet CoT\-based in\-context learning and consensus filtering for factoid questions, and a collaborative evidence\-analysis and verification pipeline for list questions\. Aggregating the officialku\_dmissubmissions by taking the best value per metric, the system achieved average scores of 0\.9110 yes/no macro F1, 0\.4626 factoid MRR, and 0\.4439 list F\-measure across four batches\. Future work should focus on reproducible open\-model variants, biomedical synonym normalization, and unified optimization of precision\-recall thresholds for list answers\.

## 8Declaration on Generative AI

During the preparation of this work, Generative AI tools were used for writing assistance, including grammar correction and sentence rephrasing\. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content\.

## References

## Appendix APrompt Examples

This appendix provides representative prompt templates used in the list\-question agent pipeline\. The templates are included for transparency; implementation details such as model\-specific wrappers, retry logic, and JSON repair are omitted for brevity\.

Evidence Analyst Agent PromptSYSTEM You are an Evidence Analyst specialized in biomedical literature\. Your task is to carefully analyze provided snippets and extract key evidence relevant to answering a biomedical question\. You do NOT answer the question itself \-\-\- you only analyze and organize the evidence\. USER Analyze the following snippets to extract evidence relevant to the question\. \[Question\]: \{question\} \[Question Type\]: list \[Snippet 1\]: … \[Snippet 2\]: … … INSTRUCTIONS: 1\. For each snippet, assess its relevance to the question \(0\.0 to 1\.0\)\. 2\. Extract key entities, facts, and relationships from the relevant snippets\. 3\. Provide a concise evidence summary that organizes the extracted information\. 4\. Determine whether the evidence is sufficient to answer the question\. 5\. Extract ALL entities/items that belong to the category asked about\. Scan every snippet exhaustively \-\-\- do not miss items\. Respond ONLY in valid JSON with this exact format: \{ "relevant\_snippets": \[\{"idx": 1, "relevance\_score": 0\.95, "key\_facts": \["fact1", "fact2"\]\}, …\], "extracted\_entities": \["entity1", "entity2", …\], "evidence\_summary": "A concise paragraph summarizing the key evidence…", "evidence\_sufficient": true, "confidence": 0\.85 \}

Prompt for the Evidence Analyst, which scores snippet relevance and extracts candidate entities\.

Reasoning Agent PromptSYSTEM You are a Reasoning Agent for BioASQ Task B\. You receive an evidence analysis report and the original snippets\. Your task is to generate the most accurate answer based on the evidence\. You must ground your answer ONLY in the provided snippets \-\-\- never invent facts\. You also explain your reasoning process and indicate which snippets support your answer\. USER You are generating an answer for a BioASQ list question\. \[Evidence Analysis Report\]: \- Evidence Summary: \{evidence\_summary\} \- Key Entities: \{extracted\_entities\} \- Evidence Sufficient: \{evidence\_sufficient\} \- Analyst Confidence: \{confidence\} \[Snippets\]: \{snippets\_text\} \(optional ICL block: "Here are some examples of how to answer: …Now answer the following question in the same format:"\) \[Question\]: \{question\} Use the evidence analysis to guide your reasoning, but base your final answer ONLY on the original snippets above\. The evidence report helps you focus on what’s important, but the snippets are the ground truth\. Return up to 7\-\-8 items\. Scan every snippet exhaustively and do not rephrase entity surface forms\. Respond ONLY in valid JSON: \{ "answer": \["item1", "item2", "item3", …\], "reasoning": "…", "supporting\_snippets": \[1, 2, 3\], "confidence": 0\.9, "alternative\_answers": \[…\] \}

Prompt for the Reasoning Agent, which drafts a grounded candidate list from the evidence report\.

Verification Agent PromptSYSTEM You are a Verification Agent for BioASQ Task B\. You independently verify whether an answer is correctly grounded in the provided snippets\. You check for factual accuracy, completeness, proper formatting, and consistency with the evidence analysis\. You do NOT generate answers \-\-\- you only verify and provide feedback\. USER Verify the following answer for a BioASQ list question\. \[Question\]: \{question\} \[Snippets\]: \{snippets\_text\} \[Answer to Verify\]: \{answer\} \[Reasoning Provided\]: \{reasoning\} \[Evidence Analysis Summary\]: \{evidence\_summary\} \[Evidence Entities\]: \{extracted\_entities\} VERIFICATION CHECKS: 1\. GROUNDING: Is every claim in the answer directly supported by at least one snippet? 2\. ACCURACY: Is the answer factually correct according to the snippets? 3\. COMPLETENESS: Does the answer address the question fully? 4\. FORMAT: Is the answer in the correct format for a list question? 5\. CONSISTENCY: Is the answer consistent with the evidence analysis report? 6\. LIST\-SPECIFIC: Check completeness of the returned entities, flag any false positives, and remove duplicate entities\. CROSS\-CHECK: Compare the answer’s entities with the Evidence Analyst’s extracted entities\. Flag any entities in the answer that don’t appear in either the snippets or the evidence report\. Respond ONLY in valid JSON: \{ "verdict": "supported" or "needs\_revision" or "unsupported", "issues": \["issue1", "issue2"\], "suggested\_correction": "corrected answer or null if no correction needed", "confidence": 0\.85 \} RULES: \- "supported": Answer is correct, grounded, and well\-formatted\. \- "needs\_revision": Answer has minor issues \(format, verbosity, slight inaccuracy\) but is mostly correct\. \- "unsupported": Answer is factually wrong, contains hallucination, or completely misses the point\. \- If verdict is "supported", issues should be empty and suggested\_correction should be null\. \- If verdict is "needs\_revision" or "unsupported", provide specific issues and a suggested\_correction\.

Prompt for the Verification Agent, which audits the list for grounding, completeness, and duplicates\.

Supervisor Agent PromptSYSTEM You are the Supervisor Agent making the final decision for a BioASQ Task B question\. You receive reports from three subordinate agents: \(1\) Evidence Analyst \-\-\- who analyzed the snippets, \(2\) Reasoning Agent \-\-\- who generated an answer, \(3\) Verification Agent \-\-\- who verified the answer\. Your job is to synthesize these reports and produce the FINAL answer\. You are the ultimate authority \-\-\- you may override any subordinate if you have good reason\. USER You are making the final decision for a BioASQ list question\. \[Question\]: \{question\} \[Original Snippets\]: \{snippets\_text\} === SUBORDINATE AGENT REPORTS === \[Evidence Analyst Report\]: \- Evidence Summary: \{evidence\_summary\} \- Key Entities: \{extracted\_entities\} \- Evidence Sufficient: \{evidence\_sufficient\} \- Confidence: \{confidence\} \[Reasoning Agent Report\]: \- Answer: \{answer\} \- Reasoning: \{reasoning\} \- Supporting Snippets: \{supporting\_snippets\} \- Confidence: \{confidence\} \- Alternative Answers: \{alternative\_answers\} \[Verification Agent Report\]: \- Verdict: \{verdict\} \- Issues: \{issues\} \- Suggested Correction: \{suggested\_correction\} \- Confidence: \{confidence\} === DECISION RULES === 1\. If Verification verdict is "supported" with high confidence \(≥\\geq0\.7\): →\\rightarrowUse the Reasoning Agent’s answer as\-is\. 2\. If Verification verdict is "needs\_revision" and a suggested\_correction exists: →\\rightarrowEvaluate whether the correction is better than the original\. Use your judgment\. 3\. If Verification verdict is "unsupported": →\\rightarrowCheck the suggested\_correction\. If valid, use it\. →\\rightarrowOtherwise, re\-derive the answer from the snippets yourself\. 4\. CROSS\-CHECK: Compare entities in the Reasoning answer against the Evidence Analyst’s extracted\_entities\. If there’s a mismatch, investigate and resolve\. 5\. ALWAYS verify your final answer is grounded in the original snippets\. 6\. When in doubt, prefer the answer with higher snippet support\. FINAL ANSWER FORMAT: Return the final answer as a JSON array of entity strings\. Respond ONLY in valid JSON: \{ "final\_answer": \["entity1", "entity2", …\], "decision\_reasoning": "Brief explanation of why you made this decision…", "source": "reasoning" or "verification\_correction" or "supervisor\_override", "revision\_needed": false \} IMPORTANT: \- "revision\_needed" should ONLY be true if you believe even your own final answer is unreliable and a complete re\-analysis from scratch might help\. \- In most cases, you should be able to produce a good final answer from the existing reports\.

Prompt for the Supervisor Agent, which synthesizes all reports into the final list answer\.