Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

# Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
Source: [https://arxiv.org/html/2608.12928](https://arxiv.org/html/2608.12928)
Łukasz GrzybowskiAffiliation:ARAAI PolandAntoni LasikAffiliation:NASK National Research Institute\[0\.6em\]Marek KubisJeremi Ignacy KaczmarekAffiliation:Poznań University of Medical SciencesAffiliation:T\. Marciniak Lower Silesian Specialist HospitalWojciech KusaAffiliation:NASK National Research Institute\[0\.8em\] Adam Mickiewicz University

###### Abstract

We introduce a Polish\-language medical visual question answering \(VQA\) benchmark, built from Polish Board Certification Examination questions for licensed physicians and dentists pursuing specialist certification\. The benchmark comprises image\-containing questions spanning diverse medical specialties and visual domains, together with a text\-only question answering \(QA\) control set\. We evaluate Polish\-oriented, general\-purpose open\-weight, and commercial vision\-language models\. The task remains challenging: the best model achieves 79\.0% accuracy on the full VQA set, and only GPT\-5\.6 surpasses the approximate human reference on the subset with available candidate responses; all other evaluated models perform worse than humans\. To assess visual grounding, we compare complete inputs with configurations omitting the image, the question, or both, and categorize questions by image importance\. Models derive more useful information from the question text than from the image and perform worse on image\-dominant questions\. Across both QA and VQA, they nevertheless achieve above\-chance accuracy from the answer choices alone, showing that non\-trivial performance can persist even when key task components are missing\.

## 1Introduction

Large language models \(LLMs\) have recently been evaluated on several Polish medical examination benchmarks, including the Polish Board Certification Examination \(pol\.*Państwowy Egzamin Specjalizacyjny*, PES\), the Medical Final Examination \(pol\.*Lekarski Egzamin Końcowy*, LEK\), the Dental Final Examination \(pol\.*Lekarsko\-Dentystyczny Egzamin Końcowy*, LDEK\), and related medical test sets[28](https://arxiv.org/html/2608.12928#bib.bib10);[18](https://arxiv.org/html/2608.12928#bib.bib12);[22](https://arxiv.org/html/2608.12928#bib.bib11)\. These studies showed that modern LLMs can achieve strong results on Polish medical multiple\-choice questions and provided evidence on how well models handle specialized medical knowledge in a non\-English setting\. However, their evaluation protocols were limited to text\-only questions\. As a result, examination items containing images were excluded, even though visual information is an important component of many real medical tasks and of some PES questions\.

This omission leaves an important gap in the evaluation of medical AI systems\. While visual question answering \(VQA\) has been widely studied in English, non\-English medical VQA remains relatively underexplored\. Polish VQA resources are also limited, especially in specialized domains such as medicine\. This is problematic because model performance in English cannot be assumed to transfer directly to Polish, and medical examination questions often require knowledge of domain\-specific terminology, clinical conventions, and local examination formats\. Consequently, there is a need for benchmarks that evaluate not only medical knowledge in Polish, but also the ability of models to combine Polish clinical text with medical images\.

In this work, we evaluate vision\-language models \(VLMs\) on image\-containing questions from PES, the Polish Board Certification Examination\. These questions are multiple\-choice examination items intended for physicians and dentists pursuing specialist certification\. They provide a challenging test bed for multimodal medical question answering, as they often require both domain knowledge and interpretation of visual evidence\. Importantly, the dataset is not composed solely of classical VQA examples where the image is the central object of a direct visual query\. In many cases, the image is only one component of a broader clinical scenario: the question may include a textual patient description, laboratory or diagnostic context, answer choices, and an image such as an electrocardiogram, radiological scan, or clinical photograph\. Therefore, the task is better understood as multimodal medical examination question answering rather than simple image recognition or image\-centered VQA\.

Beyond measuring overall model accuracy, we study how much information models obtain from different parts of the input\. Prior work has shown that models can exploit artifacts in multiple\-choice answer options or rely disproportionately on textual cues instead of genuinely using visual evidence[4](https://arxiv.org/html/2608.12928#bib.bib8);[5](https://arxiv.org/html/2608.12928#bib.bib9);[3](https://arxiv.org/html/2608.12928#bib.bib4)\. To examine this issue in the Polish medical examination setting, we evaluate models under controlled input configurations: using only the answer choices, using choices together with the question text, using choices together with the image, and using the full input consisting of choices, question text, and image\. This setup allows us to estimate the relative contribution of answer choices, textual context, and visual information\.

We also compare performance on image\-containing PES questions with performance on a text\-only question answering \(QA\) control set composed of questions that originally did not include images\. This comparison allows us to analyze differences between QA\- and VQA\-style evaluation within the same examination domain\. Additionally, we conduct a data contamination analysis to assess whether model performance may have been influenced by prior exposure to the evaluation questions\.

Our contributions are as follows:

- •We create an image\-containing question dataset from the Polish Board Certification Examination as a benchmark for Polish medical multimodal question answering\.
- •We evaluate vision\-language models on PES questions under several input configurations that separate the effects of answer choices, question text, and images\.
- •We compare model performance on image\-containing VQA questions with performance on a text\-only QA control set from the same examination domain\.

## 2Related Work

### 2\.1Polish VQA

[33](https://arxiv.org/html/2608.12928#bib.bib1)adapt the LLaVA framework to Polish and introduce LLaVA\-Bielik and LLaVA\-PLLuM\. They show that translated and filtered multimodal data can effectively bootstrap Polish VLMs and provide Polish\-oriented evaluation resources\. reVISION[11](https://arxiv.org/html/2608.12928#bib.bib15)evaluates VLMs on Polish multimodal national examination data, extending the exam\-based evaluation setting introduced in LLMzSzŁ[21](https://arxiv.org/html/2608.12928#bib.bib5)from text\-only LLMs to vision\-language models\. PoVisLE[2](https://arxiv.org/html/2608.12928#bib.bib7)further moves toward Polish\-specific vision\-language evaluation with emphasis on Polish linguistic and cultural grounding\.

Polish is also present in broader multilingual VQA resources\. EXAMS\-V[12](https://arxiv.org/html/2608.12928#bib.bib17)includes Polish among multilingual multimodal examination questions, while[30](https://arxiv.org/html/2608.12928#bib.bib6)evaluate cross\-lingual transfer to Polish on 500 machine\-translated Polish questions\.

Our work differs from the aforementioned resources by focusing on specialist\-level Polish medical examination questions that require combining clinical text, answer options, and medical images\.

### 2\.2Medical VQA

English medical VQA has been studied mainly in radiology, pathology, and biomedical image–text settings\. VQA\-RAD[23](https://arxiv.org/html/2608.12928#bib.bib18)introduced clinically generated questions and answers for radiology images, while the ImageCLEF VQA\-Med shared tasks provided a series of radiology\-focused medical VQA benchmarks[19](https://arxiv.org/html/2608.12928#bib.bib19);[7](https://arxiv.org/html/2608.12928#bib.bib20);[6](https://arxiv.org/html/2608.12928#bib.bib21);[8](https://arxiv.org/html/2608.12928#bib.bib22)\. PathVQA[20](https://arxiv.org/html/2608.12928#bib.bib23)introduces a new dataset and framework for visual question answering over pathology images\. More recent datasets scale medical VQA through visual instruction tuning, including PMC\-VQA[37](https://arxiv.org/html/2608.12928#bib.bib24)and PubMedVision introduced with HuatuoGPT\-Vision[10](https://arxiv.org/html/2608.12928#bib.bib25)\.

Non\-English and multilingual medical VQA is more limited\. SLAKE provides a bilingual English–Chinese medical VQA dataset with semantic labels and medical knowledge[24](https://arxiv.org/html/2608.12928#bib.bib26)\. WorldMedQA\-V[26](https://arxiv.org/html/2608.12928#bib.bib27)and MMMED[32](https://arxiv.org/html/2608.12928#bib.bib28)evaluate multimodal medical examination questions in multiple languages\. Other recent resources address specific languages or domains, including multilingual wound\-care VQA[34](https://arxiv.org/html/2608.12928#bib.bib29), Indonesian radiology VQA[35](https://arxiv.org/html/2608.12928#bib.bib30), and multilingual hematology VQA[25](https://arxiv.org/html/2608.12928#bib.bib31)\.

### 2\.3Biases in VQA

VQA benchmarks often contain linguistic or answer\-distribution shortcuts that allow models to answer correctly without sufficient visual grounding\.[17](https://arxiv.org/html/2608.12928#bib.bib32)addressed this issue by introducing VQA v2, where similar images are paired with the same question but different answers, making the visual signal more important\.[1](https://arxiv.org/html/2608.12928#bib.bib33)further showed that VQA models rely heavily on question\-answer priors by introducing VQA\-CP, a split with different answer distributions between training and test data\. Several works proposed methods to reduce such biases, including adversarial regularization with a question\-only model[31](https://arxiv.org/html/2608.12928#bib.bib34), RUBi, which downweights examples solvable without the image[9](https://arxiv.org/html/2608.12928#bib.bib35), and visually grounded question encoding[16](https://arxiv.org/html/2608.12928#bib.bib36)\.

This issue is also relevant for modern VLMs and medical VQA\. MIRAGE shows that frontier VLMs can generate detailed visual reasoning and obtain high scores on multimodal benchmarks even without image input[3](https://arxiv.org/html/2608.12928#bib.bib4)\.[36](https://arxiv.org/html/2608.12928#bib.bib37)propose counterfactual training and a changing\-priors medical VQA split to reduce reliance on linguistic shortcuts\. Med\-BiasX similarly targets medical language biases caused by imbalanced data and question shortcut dependence[38](https://arxiv.org/html/2608.12928#bib.bib38)\. These findings motivate our controlled input configurations, which separately evaluate performance from answer choices, question text, images, and their combination\.

## 3Dataset

### 3\.1Examination background

The dataset used in this work is based on the Polish Board Certification Examination \(PES\), a national examination for physicians and dentists pursuing specialist certification in Poland\. Candidates taking PES have already obtained a medical or dental license and completed the required specialist training, including clinical practice, courses, internships, and discipline\-specific procedural requirements\.

The examination consists of a written and an oral component\. The written part is held separately for each medical or dental specialty and typically contains 120 single\-choice questions\. Each question has five answer options, exactly one of which is correct\. Most questions are text\-only, although some include an accompanying image\. A score of at least 60% is required to pass the written examination\. Since 2022, candidates who obtain at least 70% in the written part have been exempted from the oral examination\. Unlike licensing examinations such as LEK and LDEK, PES questions are not publicly available before the exam, which makes them a suitable source of challenging specialist\-level medical questions\.

In this study, we focus on the written part of PES, as it provides standardized multiple\-choice questions with unambiguous correct answers\. This format enables automatic evaluation of model predictions while preserving the specialist\-level medical character of the task\.

### 3\.2Benchmark construction

We collected PES examination materials from the Medical Examination Center \(Centrum Egzaminów Medycznych, CEM\) website111[https://www\.cem\.edu\.pl/](https://www.cem.edu.pl/), covering examination sessions from 2023 to 2026\. In total, we processed 363 examination sheets\. We removed questions marked by CEM as invalid or no longer aligned with current medical knowledge\.

We then identified examination sheets containing at least one image\. This yielded 116 examination sheets with visual material\. From these sheets, we extracted all questions containing images and constructed the VQA subset, consisting of 286 image\-containing questions\. Each VQA item contains the question text, five answer choices, the correct answer, metadata describing the examination session and specialty, and the associated image\.

For comparison with text\-only question answering, we also constructed a QA control subset\. This subset consists of PES questions that originally did not contain any images\. To make the QA subset comparable to the VQA subset, we selected specialties for which we had at least 10 image\-containing questions\. For each examination sheet in these specialties, we sampled 10 text\-only questions\. This procedure resulted in 480 QA questions\.

The distribution of VQA and QA questions across specialties is shown in Table[1](https://arxiv.org/html/2608.12928#S3.T1)\. The distribution of image\-containing questions is not uniform across medical specialties\. Emergency medicine contributes the largest number of VQA questions, followed by maxillofacial surgery, orthopedics, and pediatric cardiology\. Specialties with fewer than 10 image\-containing questions are grouped into the “Other specialties” category\. Since the QA subset was constructed only for specialties with at least 10 VQA questions, no QA items are assigned to this grouped category\.

Table 1:Number of VQA and QA questions by medical specialty\. Specialties with fewer than 10 VQA questions are grouped as other specialties\.
### 3\.3Example PES question

Figure[1](https://arxiv.org/html/2608.12928#S3.F1)presents an example image from an emergency medicine PES question\. The item illustrates the character of the dataset: the model must combine visual evidence from a head computed tomography \(CT\) scan with medical knowledge expressed in the answer options\. The correct answer is the false statement about the presented pathology\. The original question is in Polish; the question and answer choices shown below are an English translation\.

![Refer to caption](https://arxiv.org/html/2608.12928v1/imgs/2023_1_60_medycyna_ratunkowa.jpg)Figure 1:ImageQuestion\.The attached image shows a head CT scan of a patient after trauma\. Indicate thefalsestatement about the presented pathology\.

Choices:

- A\.It most commonly results from rupture of the middle meningeal artery\.
- B\.In the classic clinical presentation, there is an initial loss of consciousness, followed by a relatively asymptomatic interval, thelucid interval\.
- C\.In this type of hematoma, blood accumulates between the skull bone and the dura mater\.
- D\.A characteristic feature is a lentiform collection of blood that does not cross the cranial sutures to which the dura mater is attached\.
- E\.It most commonly results from injury to bridging veins between the surface of the brain and the dural venous sinuses\.

Correct answer:E

Metadata\.Year: 2023, Quarter: Spring, Specialty: Emergency medicine

## 4Evaluation Methodology

We evaluate nine open\-weight and commercial vision\-language models on the PES\-VQA benchmark\. The open\-weight models comprise three Polish\-oriented VLMs: LLaVA\-Bielik\-11b\-v2\.6\-instruct, LLaVA\-PLLuM\-12b\-nc\-instruct\-250715, and LLaVA\-PLLuM\-12b\-nc\-instruct[33](https://arxiv.org/html/2608.12928#bib.bib1); three Qwen models: Qwen3\.5\-397B\-A17B, Qwen3\.5\-9B, and Qwen3\.6\-27B[29](https://arxiv.org/html/2608.12928#bib.bib13); and Gemma\-4\-31B\-it[15](https://arxiv.org/html/2608.12928#bib.bib14)\. The commercial models are GPT\-5\.4\-nano\-2026\-03\-17 and GPT\-5\.6\-sol[27](https://arxiv.org/html/2608.12928#bib.bib16)\. Shortened model labels used in the result tables refer to these full model identifiers\.

Among the open\-weight models, only the LLaVA variants do not support reasoning; the others use reasoning by default\. In the result tables, theRcolumn marks reasoning as enabled \(Y\) or disabled or unavailable \(N\)\. Only the GPT models were evaluated in both configurations:none\(N\) andmedium\(Y\)\.

### 4\.1Input configurations

The objective of our evaluation extends beyond measuring final accuracy to estimating the amount of information models obtain from different portions of the input\.

We therefore evaluate models under several controlled input configurations\. For image\-containing questions, we use four settings:

- •C: answer choices only,
- •C\+Q: answer choices and question text,
- •C\+I: answer choices and image,
- •C\+Q\+I: answer choices, question text, and image\.

The C setting measures whether a model can exploit artifacts, priors, or statistical regularities in the answer choices without access to the question itself\. The C\+Q setting evaluates text\-only performance\. The C\+I setting tests whether the image provides useful information when the question text is removed\. Finally, C\+Q\+I corresponds to the complete multimodal examination item as presented to candidates, whereas the remaining configurations are used only for our ablation studies\.

For the text\-only QA control subset, only two configurations are applicable: C and C\+Q\.

### 4\.2Prompting and evaluation metric

All experiments were conducted using Polish prompts\. The prompt instructed the model to answer a single\-choice medical examination question with options A–E and to return only a JSON object containing the selected answer\. The full prompts, together with English translations and prompting details, are provided in Appendix[A](https://arxiv.org/html/2608.12928#A1)\.

For configurations in which an image was omitted from an image\-containing question, we did not explicitly inform the models that the image was unavailable\. This design choice follows[3](https://arxiv.org/html/2608.12928#bib.bib4), who found that model performance declined markedly when models were explicitly instructed to guess without image access, compared with prompts that implicitly led them to assume that an image was present\.

We report accuracy, defined as the percentage of questions for which the model’s predicted answer matches the official answer key\. Since every question has five answer options and exactly one correct answer, random guessing corresponds to an expected accuracy of 20%\.

### 4\.3Subset with human responses

Human responses were available for only a subset of the benchmark questions\. We collected these responses and used them to construct a human results subset for comparing model performance with human performance\. The data collection and alignment procedure is described in Appendix[B](https://arxiv.org/html/2608.12928#A2)\.

### 4\.4Visual signal categories

#### 4\.4\.1Image importance

To characterize how strongly each question depends on its visual material, we assigned every VQA question to one of three image\-importance categories:

0 – Image non\-essential \(text sufficient\)\.The correct answer can be determined reliably from the question and answer choices without using the image\. The image may illustrate, confirm, or repeat information already present in the text, but it is not required to solve the question\. This category contains 15 questions \(5\.2%\)\.

1 – Image and text complementary\.Both textual and visual information are needed to determine the correct answer reliably\. The text provides information independent of the image, such as symptoms, medical history, test results, or clinical context, which must be combined with the visual evidence\. This category contains 131 questions \(45\.8%\)\.

2 – Image dominant\.The correct answer depends primarily on interpreting the image\. The text contains no substantial clinical information independent of the image and serves mainly to provide instructions, identify the type of visual material, or state the task\. This category contains 140 questions \(49\.0%\)\.

#### 4\.4\.2Visual domains

We additionally categorized the visual material by content domain\. Although the underlying taxonomy is hierarchical, we report only its top\-level categories:

IMAGE \(n=102n=102\)\.Medical images, including radiological and other diagnostic imaging modalities, microscopy, ophthalmic imaging, and clinical photography\. This category also includes images for which no more specific subtype was assigned\.

WAVEFORM \(n=95n=95\)\.Physiological signal traces, including cardiac, neurophysiological, evoked\-potential, and hemodynamic recordings\.

PLOT \(n=23n=23\)\.Plots presenting measurements, relationships, or analyses, including audiological, biomechanical, glucose\-monitoring, pressure–volume, radiotherapy, spirometry, and statistical plots\.

GRAPHIC \(n=32n=32\)\.Explanatory diagrams and schematics depicting anatomy, biomechanics, classifications, devices, mechanisms, or procedures\.

TABLE \(n=20n=20\)\.Visual material organized in tabular form, including results, comparisons, and matching tables\.

COMPOSITE \(n=14n=14\)\.Material combining multiple visual or textual elements, such as device printouts, documents, software screens, or composite test results\.

OTHER \(n=3n=3\)\.Visual material that could not be assigned to any of the categories above\.

## 5Data Contamination Analysis

To detect contamination, we used the Data Contamination Quiz \(DCQ\)[14](https://arxiv.org/html/2608.12928#bib.bib2)framework\. DCQ consists of two stages: quiz creation and examination\. First, a frontier LLM generates altered versions of the questions by paraphrasing certain words to break memorization\. The tested models are then asked to identify the original question among the paraphrased variants\. We used DeepSeek\-V4\-Pro[13](https://arxiv.org/html/2608.12928#bib.bib3)for quiz creation\. Due to computational and cost constraints, we tested one representative model from each evaluated model family for contamination\. The results presented in Table[2](https://arxiv.org/html/2608.12928#S5.T2)suggest negligible contamination, which should not influence the results of our evaluation\. For each model, DCQ reports a closed interval where the lower bound is the bias\-corrected minimum contamination level \(via Cohen’s Kappa\) and the upper bound is the maximum raw quiz accuracy across bias\-compensated permutations\. It is worth noting that the original DCQ prompt explicitly references the dataset name and split as part of the instruction\. Since our dataset is not an established, named benchmark, this framing may carry less signal, and the resulting estimates should be interpreted with some caution\.

Table 2:Contamination level ranges reported using the DCQ methodology on the PES medical QA/VQA datasets \(textual part\), provided in the format \(min contamination, max contamination\)\.
## 6Results

The overall results across input configurations are presented in Figure[2](https://arxiv.org/html/2608.12928#S6.F2)\.

The evaluation conducted with respect to the full dataset shows a strong effect of model size, with larger models achieving higher accuracy\. Among open\-weight models small enough to fit on a single consumer\-grade GPU, Qwen3\.5\-9B outperforms all LLaVA\-based models\.

For the GPT models we evaluated different reasoning\-effort settings\. Increasing reasoning effort substantially improves the performance of GPT\-5\.4\-nano\. For GPT\-5\.6\-sol, it improves QA performance but provides only a small improvement on VQA\.

### 6\.1Comparison with human examinees

![Refer to caption](https://arxiv.org/html/2608.12928v1/imgs/fig1_fig2.png)Figure 2:Model comparison across input configurations\. C\+Q represents the complete configuration for QA, while C\+Q\+I represents the complete configuration for VQA\.The comparison on the subset with human responses is presented in Table[3](https://arxiv.org/html/2608.12928#S6.T3)\. QA and VQA have a similar level of difficulty for human examinees, as human accuracy is nearly identical on the two subsets\. For the evaluated models, QA is generally easier than VQA\.

Accuracy above 60% on both QA and VQA, which corresponds to a passing score for human examinees, is achieved by Qwen3\.5\-397B\-A17B, Gemma\-4\-31B\-it, and GPT\-5\.6\. Among the evaluated models, only GPT\-5\.6\-sol outperforms human examinees on average, under both thenoneandmediumreasoning\-effort settings\. This finding indicates that the dataset is particularly challenging for models\. The commercial GPT\-5\.6 model also substantially outperforms the open\-source models\.

### 6\.2Model performance with incomplete inputs

We analyze model performance under incomplete\-input scenarios separately for QA and VQA\.

Table 3:Model and human accuracy on the QA and VQA subsets, together with subset statistics\. Human accuracy is the percentage of examinee answers matching the answer key\.#### 6\.2\.1QA

All models perform above the random\-guessing baseline of 20% on QA when given only the answer choices, without the question \(C configuration\)\. One possible explanation is that the intended question can sometimes be inferred from the answer choices, for example when the task is to identify a true statement\. Alternatively, some choices may contain an intrinsic error, such as an incorrect justification, that can be detected without knowing the question\. The best\-performing model configuration, GPT\-5\.6\-sol, selects the correct answer in 42\.9% of cases without access to the question\.

#### 6\.2\.2VQA

In the choices\-only configuration \(C\), performance is similar to that observed on QA\. When the input is incomplete and either the question or the image is missing \(C\+I or C\+Q\), the models perform better than with the answer choices alone \(C\), but worse than with the complete input \(C\+Q\+I\)\. Moreover, removing the image \(C\+Q\) is less detrimental than removing the question \(C\+I\), which shows that the models make greater use of the question text\. This pattern holds for every model except LLaVA\-PLLuM, which performs relatively poorly overall\. These results show that the models perform well under incomplete\-information conditions, although their ability to identify the correct answer decreases as more information is removed\.

### 6\.3Model performance by visual category

#### 6\.3\.1Image importance

Table[4](https://arxiv.org/html/2608.12928#S6.T4)reports accuracy for the pooled text\-sufficient and complementary categories \(0\+1\) and for the image\-dominant category \(2\)\. Every evaluated model performs worse on image\-dominant questions under both C\+Q and C\+Q\+I\. The difference under C\+Q is expected because this configuration omits the image, which category 2 questions primarily require\. More notably, the same ordering persists with the complete C\+Q\+I input\. This pattern is consistent with the input\-ablation results indicating that the models rely more heavily on textual cues than on visual evidence\.

Table 4:VQA accuracy by image\-importance category for the C\+Q and C\+Q\+I input configurations\. Categories 0 and 1 are pooled; category 2 contains image\-dominant questions\.
#### 6\.3\.2Visual domains

Table[5](https://arxiv.org/html/2608.12928#S6.T5)shows accuracy across the seven top\-level visual domains\. WAVEFORM tends to be among the best\-performing well\-represented domains\. These comparisons should be interpreted cautiously because the domain sizes are unequal and the relative performance patterns across domains differ between models\.

Table 5:VQA accuracy by visual domain\. Questions containing panels from multiple domains contribute to each corresponding domain\.

## 7Conclusion

In this work, we introduce the first Polish\-language medical VQA benchmark, accompanied by a text\-only QA control subset\. Human examinees achieve nearly identical accuracy on the two subsets, indicating that the tasks are comparable in difficulty\.

We evaluate Polish\-oriented and general\-purpose open\-weight models, as well as proprietary commercial systems\. Although QA and VQA are similarly difficult for human examinees, most evaluated models perform worse on VQA\. The input\-ablation experiments further show that models derive more useful information from the question text than from the image: on VQA, they generally perform better without the image \(C\+Q\) than without the question text \(C\+I\)\. Moreover, accuracy is lower on image\-dominant questions than on questions for which textual information is sufficient or complementary, even when the models receive the complete multimodal input\. Together, these findings suggest that current models rely more heavily on textual cues than on visual evidence when answering Polish medical examination questions\.

Across both QA and VQA, the models also achieve above\-chance accuracy when given only the answer choices, suggesting that the options provide textual cues that support inference even when both the question and the image are unavailable\. This result complements the image\-free evaluation of[3](https://arxiv.org/html/2608.12928#bib.bib4), who show that VLMs can achieve high scores on multimodal benchmarks without visual input and identify textual cues as a source of non\-visual inference\. Our more restrictive choices\-only setting further shows that some usable signal may reside in the answer options themselves\. Together, these findings call for caution when interpreting benchmark performance: high accuracy does not necessarily reflect robust medical or multimodal competence when a task remains partially solvable from incomplete inputs\. This consideration is especially important in medical settings, where model outputs may have significant consequences\.

## Limitations

Our evaluation covers a selected set of models rather than the full space of currently available vision\-language models\. The number of open\-weight and commercial systems is growing rapidly, making an exhaustive comparison impractical\. We therefore focused on models that we consider representative of different relevant categories, including Polish\-oriented VLMs, general multilingual VLMs, and commercial systems\.

We did not evaluate all possible inference settings\. In particular, we did not systematically test all reasoning\-effort levels, decoding configurations, or image\-resolution variants\. These factors may affect model performance, especially for visually demanding medical questions\. Our results should therefore be interpreted as performance under the default prompting and inference setup used in this work, rather than as the maximum attainable performance of each model\.

We did not include vision\-language models specifically trained for medicine\. Previous work on Polish medical examination benchmarks has shown that general purpose LLMs can outperform models adapted to the medical domain[18](https://arxiv.org/html/2608.12928#bib.bib12)\. Nevertheless, this observation comes mainly from text\-only evaluation and may not fully transfer to multimodal medical tasks\.

The dataset is limited in size and specialty coverage\. The VQA subset contains 286 image\-containing questions, and the distribution across specialties is uneven\. Some specialties are represented by many more items than others, while specialties with few image\-containing questions provide only limited evidence about model performance\. At the same time, the questions are high\-quality examination items prepared for specialist medical certification, which makes them a valuable resource despite their limited number\.

The benchmark reflects the structure of Polish board certification examinations rather than the full range of clinical practice\. The questions test specialist medical knowledge under a standardized written\-exam format, but they do not capture interactive patient assessment, longitudinal decision\-making, procedural skills, communication, or responsibility for real\-world outcomes\.

## Ethical Considerations

The questions used in this study originate from materials published by the Polish Medical Examination Center \(Centrum Egzaminów Medycznych, CEM\)\. We did not author the original examination questions; our contribution consists of collecting, processing, structuring, and evaluating them as a benchmark for multimodal medical question answering\. We preserve the examination character of the items and use them only for research evaluation\.

Performance on written medical examinations captures only a limited part of medical competence\. Becoming a licensed physician or dentist in Poland requires extensive education, supervised clinical training, practical experience, and formal certification\. A model that performs well on multiple\-choice exam questions should therefore not be described as equivalent to a clinician, nor should such results be used to claim that models can replace medical professionals\.

This limitation is particularly important for multimodal medical questions\. Clinical work requires gathering information from patients, performing physical examinations, interpreting diagnostic tests in context, weighing contraindications and comorbidities, communicating uncertainty, and making decisions under incomplete information\. A benchmark based on static examination items cannot evaluate these abilities comprehensively\.

LLMs and VLMs may be useful in medical education, information retrieval, and decision\-support workflows, but they can also generate incorrect, incomplete, or misleading outputs\. In medical settings, such errors may create substantial risks if model responses are treated as authoritative\. Any practical deployment of these systems should therefore include oversight by qualified healthcare professionals, clear communication of model limitations, and compliance with applicable ethical, clinical, and regulatory standards\.

## References

- Agrawalet al\.\(2018\)A\. Agrawal, D\. Batra, D\. Parikh, and A\. KembhaviDon’t just assume; look and answer: overcoming priors for visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 4971–4980\.Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p1.1)\.
- Anonymous \(2026\)AnonymousJako tako or fluent? presenting povisle: a polish vision\-language evaluation\.Note:OpenReview, ACL ARR 2026 May SubmissionAccessed: 2026\-07\-08External Links:[Link](https://openreview.net/forum?id=2RvwQgsHZ2)Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p1.1)\.
- Asadiet al\.\(2025\)M\. Asadi, J\. W\. O’Sullivan, F\. Cao, T\. Nedaee, K\. Fardi, F\. Li, E\. Adeli, and E\. AshleyMIRAGE: the illusion of visual understanding in vision\-language models\.Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p4.1),[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p2.1),[§4\.2](https://arxiv.org/html/2608.12928#S4.SS2.p2.1),[§7](https://arxiv.org/html/2608.12928#S7.p3.1)\.
- Balepuret al\.\(2024\)N\. Balepur, A\. Ravichander, and R\. RudingerArtifacts or abduction: how do LLMs answer multiple\-choice questions without the question?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10308–10330\.External Links:[Link](https://aclanthology.org/2024.acl-long.555/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.555)Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p4.1)\.
- Balepuret al\.\(2025\)N\. Balepur, R\. Rudinger, and J\. L\. Boyd\-GraberWhich of these best describes multiple choice evaluation with LLMs? a\) forced B\) flawed C\) fixable D\) all of the above\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3394–3418\.External Links:[Link](https://aclanthology.org/2025.acl-long.169/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.169),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p4.1)\.
- Ben Abachaet al\.\(2020\)A\. Ben Abacha, V\. V\. Datla, S\. A\. Hasan, D\. Demner\-Fushman, and H\. MüllerOverview of the VQA\-Med task at ImageCLEF 2020: visual question answering and generation in the medical domain\.InCLEF 2020 Working Notes,CEUR Workshop Proceedings, Vol\.2696\.External Links:[Link](https://ceur-ws.org/Vol-2696/paper_106.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Ben Abachaet al\.\(2019\)A\. Ben Abacha, S\. A\. Hasan, V\. V\. Datla, J\. Liu, D\. Demner\-Fushman, and H\. MüllerVQA\-Med: overview of the medical visual question answering task at ImageCLEF 2019\.InWorking Notes of CLEF 2019,CEUR Workshop Proceedings, Vol\.2380\.External Links:[Link](https://ceur-ws.org/Vol-2380/paper_272.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Ben Abachaet al\.\(2021\)A\. Ben Abacha, M\. Sarrouti, D\. Demner\-Fushman, S\. A\. Hasan, and H\. MüllerOverview of the VQA\-Med task at ImageCLEF 2021: visual question answering and generation in the medical domain\.InCLEF 2021 Working Notes,CEUR Workshop Proceedings, Vol\.2936,pp\. 1081–1088\.External Links:[Link](https://ceur-ws.org/Vol-2936/paper-87.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Cadeneet al\.\(2019\)R\. Cadene, C\. Dancette, H\. Ben\-Younes, M\. Cord, and D\. ParikhRUBi: reducing unimodal biases in visual question answering\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/51d92be1c60d1db1d2e5e7a07da55b26-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p1.1)\.
- Chenet al\.\(2024\)J\. Chen, C\. Gui, R\. Ouyang, A\. Gao, S\. Chen, G\. H\. Chen, X\. Wang, Z\. Cai, K\. Ji, X\. Wan, and B\. WangTowards injecting medical visual knowledge into multimodal LLMs at scale\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7346–7370\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.418/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.418)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Ciesiółka and Graliński \(2025\)M\. Ciesiółka and F\. GralińskiReVISION: a polish benchmark for evaluating vision\-language models on multimodal national exam data\.In2025 20th Conference on Computer Science and Intelligence Systems \(FedCSIS\),Vol\.,pp\. 665–673\.External Links:[Document](https://dx.doi.org/10.15439/2025F2608)Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p1.1)\.
- Daset al\.\(2024\)R\. Das, S\. Hristov, H\. Li, D\. Dimitrov, I\. Koychev, and P\. NakovEXAMS\-V: a multi\-discipline multilingual multimodal exam benchmark for evaluating vision language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7768–7791\.External Links:[Link](https://aclanthology.org/2024.acl-long.420/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.420)Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p2.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Cited by:[§5](https://arxiv.org/html/2608.12928#S5.p1.1)\.
- Golchin and Surdeanu \(2025\)S\. Golchin and M\. SurdeanuData contamination quiz: a tool to detect and estimate contamination in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 809–830\.External Links:[Link](https://aclanthology.org/2025.tacl-1.37/),[Document](https://dx.doi.org/10.1162/tacl.a.20)Cited by:[§5](https://arxiv.org/html/2608.12928#S5.p1.1)\.
- Google DeepMind \(2026\)Google DeepMindGemma 4 31b it\.Note:[https://huggingface\.co/google/gemma\-4\-31B\-it](https://huggingface.co/google/gemma-4-31B-it)Model card\. Accessed: 2026\-07\-06Cited by:[§4](https://arxiv.org/html/2608.12928#S4.p1.1)\.
- Gouthaman and Mittal \(2020\)K\. V\. Gouthaman and A\. MittalReducing language biases in visual question answering with visually\-grounded question encoder\.InComputer Vision – ECCV 2020,Lecture Notes in Computer Science, Vol\.12358,pp\. 18–34\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-58601-0%5F2)Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p1.1)\.
- Goyalet al\.\(2017\)Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. ParikhMaking the v in vqa matter: elevating the role of image understanding in visual question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6904–6913\.Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p1.1)\.
- Grzybowskiet al\.\(2025\)Ł\. Grzybowski, J\. Pokrywka, M\. Ciesiółka, J\. I\. Kaczmarek, and M\. KubisPolish\-English medical knowledge transfer: a new benchmark and results\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 9042–9063\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.480/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.480),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p1.1),[Limitations](https://arxiv.org/html/2608.12928#Sx1.p3.1)\.
- Hasanet al\.\(2018\)S\. A\. Hasan, Y\. Ling, O\. Farri, J\. Liu, M\. Lungren, and H\. MüllerOverview of the ImageCLEF 2018 medical domain visual question answering task\.InCLEF 2018 Working Notes,CEUR Workshop Proceedings,Avignon, France\.External Links:[Link](https://ceur-ws.org/Vol-2125/paper_212.pdf)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Heet al\.\(2021\)X\. He, Z\. Cai, W\. Wei, Y\. Zhang, L\. Mou, E\. Xing, and P\. XieTowards visual question answering on pathology images\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 708–718\.External Links:[Link](https://aclanthology.org/2021.acl-short.90/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-short.90)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Jassemet al\.\(2025\)K\. Jassem, M\. Ciesiółka, F\. Graliński, P\. Jabłoński, J\. Pokrywka, M\. Kubis, M\. Jabłońska, and R\. StaruchLLMzSzŁ: a comprehensive llm benchmark for polish\.arXiv preprint arXiv:2501\.02266\.Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p1.1)\.
- Lasiket al\.\(2026\)A\. Lasik, J\. Pokrywka, Ł\. Grzybowski, J\. I\. Kaczmarek, G\. Korzańska, J\. Świeczkowski\-Feiz, O\. Pastuszek, P\. Hoffman, J\. T\. Dąbrowski, and W\. KusaReassessing high\-performing llms on polish medical exams: true competence or bias\-driven performance?\.External Links:2606\.12250,[Link](https://arxiv.org/abs/2606.12250)Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p1.1)\.
- Lauet al\.\(2018\)J\. J\. Lau, S\. Gayen, A\. Ben Abacha, and D\. Demner\-FushmanA dataset of clinically generated visual questions and answers about radiology images\.Scientific Data5,pp\. 180251\.External Links:[Document](https://dx.doi.org/10.1038/sdata.2018.251),[Link](https://doi.org/10.1038/sdata.2018.251)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Liuet al\.\(2021\)B\. Liu, L\. Zhan, L\. Xu, L\. Ma, Y\. Yang, and X\. WuSlake: a semantically\-labeled knowledge\-enhanced dataset for medical visual question answering\.In2021 IEEE 18th International Symposium on Biomedical Imaging \(ISBI\),Vol\.,pp\. 1650–1654\.External Links:[Document](https://dx.doi.org/10.1109/ISBI48211.2021.9434010)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- Maliket al\.\(2026\)H\. Malik, H\. T\. Aftab, A\. Rehman, M\. Ali, and W\. SultaniMultilingual hematology visual question answering dataset\.External Links:2606\.25246,[Link](https://arxiv.org/abs/2606.25246)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- Matoset al\.\(2025\)J\. Matos, S\. Chen, S\. K\. V\. Placino, Y\. Li, J\. C\. C\. Pardo, D\. Idan, T\. Tohyama, D\. Restrepo, L\. F\. Nakayama, J\. M\. M\. Pascual\-Leone, G\. K\. Savova, H\. Aerts, L\. A\. Celi, A\. I\. Wong, D\. Bitterman, and J\. GallifantWorldMedQA\-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 7218–7231\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.402/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.402)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- OpenAI \(2026\)OpenAIGPT\-5 Model\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-5](https://developers.openai.com/api/docs/models/gpt-5)API documentation\. Accessed: 2026\-07\-06Cited by:[§4](https://arxiv.org/html/2608.12928#S4.p1.1)\.
- Pokrywkaet al\.\(2024\)J\. Pokrywka, J\. Kaczmarek, and E\. GorzelańczykGPT\-4 passes most of the 297 written polish board certification examinations\.External Links:2405\.01589,[Link](https://arxiv.org/abs/2405.01589)Cited by:[§1](https://arxiv.org/html/2608.12928#S1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4](https://arxiv.org/html/2608.12928#S4.p1.1)\.
- Raj Khanet al\.\(2021\)H\. Raj Khan, D\. Gupta, and A\. EkbalTowards developing a multilingual and code\-mixed visual question answering system by knowledge distillation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Punta Cana, Dominican Republic,pp\. 1753–1767\.External Links:[Link](https://aclanthology.org/2021.findings-emnlp.151/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.151)Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p2.1)\.
- Ramakrishnanet al\.\(2018\)S\. Ramakrishnan, A\. Agrawal, and S\. LeeOvercoming language priors in visual question answering with adversarial regularization\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/67d96d458abdef21792e6d8e590244e7-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p1.1)\.
- Riccioet al\.\(2025\)G\. Riccio, A\. Romano, M\. Barone, G\. M\. Orlando, D\. Russo, M\. Postiglione, V\. L\. Gatta, and V\. MoscatoA multilingual multimodal medical examination dataset for visual question answering in healthcare\.In2025 IEEE 38th International Symposium on Computer\-Based Medical Systems \(CBMS\),pp\. 435–440\.External Links:[Document](https://dx.doi.org/10.1109/CBMS65348.2025.00093),[Link](https://doi.org/10.1109/CBMS65348.2025.00093)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- Statkiewiczet al\.\(2026\)G\. Statkiewicz, A\. Dobrzeniecka, K\. Seweryn, A\. Krasnodębska, K\. Piosek, K\. Bogusz, S\. Cygert, and W\. KusaAnnotation\-efficient vision\-language model adaptation to the Polish language using the LLaVA framework\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 4: Student Research Workshop\),S\. Baez Santamaria, S\. A\. Somayajula, and A\. Yamaguchi \(Eds\.\),Rabat, Morocco,pp\. 569–589\.External Links:[Link](https://aclanthology.org/2026.eacl-srw.44/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-srw.44),ISBN 979\-8\-89176\-383\-8Cited by:[§2\.1](https://arxiv.org/html/2608.12928#S2.SS1.p1.1),[§4](https://arxiv.org/html/2608.12928#S4.p1.1)\.
- Yimet al\.\(2025\)W\. Yim, A\. Ben Abacha, R\. Doerning, C\. Chen, J\. Xu, A\. Subbarao, Z\. Yu, F\. Xia, M\. K\. Hall, and M\. YetisgenWoundcareVQA: a multilingual visual question answering benchmark dataset for wound care\.Journal of Biomedical Informatics170,pp\. 104888\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2025.104888),[Link](https://doi.org/10.1016/j.jbi.2025.104888)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- Yudhistiraet al\.\(2026\)P\. C\. Y\. Yudhistira, D\. R\. Malik, and N\. YudistiraDoes language shift break medical vision\-language models? indonesian radiology visual question answering case study\.Note:Accepted to MMFM\-BIOMED Workshop at CVPR 2026External Links:2606\.03693,[Link](https://arxiv.org/abs/2606.03693)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p2.1)\.
- Zhanet al\.\(2023\)C\. Zhan, P\. Peng, H\. Zhang, H\. Sun, C\. Shang, T\. Chen, H\. Wang, G\. Wang, and H\. WangDebiasing medical visual question answering via counterfactual training\.InMedical Image Computing and Computer Assisted Intervention – MICCAI 2023,Lecture Notes in Computer Science, Vol\.14221,pp\. 382–393\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-43895-0%5F36)Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p2.1)\.
- Zhanget al\.\(2023\)X\. Zhang, C\. Wu, Z\. Zhao, W\. Lin, Y\. Zhang, Y\. Wang, and W\. XiePMC\-VQA: visual instruction tuning for medical visual question answering\.arXiv preprint arXiv:2305\.10415\.External Links:[Link](https://arxiv.org/abs/2305.10415)Cited by:[§2\.2](https://arxiv.org/html/2608.12928#S2.SS2.p1.1)\.
- Zhuet al\.\(2025\)H\. Zhu, Y\. Liu, C\. Zhou, G\. Lu, and B\. ChenMed\-BiasX: robust medical visual question answering with language biases\.InMedical Image Computing and Computer Assisted Intervention – MICCAI 2025,Lecture Notes in Computer Science, Vol\.15973,pp\. 369–378\.Cited by:[§2\.3](https://arxiv.org/html/2608.12928#S2.SS3.p2.1)\.

## Appendix APrompts and Output Format

We used only two versions of the system prompt\. The first version was used when the question text was available, i\.e\., in the C\+Q and C\+Q\+I configurations\. The second version was used when the question text was removed, i\.e\., in the C and C\+I configurations\. The presence of an image did not change the textual prompt: in image\-based configurations, the image was simply attached to the model input together with the same textual prompt used in the corresponding non\-image configuration\.

All experiments were conducted using Polish prompts\. For readability, we also provide English translations below\. The English versions were not used as separate experimental prompts; they are included only as translations of the Polish prompts\.

### A\.1System prompt with question text

This prompt was used for configurations that included the question text, i\.e\., C\+Q and C\+Q\+I\.

##### Polish prompt\.

> Odpowiadasz na pytania jednokrotnego wyboru A–E z testu medycznego dla lekarzy\. Wybierz dokładnie jedną odpowiedź spośród: A, B, C, D, E\. Zwróć wyłącznie poprawny obiekt JSON w następującej strukturze: \{"response": "<LETTER\>"\} Zastąp<LETTER\>jedną wybraną literą: A, B, C, D albo E\. Pole"response"musi być typu string\. Bez wyjaśnień, bez komentarzy, bez dodatkowego tekstu\.

##### English translation\.

> You answer single\-choice questions with options A–E from a medical examination for physicians\. Select exactly one answer from: A, B, C, D, or E\. Return only a valid JSON object with the following structure: \{"response": "<LETTER\>"\} Replace<LETTER\>with one selected letter: A, B, C, D, or E\. The"response"field must be a string\. No explanations, comments, or additional text\.

### A\.2System prompt without question text

This prompt was used for configurations without the question text, i\.e\., C and C\+I\. In these settings, the model received only the answer options and, depending on the configuration, optionally the image\.

##### Polish prompt\.

> Odpowiadasz na pytania jednokrotnego wyboru A–E z testu medycznego dla lekarzy\. Nie otrzymujesz treści pytania\. Masz tylko odpowiedzi A–E\. Mimo braku treści pytania spróbuj wskazać najbardziej prawdopodobną poprawną odpowiedź\. Wybierz dokładnie jedną odpowiedź spośród: A, B, C, D, E\. Zwróć wyłącznie poprawny obiekt JSON w następującej strukturze: \{"response": "<LETTER\>"\} Zastąp<LETTER\>jedną wybraną literą: A, B, C, D albo E\. Pole"response"musi być typu string\. Bez wyjaśnień, bez komentarzy, bez dodatkowego tekstu\.

##### English translation\.

> You answer single\-choice questions with options A–E from a medical examination for physicians\. You do not receive the question text\. You only have the answer options A–E\. Despite the absence of the question text, try to identify the most likely correct answer\. Select exactly one answer from: A, B, C, D, or E\. Return only a valid JSON object with the following structure: \{"response": "<LETTER\>"\} Replace<LETTER\>with one selected letter: A, B, C, D, or E\. The"response"field must be a string\. No explanations, comments, or additional text\.

### A\.3Output schema

For models supporting structured outputs, the response was constrained to a JSON object containing exactly one required field,response\. This field was required to be a string and could take only one of five values corresponding to the answer options: A, B, C, D, or E\. No additional fields were allowed\.

## Appendix BSubset with Human Responses

To place model performance in the context of human performance, we additionally collected anonymized candidate answers published by CEM\. For each PES session, CEM publishes anonymized answer sheets of individual examinees, listing the option selected for each question, together with the official answer key\. Linking these answers to our benchmark is not straightforward\. Our analysis of the published materials revealed that each examination exists in two versions that differ in question numbering, while only one version of the examination sheet is published\. As a consequence, a given question number in the answer statistics does not necessarily correspond to the same question in the published sheet\.

To ensure correct alignment, we merged the candidate answers with our question set using the official correct answer as a consistency check\. For every matched item, the expected correct answer in the published sheet had to agree with the correct answer reported in the answer statistics\. Since a single question can match by chance, we further restricted the human answers subset to examinations for which more than one question from our benchmark was available, and required the expected correct answers to match for all available questions from that examination\. We excluded examinations that did not satisfy this condition, as we could not reliably determine which version they corresponded to\.

This procedure yielded a subset of questions with associated human answers, for which we computed the accuracy of examinees\. Since this subset is smaller than the full benchmark, we treated the resulting value as an approximate human reference point rather than an item\-level comparison with model accuracy\.

## Appendix CUsage of GenAI in Research

We used ChatGPT and Codex to assist with manuscript writing, code development, and literature searches\. All generated content and outputs were reviewed and verified by the authors, who remain fully responsible for the final manuscript\.