You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models

# You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models
Source: [https://arxiv.org/html/2608.18116](https://arxiv.org/html/2608.18116)
Andrea Morales\-Garzón1Salvador Lopez\-Joya1Miguel López\-Pérez1 Maria J\. Martin\-Bautista1 1Dept\. Computer Science and Artificial Intelligence, University of Granada amoralesg@decsai\.ugr\.es, slopezjoya@ugr\.es, \{mlopez,mbautis\}@decsai\.ugr\.es

## 1Introduction

Vision–language models \(VLMs\) such as CLIP\[[17](https://arxiv.org/html/2608.18116#bib.bib2)\]and SigLIP\[[21](https://arxiv.org/html/2608.18116#bib.bib12)\]enable zero\-shot classification by aligning images with natural language descriptions of classes\. In this setting, the formulation of text prompts plays a critical role in model performance\. A widely adopted strategy to improve robustness is*prompt ensembling*\[[10](https://arxiv.org/html/2608.18116#bib.bib27)\], where multiple prompt templates are used to predict each class and their outputs are aggregated\. The underlying assumption is that a diverse prompt pool can mitigate variability introduced by weaker individual prompts\[[16](https://arxiv.org/html/2608.18116#bib.bib28)\]\.

In practice, prompt ensembling is not always beneficial\. When VLMs trained on general\-purpose data are applied to specialized domains, prompt engineering becomes substantially more challenging\. In preliminary experiments, it has been observed that adding more prompts can degrade performance, particularly when the pool contains poorly aligned or noisy templates\[[13](https://arxiv.org/html/2608.18116#bib.bib7)\]\. This raises a fundamental question:*how can we make prompt ensembling robust to prompt quality, and which prompts deserve to be trusted?*

Zero\-shot Prompt Ensembling \(ZPE\) addresses this by assigning higher weights to prompts with stronger discriminative signal, estimated without any external validation set\[[1](https://arxiv.org/html/2608.18116#bib.bib1)\]\. While ZPE has been shown to improve accuracy over uniform ensembling in general settings, its behavior in domain\-shifted scenarios remains underexplored\. In particular, it is unclear whether the disagreement signal that drives ZPE carries meaningful information when the model operates far from its training distribution\.

In this study, we investigate these questions across specific agrifood domains, including food classification and agriculture\-related tasks such as plant disease recognition\. We treat food\-related datasets as*in\-distribution*\(ID\) proxies based on two observations: \(i\) CLIP’s zero\-shot accuracy on food datasets such as Food\-101 approaches fully supervised performance \(Radford et al\., 2021\), indicating strong coverage of food imagery in web\-crawled pretraining corpora; and \(ii\) platforms heavily represented in these corpora \(e\.g\., Pinterest, Shopify\) are dominated by food and consumer content, whereas agricultural pathology imagery remains confined to specialist repositories, making it naturally*out\-of\-distribution*\(OOD\) for these models\. Our own results corroborate this, as we observe robust performance in food\-domain benchmarks but notable disparities in OOD agricultural and bean disease datasets, highlighting the sensitivity of VLMs to domain\-specific vocabulary and fine\-grained visual variation\. Our empirical study reveals a nuanced picture: ZPE provides limited benefit under ID conditions, where uniform ensembling already performs well, but becomes substantially more useful under OOD conditions, where domain shift is severe\.

Building on this finding, we introducePID\(Prompt\-based Inconsistency Detection\), a method that repurposes prompt disagreement within the ZPE ensemble as a proxy for epistemic uncertainty\. PID serves as a failure detector that improves model reliability under domain shift without modifying the prediction pipeline or incurring additional computational cost\. By gating predictions through disagreement\-based uncertainty estimates, PID offers a principled way to flag unreliable outputs precisely in the OOD regime where standard confidence measures tend to break down\.

The contributions of this work are:

- •We evaluate ZPE across multiple agrifood domains under both ID and OOD conditions, showing that its benefit is regime\-dependent: marginal under ID but substantial under OOD data\.
- •We identify consistent performance disparities in OOD settings, particularly in agriculture and bean disease benchmarks, and contrast these with robust ID results in food classification\.
- •We introduce PID, a prompt disagreement measure that acts as a quality\-gated epistemic uncertainty signal within the ZPE framework, and demonstrate its effectiveness as a reliability proxy for OOD selective prediction\.
- •We analyze from a lexical perspective the prompt quality, showing that terms referring to concrete visual content are associated with higher ZPE scores, while more abstract or weakly visual words tend to correspond to lower\-quality prompts\.

## 2Related Work

Recent progress in VLMs, particularly since the introduction of CLIP\[[17](https://arxiv.org/html/2608.18116#bib.bib2)\], has shown that strong zero\-shot performance depends not only on the pretrained model, but also on how we interact with it through language\. This has motivated a growing line of work exploring how to design, select, and aggregate prompts, as well as how to assess the reliability of their predictions\. In this section, we review prior research on prompt engineering for VLMs, calibration in zero\-shot VLMs, agrifood image classification, and uncertainty estimation in ensembles\.

#### Prompt engineering for VLMs\.

Prompt design plays a critical role in zero\-shot performance, as different formulations of the same class can lead to different predictions\. Menon & Vondrick proposed classification by description, replacing class\-name prompts with a set of descriptors automatically generated with an LLM\. Instead of querying “a photo of a tiger”, the model checks for discriminative features \(stripes, claws, size\)\. This produces better accuracy and interpretability, since each classification decision is grounded in verifiable visual attributes\[[11](https://arxiv.org/html/2608.18116#bib.bib3)\]\. Learned prompt tuning methods such as CoOp and CoCoOp\[[24](https://arxiv.org/html/2608.18116#bib.bib5),[23](https://arxiv.org/html/2608.18116#bib.bib4)\]replace fixed templates with learnable context vectors, improving accuracy on seen classes at the cost of requiring labeled data and reduced zero\-shot generalizability — limitations that ZPE avoids by operating entirely without supervision\. ZPE assigns weights to prompts in a large pool based on their discriminative signal and produces a weighted average ensemble prediction\[[1](https://arxiv.org/html/2608.18116#bib.bib1)\]\. In our work, we build on ZPE and extend it to the agrifood domain, analyzing prompt quality from a lexical perspective and using prompt disagreement as an uncertainty signal for failure detection\.

#### Calibration in zero\-shot models\.

Calibration has been widely studied in modern neural networks, where it is often observed that high accuracy does not necessarily imply well\-calibrated predictions\. Deep neural networks are shown to be systematically overconfident, a phenomenon that becomes more pronounced with increased model depth, width, batch normalisation, and weight decay\[[5](https://arxiv.org/html/2608.18116#bib.bib8)\]\. Temperature scaling is introduced as a simple yet highly effective post\-hoc method that rescales logits with a single parameter, improving calibration without affecting classification accuracy\. More recent work has revisited these findings in the context of large\-scale vision models\. Minderer et al\. evaluated several architectures, including Vision Transformers, MLP\-Mixers, and large pretrained models such as CLIP\[[12](https://arxiv.org/html/2608.18116#bib.bib11)\], showing that modern architectures are often significantly better calibrated and that calibration can even improve with model scale under distribution shift\. In our work, we do not perform explicit calibration, but we show that prompt disagreement within ZPE correlates with unreliable predictions under domain shift, offering a complementary and label\-free reliability signal\.

#### Agrifood image classification\.

Recent work has begun to explore the application of vision\-language models in agricultural and food\-related domains, where fine\-grained recognition, domain shift, and limited supervision pose significant challenges\. Studies such as AgriCLIP\[[14](https://arxiv.org/html/2608.18116#bib.bib14)\]and\[[7](https://arxiv.org/html/2608.18116#bib.bib15)\]show that although CLIP and BLIP\-2\[[9](https://arxiv.org/html/2608.18116#bib.bib24)\]provide strong baselines, their performance degrades significantly in tasks requiring fine\-grained agricultural expertise, environmental robustness, or reasoning over contextual information\. Similarly, E\-CLIP\[[22](https://arxiv.org/html/2608.18116#bib.bib16)\]and\[[6](https://arxiv.org/html/2608.18116#bib.bib17)\]propose hybrid architectures that combine CLIP embeddings with detection heads or structured botanical features to improve robustness in crop and plant disease recognition\.

To address the lack of domain\-specific data, several multimodal datasets have been proposed, including VL\-PAW\[[19](https://arxiv.org/html/2608.18116#bib.bib18)\], AgriCoT\[[18](https://arxiv.org/html/2608.18116#bib.bib19)\], WisWheat\[[20](https://arxiv.org/html/2608.18116#bib.bib20)\], and AgMMU\[[4](https://arxiv.org/html/2608.18116#bib.bib21)\], which introduce image–text pairs, reasoning tasks, and expert\-level annotations for agricultural decision\-making\.

For evaluation, Food\-101\[[2](https://arxiv.org/html/2608.18116#bib.bib22)\]remains the standard benchmark for fine\-grained food recognition, comprising 101 categories with substantial intra\-class variability due to diverse real\-world imaging conditions\. Despite its relative simplicity, it serves as a key reference for vision\-language models\. Beans, for instance, is a three\-class plant disease dataset \(angular leaf spot, bean rust, and healthy\) with a pronounced domain gap, as its labels require specialized botanical knowledge largely absent from general\-purpose pretraining\. As a result, zero\-shot models often perform close to chance, making it a useful stress test for robustness under distribution shift\.

#### Epistemic uncertainty in ensembles\.

Ensemble disagreement has been established as a practical proxy for epistemic uncertainty, capturing the model’s lack of confidence due to insufficient or mismatched training signal\[[8](https://arxiv.org/html/2608.18116#bib.bib25)\]\. The diversity among ensemble members is key to this property: more diverse predictions correlate with higher uncertainty, particularly under distribution shift\[[3](https://arxiv.org/html/2608.18116#bib.bib10)\]\. Critically, ensemble\-based uncertainty estimates are known to degrade gracefully but detectably under dataset shift, making disagreement a reliable signal precisely in the OOD regime\[[15](https://arxiv.org/html/2608.18116#bib.bib9)\]\. In the context of prompt ensembling, we interpret disagreement across prompt predictions as an indicator of epistemic uncertainty under domain shift: when a model operates far from its training distribution, prompts probing the same class from different linguistic angles are less likely to agree, producing a detectable inconsistency signal\. PID operationalizes this intuition directly, without requiring additional training or distributional assumptions\.

## 3Methodology

### 3\.1Zero\-Shot classification

Given an imagexxand a set of class names\{c1,…,cC\}\\\{c\_\{1\},\\ldots,c\_\{C\}\\\}, a VLM encodes the image into𝐯=fθ​\(x\)∈ℝD\\mathbf\{v\}\\\!=\\\!f\_\{\\theta\}\(x\)\\\!\\in\\\!\\mathbb\{R\}^\{D\}and each text prompt into𝐭p,c=gϕ​\(tmplp​\(c\)\)∈ℝD\\mathbf\{t\}\_\{p,c\}\\\!=\\\!g\_\{\\phi\}\(\\mathrm\{tmpl\}\_\{p\}\(c\)\)\\\!\\in\\\!\\mathbb\{R\}^\{D\}\. The logit of classccunder promptppis:

Lc\(p\)​\(x\)=α⋅⟨𝐯​\(x\),𝐭p,c⟩,L^\{\(p\)\}\_\{c\}\(x\)=\\alpha\\cdot\\langle\\mathbf\{v\}\(x\),\\,\\mathbf\{t\}\_\{p,c\}\\rangle,\(1\)whereα\\alphais a learned logit scale, and the single\-prompt prediction isc^\(p\)=arg⁡maxc⁡Lc\(p\)​\(x\)\\hat\{c\}^\{\(p\)\}\\\!=\\\!\\arg\\max\_\{c\}L^\{\(p\)\}\_\{c\}\(x\)\. We evaluate two VLM families: CLIP\[[17](https://arxiv.org/html/2608.18116#bib.bib2)\]uses a contrastive loss \(α≈100\\alpha\\\!\\approx\\\!100\); SigLIP\[[21](https://arxiv.org/html/2608.18116#bib.bib12)\]uses a sigmoid loss \(α≈117\\alpha\\\!\\approx\\\!117\), removing the dependence on batch normalisation\.

### 3\.2Zero\-Shot Prompt Ensembling \(ZPE\)

Prompt choice significantly impacts zero\-shot accuracy, yet practitioners often select templates arbitrarily\. In\[[1](https://arxiv.org/html/2608.18116#bib.bib1)\], the authors proposed ZPE, a self\-supervised weighting scheme that assigns higher importance to prompts with stronger discriminative signal, without requiring any labels\.

Given a pool ofPPprompts evaluated overNNtest images, ZPE computes a scalar quality score per prompt over the full test set, then uses those scores to weight the ensemble\. This is atransductiveprocedure: allNNimages must be seen before any individual prediction is made\.

#### Prompt scoring\.

Two scoring variants are proposed\[[1](https://arxiv.org/html/2608.18116#bib.bib1)\]:

\(1\) Raw\(Algorithm 1\)\. The score is the mean maximum logit across the test set:

sraw\(p\)=1N​∑i=1Nmaxc⁡Lc\(p\)​\(xi\)\.s^\{\(p\)\}\_\{\\mathrm\{raw\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\max\_\{c\}\\;L^\{\(p\)\}\_\{c\}\(x\_\{i\}\)\.\(2\)
\(2\) Normalized\(Algorithm 2\)\. Logits are first centred by subtracting the expected logit under the data distribution, approximated by the test\-set mean:

snorm\(p\)=1N​∑i=1Nmaxc⁡\(Lc\(p\)​\(xi\)−μ^c\(p\)\),s^\{\(p\)\}\_\{\\mathrm\{norm\}\}=\\frac\{1\}\{N\}\\\!\\sum\_\{i=1\}^\{N\}\\max\_\{c\}\\bigl\(L^\{\(p\)\}\_\{c\}\(x\_\{i\}\)\-\\hat\{\\mu\}^\{\(p\)\}\_\{c\}\\bigr\),\(3\)whereμ^c\(p\)=1N​∑iLc\(p\)​\(xi\)\\hat\{\\mu\}^\{\(p\)\}\_\{c\}\\\!=\\\!\\frac\{1\}\{N\}\\sum\_\{i\}L^\{\(p\)\}\_\{c\}\(x\_\{i\}\)estimates𝔼​\[Lc\(p\)\]\\mathbb\{E\}\[L^\{\(p\)\}\_\{c\}\]\. This centring removes the unconditional logit bias of each prompt, making scores comparable across templates\.

#### Weighted prediction\.

Scores are converted into prompt weights via a softmax with temperatureτ\>0\\tau\\\!\>\\\!0, and the ensemble prediction is:

c^ZPE=arg⁡maxc​∑p=1Pwp⋅Lc\(p\)​\(x\),\\hat\{c\}^\{\\mathrm\{ZPE\}\}=\\arg\\max\_\{c\}\\sum\_\{p=1\}^\{P\}w\_\{p\}\\cdot L^\{\(p\)\}\_\{c\}\(x\),\(4\)wp=es\(p\)/τ∑p′es\(p′\)/τ\.w\_\{p\}=\\frac\{e^\{s^\{\(p\)\}/\\tau\}\}\{\\sum\_\{p^\{\\prime\}\}e^\{s^\{\(p^\{\\prime\}\)\}/\\tau\}\}\.\(5\)Asτ→∞\\tau\\\!\\to\\\!\\infty, Eq\.[4](https://arxiv.org/html/2608.18116#S3.E4)recovers uniform ensembling; smallτ\\tauconcentrates weight on the best\-scoring prompt\. We selectτ∗\\tau^\{\*\}by sweeping a grid between 0\.1 and 10\.0 and choosing the value that maximizes classification accuracy on the held\-out validation split\.

### 3\.3Prompt\-based Inconsistency Detection \(PID\)

ZPE collapses the ensemble into a single point prediction, discarding the information encoded in prompt*disagreement*\. We propose PID, which repurposes this disagreement as a per\-sample failure detector: given imagexx, how likely is the model to be wrong?

PID adds a confidence scoreκ​\(x\)\\kappa\(x\)that enablesselective prediction: predict on high\-κ\\kappasamples, abstain on low\-κ\\kappaones\. It does*not*change the predicted class or its probabilities\.

#### Quality\-gated disagreement\.

Letqp​\(x\)=softmax​\(L\(p\)​\(x\)\)q\_\{p\}\(x\)\\\!=\\\!\\mathrm\{softmax\}\(L^\{\(p\)\}\(x\)\)be the class distribution of promptpp, andμ​\(x\)=∑pwp​qp​\(x\)\\mu\(x\)\\\!=\\\!\\sum\_\{p\}w\_\{p\}\\,q\_\{p\}\(x\)the ZPE\-norm BMA mean\. The quality\-gated disagreement is:

D​\(x\)=∑p=1Pwpagr⋅KL​\(qp​\(x\)∥μ​\(x\)\),D\(x\)=\\sum\_\{p=1\}^\{P\}w\_\{p\}^\{\\mathrm\{agr\}\}\\cdot\\mathrm\{KL\}\\\!\\bigl\(q\_\{p\}\(x\)\\;\\\|\\;\\mu\(x\)\\bigr\),\(6\)wherewpagr=softmax​\(snorm\(p\)/τa\)w\_\{p\}^\{\\mathrm\{agr\}\}\\\!=\\\!\\mathrm\{softmax\}\(s^\{\(p\)\}\_\{\\mathrm\{norm\}\}/\\tau\_\{a\}\)uses a separate temperatureτa\\tau\_\{a\}\. Whenτa<τ\\tau\_\{a\}\\\!<\\\!\\tau, only high\-scoring prompts contribute to the disagreement measurement \(*quality\-gating*\)\. Whenτa→∞\\tau\_\{a\}\\\!\\to\\\!\\infty, all prompts contribute equally, recovering plain mutual information\.

#### Dirichlet concentration\.

The per\-sample confidence is modeled as:κ​\(x\)=cD​\(x\),\\kappa\(x\)=\\frac\{c\}\{D\(x\)\},wherec\>0c\\\!\>\\\!0is a scalar\. Highκ​\(x\)\\kappa\(x\): prompts agree→\\toprediction likely correct\. Lowκ​\(x\)\\kappa\(x\): prompts disagree→\\toabstain or escalate\.

#### Fitting\.

Bothτa\\tau\_\{a\}andccare fitted on a held\-out validation split \(20% of the evaluation set, sampled before scoring\)\.τa∗\\tau\_\{a\}^\{\*\}is chosen to maximize AUROC ofκ​\(x\)\\kappa\(x\)as a binary correctness predictor;c∗c^\{\*\}minimizes Dirichlet\-Categorical NLL\.

#### Role ofτa\\tau\_\{a\}\.

The shape of theτa\\tau\_\{a\}ablation curve is itself a domain diagnostic\. A*peaked*curve \(optimal at lowτa\\tau\_\{a\}\) indicates that quality\-gating is useful: some prompts are reliably better than others\. A*monotonically increasing*curve indicates full domain shift: no prompt is trustworthy, so collective disagreement across all prompts is the most informative signal\.

### 3\.4Calibration Metrics

We also measure the standard calibration quality of the ensemble:

- •ECE: average gap between confidence and accuracy in 15 equal\-width bins\. It measures how well calibrated the model is\.
- •NLL: computes the negative log\-loss\. It retains information about how good the predicted probabilities are,−1N​∑nlog⁡pc^​\(xn\)\-\\frac\{1\}\{N\}\\sum\_\{n\}\\log p\_\{\\hat\{c\}\}\(x\_\{n\}\)\.
- •Brier Score: it is the mean\-squared error between probabilities and real labels\. It does not penalize wrong predictions as much as NLL,1N​∑n‖𝐩​\(xn\)−𝐲n‖2\\frac\{1\}\{N\}\\sum\_\{n\}\\\|\\mathbf\{p\}\(x\_\{n\}\)\-\\mathbf\{y\}\_\{n\}\\\|^\{2\}\.

## 4Experimental setup

### 4\.1Agrifood datasets

We evaluate on four benchmarks spanning food and agricultural image classification\. All class names are preprocessed to replace underscores with spaces\. This step is critical for beans: naive underscore names degrade accuracy by∼\\sim30 percentage points due to tokenization mismatch in the text encoder\.

- •Food\-101101 food categories, 1,000 test images per class\.
- •Food\-1111 broad food groups, 1,000 samples per class\.
- •Agricultural crops30\-category crop image dataset \(829 images\), with classes such as rice, maize, or banana, designed for coarse\-grained crop classification\.
- •BeansBean leaf disease classification \(angular leaf spot, bean rust, healthy\)\.

### 4\.2Models

We evaluate six models spanning two architectures and three scales: SigLIP\[[21](https://arxiv.org/html/2608.18116#bib.bib12)\]\(base, SO/400M, large\) and CLIP\[[17](https://arxiv.org/html/2608.18116#bib.bib2)\]\(B/32, B/16, L\)\. This selection enables systematic comparison across both architectural paradigms and model capacity, allowing us to assess how design choices in vision\-language modeling affect performance on our tasks\. SigLIP, which introduces sigmoid\-based loss as an alternative to softmax\-based contrastive learning, represents recent advances in efficient vision\-language alignment, while CLIP remains the de facto standard in the field\. By evaluating models at base, intermediate, and large scales within each architecture, we characterize the scaling behavior and trade\-offs between model size and performance across different representation learning frameworks\.

### 4\.3Prompt pools

We used four prompt pools: two general ones from the study detailed in\[[1](https://arxiv.org/html/2608.18116#bib.bib1)\], and two domain\-specific pools that we designed to provide visual angles not covered by generic templates\. Lexical diversity and domain\-specific vocabulary varies across pools, motivating the analysis in Section[5\.2](https://arxiv.org/html/2608.18116#S5.SS2)\. They are available in the code repository111Code and supplementary materials are available at:https://github\.com/ugritai/pid\_agrifood\.\.

- •Pool\-247: 247 templates with standard photographic and descriptive templates\.
- •Pool\-426: it extends Pool\-247 with 179 ChatGPT\-generated prompts\.
- •Pool\-Food: \(ours, 51 prompts\)templates targeting food appearance, culinary context and texture, for the Food\-101 and Food\-11 datasets\. Examples: “a plate of \{\}\.”, “a close\-up of the texture of \{\}\.”, “street food: \{\}\.”\.
- •Pool\-Agri \(ours, 52 prompts\): templates targeting plant parts, disease symptoms, and agronomic context, designed for the Agriculture and Beans datasets\. Examples: “a close\-up photo of a \{\} leaf\.”, “a photo of \{\} showing rust disease\.”\.

## 5Results

Table 1:Average accuracy across pools using a uniform ensemble\.### 5\.1Performance and calibration

![Refer to caption](https://arxiv.org/html/2608.18116v1/tg_image_3258986786.png)Figure 1:Zero\-shot performance for SigLIP SO400M and CLIP ViT\-L/14 \(all prompt pools\)\.Table[1](https://arxiv.org/html/2608.18116#S5.T1)reports the results of the VLM backbones \(CLIP and SigLIP\) averaging over the different prompt pools in a uniform ensemble\. We observe that CLIP\-L/14 and SigLIP\-400M are consistently the best of their family, respectively\. Therefore, to simplify the subsequent performance analysis we only report the results for these two backbones\.

Fig\.[1](https://arxiv.org/html/2608.18116#S5.F1)shows how performance evolves from uniform prompt ensembling \(Uniform\) to ZPE\-weighted ensembling \(ZPE, ZPE\-norm\) under three prompt pools: Pool\-247 \(dashed\), Pool\-426 \(dotted\), and the domain\-specific pool \(solid\)\. Rows report accuracy \(↑\), ECE \(↓\), NLL \(↓\), and Brier score \(↓\); columns correspond to datasets\. Overall, SigLIP SO400M \(orange\) consistently outperforms CLIP ViT\-L/14 \(blue\)\.

Our evaluation reveals distinct patterns across ID and OOD datasets\. On well\-aligned ID domains, both architectures achieve high accuracy and perform similarly with minimal calibration error, and their performance remains robust across prompt ensemble strategies\. However, on OOD Agriculture domain—proxies for increasing domain shift—both models exhibit higher variation across models and prompt pools, particularly in calibration metrics\. In the challenging Beans dataset specifically, we observe dramatic performance degradation and severe miscalibration, with performance becoming highly sensitive to prompt pool and weighting strategy choices\. Notably, ZPE\-weighted strategies begin to show differentiated behavior in this regime, suggesting that prompt reweighting mechanisms are beneficial when models encounter domain\-shifted data\. This behavior highlights the critical importance of uncertainty quantification and out\-of\-distribution detection mechanisms: models maintain well\-calibrated confidence on in\-distribution data while becoming dangerously overconfident under domain shift\. The consistency of these degradation patterns across both SigLIP and CLIP variants—despite SigLIP’s superior in\-distribution performance—suggests that the bottleneck is fundamentally a domain shift problem rather than an architectural limitation, motivating the need for explicit OOD detection frameworks that can reliably flag when visual inputs deviate significantly from the training distribution\.

### 5\.2Lexical analysis of prompt quality

#### ZPE score as a signal of prompt quality\.

An important property of ZPE is that prompt scores are computed without using ground\-truth labels\. Instead, the score reflects how much the ensemble prediction changes when a given prompt is considered with respect to the ensemble mean\. To assess whether this signal is meaningful, we compare the ZPE\-normalized score of each prompt with its single\-prompt accuracy\.

An example of this finding is the prompt“a fundus image with signs of\{\}”, which ranks among the worst prompts in all four datasets\. Its ZPE scores are consistently low \(6\.4 in Food\-101, 2\.4 in Food\-11, 4\.9 in Agriculture, and 0\.8 in Beans\), and this is matched by low accuracies \(0\.48, 0\.60, 0\.60, and 0\.37, respectively\)\. This suggests that ZPE is able to identify strongly mismatched prompts without requiring access to labels\. This is particularly relevant in agrifood settings, where tasks span heterogeneous visual domains, from plated food images to crop and plant pathology\. In this context, prompt pools may contain templates that are semantically plausible but visually mismatched\. Here, ZPE is useful to detect prompt quality and discard out\-of\-scope prompts\.

#### Word\-level discriminability\.

To better understand which lexical properties are associated with prompt quality, we compute a discriminability score for each word as the log\-ratio of its frequency in the top\-25% versus bottom\-25% of prompts ranked by ZPE\-norm score \(see Fig\.[2](https://arxiv.org/html/2608.18116#S5.F2)\)\. Regarding food\-101 and Agriculture datasets, the words with the highest positive discriminability are mainly related to viewpoint and image framing, such asaerial,view,satelliteandoverhead\. This suggests that prompts describing a concrete visual perspective tend to be more useful in these datasets\. In contrast, words with negative discriminability are often less visually grounded\. In Food\-101, terms such astype,sign,shapes, andobjectsare associated with poorer prompts, while in Agriculture, low\-scoring prompts often include words such asdemonstration,doing, andperforming\.

![Refer to caption](https://arxiv.org/html/2608.18116v1/x1.png)Figure 2:Word discriminability of top vs\. bottom 25% ZPE\-ranked prompts per dataset \(SigLIP\-SO400M, Pool\-247\)\.![Refer to caption](https://arxiv.org/html/2608.18116v1/fixed_lexical.png)Figure 3:Cross\-domain ZPE\-norm prompt scores: Food vs\. Agriculture \(SigLIP\-SO400M, Pool\-247\)\.ρ=0\.765\\rho=0\.765indicates strong cross\-domain prompt quality transferability\.![Refer to caption](https://arxiv.org/html/2608.18116v1/uncertainty_final.png)Figure 4:PID\-based uncertainty evaluation across all prompt pools\.
#### Cross\-domain effects: food vs\. agriculture\.

Since Pool\-247 is a generic pool, we also analyze whether ZPE\-norm captures domain preferences when comparing Food\-101 and Agriculture\. Fig\.[3](https://arxiv.org/html/2608.18116#S5.F3)plots the ZPE score of each prompt in the food domain against its score in the agriculture domain\. The two score vectors are strongly correlated \(ρ=0\.765\\rho=0\.765\), indicating that many prompts work well for both domains, which make sense, since they are relatively close topics\.

In addition, prompts that are exclusive to the top\-30 of each domain reveal a consistent lexical pattern\. Food\-specific prompts more frequently use definite constructions \(e\.g\., “aerial photo ofthe\{\}” or “satellite photo ofthe\{\}”\), whereas agriculture\-specific prompts tend to use indefinite forms \(e\.g\., “aerial photo ofa\{\}”, “aerial view ofa\{\}”\)\. While we do not explicitly model linguistic phenomena, this pattern suggests that prompt effectiveness may be sensitive to subtle grammatical choices \(e\.g\., the/a\)\. One possible explanation is that definite and indefinite forms induce different visual expectations, which may interact with how categories are represented across these two domains\. Again, note that this effect is captured without access to domain labels\.

Overall, this analysis indicates that ZPE does not only separate clearly poor prompts from useful ones, but also reflects more subtle lexical and domain\-specific effects\.

### 5\.3Uncertainty estimation and selective classification

Fig\.[4](https://arxiv.org/html/2608.18116#S5.F4)shows the uncertainty as a proxy to detect prediction errors measured with AUROC\. We compare both maximum confidence in the ensemble \(max\-conf\) and PID\. Rather than a universal replacement for max\-conf, we present PID as a complementary signal whose value is regime\-dependent\. On ID datasets, \(Food\-101 and Food\-11\), max\-conf is a strong baseline for correctness prediction \(AUROC 0\.88–0\.94 depending on model\)\. In this regime \(where the model operates near its training distribution\) the model’s own confidence already reflects when it will fail\. PID adds no substantial benefit here: the model already knows what it does not know\.

The picture changes sharply when the domain is far from pretraining, i\.e\., OOD datasets\. On Agriculture, PID values overcome max\-conf in several cases to detect errors\. On Beans, max\-conf collapses as a predictor of correctness \(AUROC 0\.27–0\.72 depending on model\)\. The model’s confidence no longer reflects visual evidence—it reflects lexical biases in the embedding space\. PID recovers a useful signal: for CLIP\-B/16 on Beans \(Pool\-247\), PID achieves AUROC = 0\.646 vs\. max\-conf AUROC = 0\.275, a gap of\+0\.37\+0\.37\(the largest in the entire experiment\)\. The quality\-gating mechanism is key: since CLIP has no representation of plant diseases, no single prompt is reliably good\. But the*collective*disagreement among all prompts, captured by a largeτa\\tau\_\{a\}, faithfully reflects the model’s errors\.

### 5\.4τa\\tau\_\{a\}as a domain diagnostic

Fig\.[5](https://arxiv.org/html/2608.18116#S5.F5)shows PID AUROC as a function ofτa\\tau\_\{a\}for each dataset\. The shape of this curve is itself informative:

- •Peaked curve\(Food\-101,τa∗≈0\.2\\tau\_\{a\}^\{\*\}\\approx 0\.2–0\.50\.5\): quality\-gating helps\. Best prompts define the disagreement signal; including all prompts equally introduces noise\.
- •Monotonically increasing curve\(Beans with CLIP,τa∗=1\.0\\tau\_\{a\}^\{\*\}=1\.0, plateau at 100\): no quality\-gating needed\. All prompts are equally weak, so collective disagreement is the only reliable signal; concentrating on the “best” prompts \(which are no better than the rest\) would discard information\.

Theτa\\tau\_\{a\}curve serves as a*domain\-shift diagnostic*: a peaked curve indicates the VLM has some grounding in the domain; a monotonically increasing curve indicates domain shift\.

![Refer to caption](https://arxiv.org/html/2608.18116v1/x2.png)Figure 5:PID AUROC as a function ofτa\\tau\_\{a\}across all datasets and prompt pools\. Peaked curves \(Food\) indicate quality\-gating is beneficial; monotonically increasing curves \(Beans with CLIP\) indicate full domain shift\.

## 6Conclusion

We have studied zero\-shot prompt ensembling in the agrifood domain and shown that ZPE works as an unsupervised domain\-alignment detector: without any labels, it assigns higher scores to prompts whose vocabulary matches the visual domain\. This explains why small domain\-specific pools consistently outperform large generic ones, and directly motivates PID, a quality\-gated uncertainty method that repurposes prompt disagreement to predict model failures\.

PID is used to measure the epistemic uncertainty in the disagreement, incorporating a hyperparameter to only compute the disagreement in high\-quality prompts\. We use this uncertainty for selective classification, finding that PID estimates errors reliably and competitively with max\-conf under mild shift, and provides tighter error bounds under severe domain shift\. Theτa\\tau\_\{a\}ablation curve serves as a practical diagnostic: its shape reveals if the model is domain\-grounded or is operating under full domain shift\.

These contributions provide a more complete picture of when and why prompt ensembling works in agrifood domains, and offer practical guidance for prompt pool design in specialised visual domains, and how quality\-gating impacts performance, calibration, and selective prediction\.

#### Future work\.

There are still several open questions that we may address in future work\. For example, why do semantically weak prompts score high? An interesting observation is that some prompts with seemingly weak or unrelated semantics still appear among high\-scoring prompts\. This suggests that prompt effectiveness is not solely driven by explicit semantic alignment\. One possibility is that they capture indirect visual cues or regularization effects in the model’s representation space\. Understanding these mechanisms remains an open question for future work\. In another vein, the role of PID has to be studied more in\-depth in a wider setting to fully unveil, when epistemic uncertainty derived from disagreement can improve max\-conf as a proxy for reliability in the outcomes\. Finally, our analysis is restricted to English, given the predominantly English\-trained encoders used here\. Extending it to morphologically richer languages such as Spanish, where phenomena like grammatical gender and noun–adjective agreement may affect prompt consistency, is a promising direction, requiring multilingual VLMs and designing Spanish prompt pools\.

#### Limitations\.

Optimalτ∗\\tau^\{\*\},τa∗\\tau\_\{a\}^\{\*\}andccvalues are fitted on validation data, which deviates from a fully zero\-shot setting\. Future work could explore data\-free approaches to temperature selection\. Additionally, the Beans dataset shows near\-random performance with CLIP models, likely due to its small size, class imbalance, and the large domain gap from pretraining; this makes it a useful stress\-test but limits conclusions about PID on larger out\-of\-domain benchmarks\. We also assume that food\-related data is more extensively represented in pre\-trained VLMs due to the wide availability of food\-related content on the web and in training datasets, in contrast to agriculture, which remains a less explored domain\. Further large\-scale studies are required to validate this assumption\.

## Acknowledgments

Work supported by the FederaTrans project: Grant PID2024\-158373OB\-I00 funded by MICIU/AEI/10\.13039/501100011033 and by ERDF/EU\. It is also funded by the European Union \(BAG\-INTEL project, grant agreement no\. 101121309 and CUSTOMAI project, grant agreement no\. 101226029\)\.

## References

- \[1\]J\. U\. Allingham, J\. Ren, M\. W\. Dusenberry, X\. Gu, Y\. Cui, D\. Tran, J\. Z\. Liu, and B\. Lakshminarayanan\(2023\)A simple zero\-shot prompt weighting technique to improve prompt ensembling in text\-image models\.InInternational Conference on Machine Learning,pp\. 547–568\.Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p3.1),[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.18116#S3.SS2.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.18116#S3.SS2.p1.1),[§4\.3](https://arxiv.org/html/2608.18116#S4.SS3.p1.1)\.
- \[2\]\(2014\)Food\-101–mining discriminative components with random forests\.InEuropean conference on computer vision,pp\. 446–461\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p3.1)\.
- \[3\]S\. Fort, H\. Hu, and B\. Lakshminarayanan\(2019\)Deep ensembles: a loss landscape perspective\.arXiv preprint arXiv:1912\.02757\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px4.p1.1)\.
- \[4\]A\. Gauba, I\. Pi, Y\. Man, Z\. Pang, V\. S\. Adve, and Y\. Wang\(2025\)AGMMU: a comprehensive agricultural multimodal understanding benchmark\.arXiv preprint arXiv:2504\.10568\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p2.1)\.
- \[5\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]G\. Imeraj and H\. Iyatomi\(2025\)CLIP\-guided fusion of image and leaf dictionary features for plant disease classification\.In2025 IEEE International Conference on Systems, Man, and Cybernetics \(SMC\),pp\. 4603–4608\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]M\. Kaur, R\. Singh, S\. Alirezaee, and I\. Hussain\(2025\)Visual\-language transformer\-based tomato leaf disease detection for portable greenhouse monitoring device\.Plant Methods21\(1\),pp\. 139\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p1.1)\.
- \[8\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell\(2017\)Simple and scalable predictive uncertainty estimation using deep ensembles\.30\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px4.p1.1)\.
- \[9\]J\. Li, D\. Li, S\. Savarese, and S\. Hoi\(2023\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational conference on machine learning,pp\. 19730–19742\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p1.1)\.
- \[10\]P\. Liu, W\. Yuan, J\. Fu, Z\. Jiang, H\. Hayashi, and G\. Neubig\(2023\)Pre\-train, prompt, and predict: a systematic survey of prompting methods in natural language processing\.55\(9\),pp\. 1–35\.Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p1.1)\.
- \[11\]S\. Menon and C\. Vondrick\(2022\)Visual classification via description from large language models\.arXiv preprint arXiv:2210\.07183\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]M\. Minderer, J\. Djolonga, R\. Romijnders, F\. Hubis, X\. Zhai, N\. Houlsby, D\. Tran, and M\. Lucic\(2021\)Revisiting the calibration of modern neural networks\.Advances in neural information processing systems34,pp\. 15682–15694\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]A\. Morales\-Garzón, S\. Lopez\-Joya, M\. López\-Pérez, and M\. J\. Martin\-Bautista\(2026\)Evaluating zero\-shot vision–language models for agri\-food image classification\.InInternational Conference on Information Processing and Management of Uncertainty in Knowledge\-Based Systems,Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p2.1)\.
- \[14\]U\. Nawaz, A\. Muhammad, H\. Gani, M\. Naseer, F\. S\. Khan, S\. Khan, and R\. Anwer\(2025\)AgriCLIP: adapting clip for agriculture and livestock via domain\-specialized cross\-model alignment\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 9630–9639\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. Dillon, B\. Lakshminarayanan, and J\. Snoek\(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px4.p1.1)\.
- \[16\]S\. Pitis, M\. R\. Zhang, A\. Wang, and J\. Ba\(2023\)Boosted prompt ensembles for large language models\.Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p1.1)\.
- \[17\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p1.1),[§2](https://arxiv.org/html/2608.18116#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.18116#S3.SS1.p1.10),[§4\.2](https://arxiv.org/html/2608.18116#S4.SS2.p1.1)\.
- \[18\]Y\. Wen, Q\. Li, Z\. Ye, J\. Zhang, J\. Wu, Z\. Mai, S\. Lou, Y\. Chen, H\. Huang, X\. Fan,et al\.\(2025\)AgriCoT: a chain\-of\-thought benchmark for evaluating reasoning in vision\-language models for agriculture\.arXiv preprint arXiv:2511\.23253\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p2.1)\.
- \[19\]G\. Yu, L\. H\. Anh, D\. T\. Vu, J\. Lee, Z\. U\. Rahman, H\. Lee, J\. Jo, and J\. Kim\(2025\)VL\-paw: a vision–language dataset for pear, apple and weed\.Electronics14\(10\),pp\. 2087\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p2.1)\.
- \[20\]B\. Yuan, S\. Song, J\. Fernandez, Y\. Luo, M\. Baktashmotlagh, and Z\. Wang\(2025\)WisWheat: a three\-tiered vision\-language dataset for wheat management\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 13060–13067\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p2.1)\.
- \[21\]X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer\(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 11975–11986\.Cited by:[§1](https://arxiv.org/html/2608.18116#S1.p1.1),[§3\.1](https://arxiv.org/html/2608.18116#S3.SS1.p1.10),[§4\.2](https://arxiv.org/html/2608.18116#S4.SS2.p1.1)\.
- \[22\]Y\. Zhang, Y\. Shao, C\. Tang, Z\. Liu, Z\. Li, R\. Zhai, H\. Peng, and P\. Song\(2025\)E\-clip: an enhanced clip\-based visual language model for fruit detection and recognition\.Agriculture15\(11\)\.External Links:ISSN 2077\-0472Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. Liu\(2022\)Conditional prompt learning for vision\-language models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 16816–16825\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. Liu\(2022\)Learning to prompt for vision\-language models\.International Journal of Computer Vision \(IJCV\)\.Cited by:[§2](https://arxiv.org/html/2608.18116#S2.SS0.SSS0.Px1.p1.1)\.