Low-cost concept-based localized explanations: How far can we get with training-free approaches?

# Low-cost concept-based localized explanations: How far can we get with training-free approaches?
Source: [https://arxiv.org/html/2606.29069](https://arxiv.org/html/2606.29069)
Darian Fernández\-Gutiérrez1,2\*, Rafael Bello1, Marilyn Bello2and Natalia Díaz\-Rodríguez21Dept\. of Computer Science, Central University ”Marta Abreu” of Las Villas \(UCLV\), 50100 Santa Clara, Cuba2Dept\. of Computer Science and Artificial Intelligence, University of Granada \(UGR\), 18071 Granada, Spain\.\*Corresponding author:dfgutierrez@correo\.ugr\.es

###### Abstract

Concept\-based Explainable AI \(C\-XAI\) seeks human\-understandable explanations grounded in semantic concepts, yet validation is limited by the scarcity of fine\-grained concept annotations\. We evaluate whether mid\-scale Multimodal Large Language Models \(MLLMs\) can perform localized concept naming under strict zero\-shot conditions by assigning labels to bounding\-box regions at both object and part levels\. We propose a reproducible zero\-shot evaluation protocol forConceptNaming \(CoNa\) with \(i\) closed\-set, category\-constrained prompting for moderate vocabularies and \(ii\) Open\-CoNa, an embedding\-similarity\-based strategy for large label spaces\. Experiments with four MLLMs \(7B–32B\) show consistent performance trends across datasets, reaching 62%–88% object\-level exact\-match accuracy, highlighting the potential of training\-free concept annotation from localized regions\. We discuss limitations and failure modes and release a reproducible framework to support future low\-cost C\-XAI research\.

## IIntroduction

The zero\-shot capabilities of recent MLLMs have motivated their use in multimodal explainability, where language provides rapid annotation capabilities for images using the closest representation in text \(e\.g\., CLIP\[[19](https://arxiv.org/html/2606.29069#bib.bib9)\]\)\. However, multimodal models typically rely on paired image–text training, while current eXplainable Artificial Intelligence \(XAI\)\[[1](https://arxiv.org/html/2606.29069#bib.bib12),[4](https://arxiv.org/html/2606.29069#bib.bib13)\]methods remain limited in delivering low\-level, fine\-grained explanations for such systems\. Fine\-grained annotations are often unavailable, so explanations usually remain coarse and provide limited support for detailed inspection\. Moreover, even if the decision process is transparent, this does not guarantee auditability, particularly for non\-technical audiences\[[2](https://arxiv.org/html/2606.29069#bib.bib7)\]\.

Concept\-based XAI \(C\-XAI\) aims to make explanations more human\-aligned by grounding them in natural\-language semantic concepts rather than raw features \(e\.g\., pixels\)\[[18](https://arxiv.org/html/2606.29069#bib.bib8)\]\. This aligns with transparency requirements for high\-risk AI systems, which must communicate their behavior to diverse stakeholders \(UNESCO principles and the EU AI Act\)\[[7](https://arxiv.org/html/2606.29069#bib.bib14)\]\.

The recent success of MLLMs as tools for zero\-shot learning \(ZSL\) makes textual encoding of visual concepts useful for automatic annotation of interpretable visual concepts\. However, a challenge in the context of explaining visual language models \(VLMs\) lies in the model understanding \(i\.e\., identifying and localizing visually\) granular concepts111This distinction is essential in concept\-based XAI approaches, where explanations must rely on semantic units that are clear, coherent, and understandable to end users\[[10](https://arxiv.org/html/2606.29069#bib.bib5)\]\., i\.e\., understanding fine\- and coarse\-grained level concepts localized in any region of an image\. This has been demonstrated by VLMs showing language\-derived biases\[[15](https://arxiv.org/html/2606.29069#bib.bib15)\]\.

Since manual object\- and part\-level annotation is expensive and difficult to scale\[[8](https://arxiv.org/html/2606.29069#bib.bib1)\], it is important to evaluate whether current small\- and mid\-scale MLLMs can reliably assign semantic labels to localized regions under strictzero\-shotsettings\. This capability would support semi\-automatic annotation workflows for fine\-grained C\-XAI\. However, the impact of factors such as vocabulary size, region type, prompt design, and hallucinations on performance remains insufficiently understood\[[12](https://arxiv.org/html/2606.29069#bib.bib20)\]\.

The goal of this work is to conduct a rigorous and systematic evaluation of small\- and mid\-scale multimodal models \(up to 32B parameters\) to determine their capability to identify localized visual concepts inzero\-shotmode\. To this end, we propose a controlled experimental protocol in which each model receives an image and a delimited region \(bounding box\) along with a closed set of candidate labels, from which it must select the most appropriate concept category for the visual content\. The analysis is performed at both object\- and the fine\-grained part\-level, enabling a comparison of the models’ sensitivity to different degrees of semantic granularity\. The main contributions of this study are as follows:

- •Azero\-shotevaluation protocol, called Concept Naming \(CoNa\) and Open\-CoNa, to measure the ability of small\- and mid\-scale MLLMs to assign semantic labels to localized visual regions\. This pipeline takes as input an image together with the corresponding bounding boxes to ensure a controlled, region\-specific evaluation\.
- •An analysis of several MLLMs in object\- and part\-level visual naming accuracy, as well as “unknown” category rates, evaluated using the PASCAL\-Part, ADE20K, and LIP datasets, all of which contain concept\-level annotated ground truth\.
- •Empirical evidence showing the potential and limitations of mid\-scale MLLMs for their future integration into semi\-automatic conceptual dataset construction workflows\.

## IIRelated Work

An increasing body of research is examining the capabilities and limitations of multimodal models for zero\-shot visual reasoning\. These models, pretrained on large\-scale image–text corpora, integrate visual and linguistic information within a shared embedding space, enabling them to assign semantic labels to image regions without task\-specific fine\-tuning\. While global capabilities are widely studied, their behavior under localized concept\-level naming constraints remains less explored\. This motivates a systematic evaluation of reliable object and part\-level naming\.

Multimodal Models and Zero\-Shot Visual Reasoning

Large\-scale multimodal pretraining learns shared vision–language embeddings that align images and text within a unified representation space\. CLIP showed that contrastive training on image–text pairs yields strong semantic alignment and enables zero\-shot recognition without task\-specific fine\-tuning, laying the groundwork for many subsequent multimodal foundation models\[[19](https://arxiv.org/html/2606.29069#bib.bib9)\]\.

The present study focuses on low\- and mid\-scale multimodal models \(up to 32B parameters\), which offer a favorable balance between reasoning capability and deployability in real\-world environments\. The models evaluated in this work—LLaVA 1\.6–7B, Gemma 3–12B, Mistral Small 3\.1 and Qwen2\.5\-VL–32B—were selected for their various multimodal alignment mechanisms and their relevance for region\-level reasoning\. However, prior evaluations of these models have focused largely on global tasks such as image captioning, OCR, or VQA, without examining their behavior when constrained to identify localized concepts from closed vocabularies, a requirement that is essential for C\-XAI\.

Fine\-grained labeling by concept\-level naming

Existing concept\-extraction techniques, such as TCAV\[[10](https://arxiv.org/html/2606.29069#bib.bib5)\], ACE\[[8](https://arxiv.org/html/2606.29069#bib.bib1)\], and Concept Bottleneck Models\[[11](https://arxiv.org/html/2606.29069#bib.bib4)\], rely on curated concept datasets and supervised training pipelines\. These methods assume that meaningful, human\-aligned concepts are already available but do not address the challenge of automatically obtaining concept labels at either the object or part\-level\.

Recent work on open\-vocabulary detection and part segmentation \(e\.g\., GroundingDINO\[[14](https://arxiv.org/html/2606.29069#bib.bib6)\]\) can localize or segment novel categories, but these models still depend on datasets with explicit vision–language correspondence annotations or on promptable detectors that require specifying which concept to detect\. This prevents evaluating their intrinsic ability to understand human\-aligned concepts, and their performance typically degrades at the part\-level\.

In contrast, our work evaluates whether MLLMs can serve as training\-free concept labelers, assigning semantic labels to localized regions without additional supervision\. This fills a gap by assessing their intrinsic naming capability under strict zero\-shot, low\-cost conditions with predefined concept taxonomies\.

Localized Recognition and Zero\-Shot Constraints

The introduction of the Segment Anything Model \(SAM\)\[[5](https://arxiv.org/html/2606.29069#bib.bib21)\]decoupled localization from categorization, enabling modular pipelines where one component determines where a region is and another determines what it represents\. Following this perspective, we assume regions are accurately localized \(e\.g\., via SAM3\) and focus on whether MLLMs can assign coherent concept labels under zero\-shot constraints\.

Importantly, while promptable segmenters \(e\.g\., SAM3\) require concept names as input, in order to properly audit C\-XAI, we should not assume that neither the user nor the model knows in advance which concepts are present in an image\. Despite recent progress, systematic evaluations of mid\-scale MLLMs \(≤\\leq32B\) for fine\-grained localized concept identification remain scarce, particularly regarding: \(1\) object\- and object\-part\-level labeling under closed\-set constraints, \(2\) labeling using large label spaces \(e\.g\., ADE20K whose label set does not fit context windows\), \(3\) interactions between preprocessing/resolution and conceptual accuracy, and \(4\) failure modes such as hallucination, ambiguity, and explicit “unknown” responses\.

This study addresses these gaps by proposing a reproducible protocol to assess the reliability of multimodal models as zero\-shot visual concept labelers over given concept segmentation masks, providing empirical evidence relevant to concept\-based XAI\[[18](https://arxiv.org/html/2606.29069#bib.bib8)\]and transparent AI in safety\-critical settings\.

## IIIMethodology

This section describes the evaluation protocol designed to measure the ability of low\- and mid\-scale MLLMs to assign semantic concept labels to localized visual regions under strict zero\-shot conditions\. The methodology aims to isolate the intrinsic visual–linguistic alignment of each model by standardizing all sources of variability, including category definitions, prompt structure, visual preprocessing, and inference constraints\. The overall pipeline is illustrated in Fig\.[1](https://arxiv.org/html/2606.29069#S3.F1)\.

![Refer to caption](https://arxiv.org/html/2606.29069v1/images/CoNa.png)Figure 1:Vision Concept Naming \(CoNa\) pipeline, closed vocabulary setting\. The input image region and the predefined category dictionary are combined through a concept category\-constrained \(closed approach\) prompt builder, which formulates a structured prompt restricting the MLLM to select exactly one label from the candidate list\. The MLLM then produces the most appropriate concept name among the allowed options, yielding the final assigned label\.Category Dictionary and Closed\-Set Constraints

For each dataset, a concept dictionary was automatically extracted from the official object and part annotations, ensuring an exact correspondence between ground\-truth semantics and the categories available during inference\. An additional token,unknown, was added to capture ambiguous or unrecognized predictions\.

A closed\-set regime was adopted for all datasets except ADE20K\. Under this setting, models are required to choose exactly one category from the valid vocabulary, preventing vocabulary drift, hallucinated labels, or paraphrastic responses\. This constraint enables a cleaner assessment of conceptual alignment by ensuring that any discrepancy between prediction and ground truth reflects semantic inconsistency rather than linguistic variability\.

Prompt Construction and Response Format: To maintain consistent textual conditions across models, a standardized, dataset\-agnostic prompt builder was implemented so all prompts follow the same template:

“Select one label for the object inside the red box\. Choose only from:\{…\}\. Respond with a single word\.”

Restricting responses to a single word served three purposes: \(i\) suppressing generative drift and long\-form hallucinations common in instruction\-tuned models; \(ii\) ensuring comparability across architectures by reducing stylistic variability; \(iii\) allowing an exact\-match correspondence between the model outputs and the closed\-set dictionary\. Any output containing multiple tokens or categories outside the dictionary was considered invalid and automatically assigned theunknownlabel\.

Visual Preprocessing and Region Delimitation

To ensure uniform visual input, all images were processed through a three\-step pipeline: \(1\) region delimitation using ground\-truth bounding boxes or tight crops derived from segmentation masks; \(2\) contextual cropping with fixed padding to preserve local cues and avoid overly tight crops; and \(3\) resizing all regions to 512×\\times512 pixels, a resolution selected to balance detail and model capacity\. This standardized preprocessing prevents model differences from being driven by crop size, zoom level, or resolution variations\.

Embedding\-Based Matching for Large Label Spaces

For ADE20K, the closed\-set protocol was impractical due to prompt length constraints induced by the large concept dictionary \(∼\\sim3000 labels\)\. In this case, models produced a free\-form single\-word prediction, which was then mapped to the closest valid category using the embedding\-based procedure in Fig\.[2](https://arxiv.org/html/2606.29069#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2606.29069v1/images/Open_CoNa.png)Figure 2:Open\-vocabulary Concept Naming \(CoNa\) pipeline: the MLLM receives no predefined list of possible naming labels\. Instead, it generates a free\-form single\-word description for each input image region\. This generated term, together with all categories in the dataset\-defined concept dictionary, is encoded into vectors using a text \(here nomic\-embed\-text\[[17](https://arxiv.org/html/2606.29069#bib.bib16)\]\) encoder\. Cosine similarity \(≥\\geqτ\\tau\) is then computed between the embedding of the generated term and all dictionary embeddings\. The dictionary’s concept with the highest similarity is selected as the final assigned label, enabling concept labeling without providing all possible labels beforehand\.Both the dictionary labels and the model outputs were encoded using nomic\-embed\-text\[[17](https://arxiv.org/html/2606.29069#bib.bib16)\]text embeddings, because it produces stable embeddings specifically optimized for semantic retrieval and comparison tasks\. Cosine similarity was computed between the predicted term and each dictionary entry, assigning the category corresponding to the highest similarity exceeding a thresholdτ\\tau, which was tuned to maximize agreement with the ground\-truth labels\.

Word embeddings were used instead of sentence embeddings for two main reasons: \(i\) the dictionary is taken from the dataset labels and it is primarily composed of decontextualized nouns, and \(ii\) for evaluation simplicity \(we use 1\-gram answers\)\.

## IVExperiments

TABLE I:Comparison of low\- and mid\-scale MLLMs evaluated for zero\-shot segmented concept naming\. We consider both: Low\-cost models \(defined as architectures below 10B parameters\) and mid\-scale models \(as those up to 32B parameters\)\.Note:VRAM refers to the approximate GPU memory required to load and run each model during inference\.LLM Model AnnotatorBase architectureMain tasksParametersMemory required \(VRAM\)Relevance for this studyLLaVA\-1\.6–7B\[[13](https://arxiv.org/html/2606.29069#bib.bib18)\]ViT\-L/14 \+ LLaMA\-2–7BVQA, captioning, basic grounding7B \(low\-scale\)∼\\sim4\.7 GBOpen\-source baseline; efficient and widely used for zero\-shot spatial grounding\.Gemma 3–12B\[[20](https://arxiv.org/html/2606.29069#bib.bib17)\]Vision encoder \+ Gemma TransformerCaptioning, scene reasoning, OCR12B \(mid\-scale\)∼\\sim8\.0 GBLightweight and fast; used to analyze grounding degradation and robustness at moderate scales\.Mistral\-Small 3\.1\[[16](https://arxiv.org/html/2606.29069#bib.bib11)\]Multimodal encoder \+ Mistral TransformerCaptioning, OCR, VQA, dense description24B \(mid\-scale\)∼\\sim15 GBStronger reasoning with moderate cost; good performance–efficiency trade\-off in region\-level interpretation\.Qwen2\.5\-VL–32B\[[3](https://arxiv.org/html/2606.29069#bib.bib19)\]ViT\-L \+ Qwen2\.5 LLMCaptioning, grounding, visual reasoning32B \(mid\-scale\)∼\\sim21 GBHighest\-performing open model among those evaluated; top reference for zero\-shot semantic labeling of regions\.

TABLE II:Concept\-based datasets used to assess the MLLMs’ abilities to produce concept\-based explanations by asking them to name bounding boxes in images\.Concept\-based DatasetNumber of Images\# Object classes\# Part classesTotal Obj\.Total PartsAdditional CharacteristicsADE20K\[[21](https://arxiv.org/html/2606.29069#bib.bib3)\]∼\\sim10,000 train∼\\sim1545∼\\sim330241385127408∼\\sim3,688 object and stuff categories; provides semantic, instance, and panoptic segmentation for large\-scale scene understanding\.PASCAL\-Part\[[6](https://arxiv.org/html/2606.29069#bib.bib2)\]∼\\sim10,000 train/val20892497115746489 fine\-grained part categories; contains class\- and part\-level segmentation masks, enabling compositional part–whole analysis and localized part naming\.LIP \(Look\-into\-Person\)\[[9](https://arxiv.org/html/2606.29069#bib.bib10)\]∼\\sim10,000 val119997272335Pixel\-level parsing annotations for human body regions \(e\.g\., arms, legs, torso\), optimized for structured human part segmentation\. Object\-level evaluation corresponds to the person instance, while part labels define the fine\-grained taxonomy\.

This section presents the experimental evaluation conducted to analyze the ability of multimodal language–vision models to assign conceptual labels to localized visual regions under strictly zero\-shot conditions\. We describe the evaluated models, the datasets employed, the initial preprocessing experiment, and the two protocols applied: closed\-set CoNa and Open\-CoNa\.

Experimental Setup

The evaluation was designed to quantify the capacity of multimodal models to assign semantically coherent labels under a strictly zero\-shot regime\. Two levels of visual granularity were considered \(objects and parts\), ensuring that all models were tested under identical preprocessing, prompting, and inference conditions\.

Four representative multimodal models were compared \(Table[I](https://arxiv.org/html/2606.29069#S4.T1)\): LLaVA 1\.6 \(7B\), Gemma 3 \(12B\), Mistral Small 3\.1 \(24B\), and Qwen2\.5\-VL \(32B\)\. All were executed in the same containerized environment \(Ollama\) to guarantee deterministic inference\. For each query, the model context was reset, sampling was disabled, and fixed cropping and resolution rules were applied\.

Three widely used datasets were included: ADE20K\[[21](https://arxiv.org/html/2606.29069#bib.bib3)\], PASCAL\-Part\[[6](https://arxiv.org/html/2606.29069#bib.bib2)\], and LIP\[[9](https://arxiv.org/html/2606.29069#bib.bib10)\]\(Table[II](https://arxiv.org/html/2606.29069#S4.T2)\)\. These datasets were selected because they provide a hierarchical concept structure, enabling evaluation with localized regions not only for objects but also for their constituent parts via bounding\-box\-level supervision \(or regions derived from part masks\)\. All annotations were used without modification to preserve closed\-set consistency\. When a dataset did not include bounding boxes \(e\.g\., ADE20K\), we computed them from the dataset ground\-truth segmentation masks: for each annotated region, we took the smallest axis\-aligned rectangle that fully encloses its mask\.

Initial Preprocessing Experiment on PASCAL\-Part

Before running the main experiments, an initial study was conducted to determine the optimal preprocessing strategy for part\-level recognition\. PASCAL\-Part was selected due to its fine\-grained part annotations, which allow precise evaluation of the models’ sensitivity to recognize low\-level and low\-resolution details\.

A total of 500 images from the PASCAL\-Part dataset were used to compare three configurations: \(i\) tight crops at original resolution; \(ii\) crops resized to 256×\\times256 pixels; and \(iii\) crops resized to 512×\\times512 pixels\.

This analysis isolated the effects of resolution and spatial detail on conceptual labeling, using the same prompt and full PASCAL\-Part vocabulary\. Results showed a consistent improvement with 512×\\times512, particularly for small or highly localized parts\. This configuration was therefore adopted for all subsequent experiments in both open and closed CoNa\.

CoNa Protocol: Closed\-Set Concept Labeling

After establishing the optimal preprocessing, the CoNa protocol was applied\. Under this setting, the model must select exactly one label from the dataset’s concept dictionary, avoiding linguistic drift and free\-form generation\. This closed\-set regime was applied to all datasets except ADE20K and enabled a direct evaluation of intrinsic visual–semantic alignment\.

Open\-CoNa Protocol for ADE20K

ADE20K contains approximately 3,000 categories, making CoNa impractical due to the extreme length of the prompts\. For this dataset, the Open\-CoNa protocol we propose is an embedding\-similarity\-based concept matching procedure consisting of: \(1\) the MLLM generates a single\-word prediction; \(2\) this word and all dataset categories were projected into a unimodal embedding space using thenomic\-embed\-text\[[17](https://arxiv.org/html/2606.29069#bib.bib16)\]text encoder; \(3\) cosine similarity is computed between the generated term and all dictionary entries; and \(4\) the final category corresponded to the label with the highest similarity above a predefined threshold222We performed a random search over the interval\[0\.50,0\.75\]\[0\.50,0\.75\]with a step size of0\.050\.05, obtainingτ=0\.60\\tau=0\.60as the best\-performing threshold\.\. This approach avoids the closed\-set performance degradation caused by a list of potential concept labels that exceeds the model’s context window\.

Evaluation Metrics

Model performance under open and closed CoNa was quantified using two metrics: \(i\) Exact Match \(EM\), defined as the proportion of predictions that exactly match the ground\-truth label, and \(ii\) Unknown rate \(UNK\), defined as the proportion of outputs mapped to theunknownlabel, reflecting uncertainty or semantic mismatch\. Metrics were computed independently for object classes and object parts\.

## VResults

This section reports the empirical findings obtained from the full experimental pipeline\. We first describe the outcome of the preliminary evaluation used to determine the optimal preprocessing configuration for part\-level analysis, followed by the global results across all models and datasets\.

Preliminary Evaluation on Image Cropping and Resolution Scaling

A preliminary study was conducted on 500 PASCAL\-Part images, comprising 1,231 evaluated objects and 7,847 evaluated parts\. The objective was to assess how different preprocessing strategies affect part\-level conceptual labeling\. Three configurations were compared using Qwen2\.5\-VL \(32B\): \(i\) tight part crops, \(ii\) crops resized to256×256256\\times 256, and \(iii\) crops resized to512×512512\\times 512\.

Part\-level naming performance varies substantially when increasing the crop resolution333Object\-level accuracy \(Obj\. Acc\.\) and object\-level unknown rates \(UNK Obj\.\) remain constant across all configurations because object regions were not modified; only part crops were altered\. Accuracy refers exclusively to part\-level predictions; object categories were only used to localize regions and were not included in the evaluation\.\. This occurs because low\-resolution resizing blurs fine\-grained cues \(e\.g\., contours, textures\) in small parts, increasing ambiguity\. Higher resolutions \(512×512512\\times 512\) preserve spatial details, reducing theUNKrate and improving part\-level EM \(Table[III](https://arxiv.org/html/2606.29069#S5.T3)\)\. Consequently, we adopted this configuration for all subsequent experiments

TABLE III:Ablation study comparing image preprocessing strategies for CoNa using Qwen2\.5\-VL \(32B\) on PASCAL\-Part\.UNKnown answers report both percentage and absolute count\.Full Evaluation Across All Models and Datasets

Using the512×512512\\times 512preprocessing configuration—previously identified as the best\-performing setting—the evaluation was conducted with CoNa on PASCAL\-Part and LIP, where the concept dictionaries define compact and well\-structured object and part label spaces\. In contrast, Open\-CoNa was employed for ADE20K due to its substantially larger and more heterogeneous label set\. Table[IV](https://arxiv.org/html/2606.29069#S5.T4)summarizes the global Exact Match \(EM\) accuracy and Unknown \(UNK\) rates for both object\- and part\-level predictions under these configurations\.

TABLE IV:CoNa and Open\-CoNa \(ADE20K\) performance across all evaluated MLLMs \(with512×512512\\times 512preprocessing configuration\)\. Exact Match \(EM\) reports object\- and part\-level predictions \(%\); higher is better \(↑\\uparrow\)\. Unknown \(UNK\) and Hallucinations reported as percentage and absolute count; lower is better \(↓\\downarrow\)\. Hallucination is defined only under closed\-set constraints \(PASCAL\-Part and LIP\) as out\-of\-vocabulary \(predictions outside the label set\) or format\-violating outputs\. In ADE20K \(Open\-CoNa\), due to embedding\-similarity\-based labeling, hallucinations cannot occur \(–\)\.
## VIDiscussion

We acknowledge several limitations\. Open\-CoNa maps free\-form outputs via embedding similarity, whereas closed\-set prompting offers tighter control over the label space; however, this assumption may not hold in open\-world settings with evolving taxonomies\. Image resizing can distort very small regions, affecting fine\-grained recognition\. In addition, using a unimodal text encoder for similarity may not fully capture multimodal semantic geometry and can introduce alignment bias\. The single\-word constraint reduces linguistic drift but may exclude valid multi\-word concepts\. Finally, mid\-scale MLLMs involve practical trade\-offs: inference cost increases with model size, and context\-window limits constrain closed\-set prompting for large vocabularies such as ADE20K\. Nevertheless, while fine\-grained part naming remains challenging, our study shows that training\-free approaches can reach strong object\-level accuracy and non\-trivial part\-level performance, positioning mid\-scale MLLMs as a cost\-effective starting point for concept\-based localized explanations\.

## VIIConclusion

This work provides a reproducible zero\-shot framework for localized concept labeling using CoNa and Open\-CoNa across vocabularies of different sizes\. Across three datasets, performance generally improves with model scale, reaching up to nearly 90% Exact Match at the object level and around 50% at the part level\. Our preprocessing ablation further indicates that512×512512\\times 512inputs improve part\-level naming by preserving spatial detail for small regions\. Overall, these results support mid\-scale MLLMs as practical, training\-free tools for concept annotation in C\-XAI, while motivating future work on hierarchical reasoning and evaluation under dynamic or partially known vocabularies\. Code is available444https://github\.com/darianfgUgr/CoNa\.

## Acknowledgment

D\. Fernández\-Gutiérrez gratefully acknowledges the financial support provided by the Junta de Andalucía and the Asociación Universitaria Iberoamericana de Postgrado \(AUIP\)\. Díaz acknowledges TSI\-100927\-2023\-1 Project \(Transformation and Resilience Plan from the EU NextGen through the Ministry for Digital Transformation and the Civil Service\), Google Research Scholar and Grants PID2023\-149128NB\-I00 and PID2023\-150070NB\-I00 funded by MICIU/AEI /10\.13039/501100011033 and by ERDF, EU, and A\. Porrello\. This publication is part of the R&D&I project PID2024\-156434NB\-I00 \(CONFIA2\), funded by MICIU/AEI/10\.13039/501100011033 and FEDER/EU\.

## References

- \[1\]S\. Ali, T\. Abuhmed, S\. El\-Sappagh, K\. Muhammad, J\. M\. Alonso\-Moral, R\. Confalonieri, R\. Guidotti, J\. Del Ser, N\. Díaz\-Rodríguez, and F\. Herrera\(2023\-11\-01\)Explainable artificial intelligence \(XAI\): what we know and what is left to attain trustworthy artificial intelligence\.99,pp\. 101805\.External Links:ISSN 1566\-2535,[Link](https://www.sciencedirect.com/science/article/pii/S1566253523001148),[Document](https://dx.doi.org/10.1016/j.inffus.2023.101805)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p1.1)\.
- \[2\]A\. B\. Arrieta, N\. Díaz\-Rodríguez, J\. D\. Ser, A\. Bennetot, S\. Tabik, A\. Barbado, S\. García, S\. Gil\-López, D\. Molina, R\. Benjamins, R\. Chatila, and F\. Herrera\(2019\-12\-26\)Explainable artificial intelligence \(XAI\): concepts, taxonomies, opportunities and challenges toward responsible AI\.arXiv\.External Links:[Link](http://arxiv.org/abs/1910.10045),[Document](https://dx.doi.org/10.48550/arXiv.1910.10045),1910\.10045 \[cs\]Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p1.1)\.
- \[3\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\-02\-19\)Qwen2\.5\-VL technical report\.arXiv\.External Links:[Link](http://arxiv.org/abs/2502.13923),[Document](https://dx.doi.org/10.48550/arXiv.2502.13923),2502\.13923 \[cs\]Cited by:[TABLE I](https://arxiv.org/html/2606.29069#S4.T1.4.4.4.4.4.4.4.4.2.1.1)\.
- \[4\]\(2025\-12\-22\)A three\-level framework for LLM\-enhanced explainable AI: from technical explanations to natural language\.External Links:ISSN 1572\-9419,[Link](https://doi.org/10.1007/s10796-025-10668-1),[Document](https://dx.doi.org/10.1007/s10796-025-10668-1)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p1.1)\.
- \[5\]N\. Carion, L\. Gustafson, Y\. Hu, S\. Debnath, R\. Hu, D\. Suris, C\. Ryali, K\. V\. Alwala, H\. Khedr, A\. Huang, J\. Lei, T\. Ma, B\. Guo, A\. Kalla, M\. Marks, J\. Greer, M\. Wang, P\. Sun, R\. Rädle, T\. Afouras, E\. Mavroudi, K\. Xu, T\. Wu, Y\. Zhou, L\. Momeni, R\. Hazra, S\. Ding, S\. Vaze, F\. Porcher, F\. Li, S\. Li, A\. Kamath, H\. K\. Cheng, P\. Dollár, N\. Ravi, K\. Saenko, P\. Zhang, and C\. Feichtenhofer\(2025\)SAM 3: segment anything with concepts\.External Links:2511\.16719,[Link](https://arxiv.org/abs/2511.16719)Cited by:[§II](https://arxiv.org/html/2606.29069#S2.p10.1)\.
- \[6\]X\. Chen, R\. Mottaghi, X\. Liu, S\. Fidler, R\. Urtasun, and A\. Yuille\(2014\-06\)Detect what you can: detecting and representing objects using holistic models and body parts\.In2014 IEEE Conference on Computer Vision and Pattern Recognition,pp\. 1979–1986\.External Links:ISBN 978\-1\-4799\-5118\-5,[Link](https://ieeexplore.ieee.org/document/6909651),[Document](https://dx.doi.org/10.1109/CVPR.2014.254)Cited by:[TABLE II](https://arxiv.org/html/2606.29069#S4.T2.5.5.5.5.5.5.5.5.2.1.1),[§IV](https://arxiv.org/html/2606.29069#S4.p5.1)\.
- \[7\]N\. Díaz\-Rodríguez, J\. Del Ser, M\. Coeckelbergh, M\. López de Prado, E\. Herrera\-Viedma, and F\. Herrera\(2023\-11\-01\)Connecting the dots in trustworthy artificial intelligence: from AI principles, ethics, and key requirements to responsible AI systems and regulation\.99,pp\. 101896\.External Links:ISSN 1566\-2535,[Link](https://www.sciencedirect.com/science/article/pii/S1566253523002129),[Document](https://dx.doi.org/10.1016/j.inffus.2023.101896)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p2.1)\.
- \[8\]A\. Ghorbani, J\. Wexler, J\. Y\. Zou, and B\. Kim\(2019\)Towards automatic concept\-based explanations\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/77d2afcb31f6493e350fca61764efb9a-Abstract.html)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p4.1),[§II](https://arxiv.org/html/2606.29069#S2.p6.1)\.
- \[9\]K\. Gong, X\. Liang, D\. Zhang, X\. Shen, and L\. Lin\(2017\-07\)Look into person: self\-supervised structure\-sensitive learning and a new benchmark for human parsing\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 6757–6765\.External Links:ISBN 978\-1\-5386\-0457\-1,[Link](http://ieeexplore.ieee.org/document/8100198/),[Document](https://dx.doi.org/10.1109/CVPR.2017.715)Cited by:[TABLE II](https://arxiv.org/html/2606.29069#S4.T2.6.6.6.6.6.6.6.6.2.1.1),[§IV](https://arxiv.org/html/2606.29069#S4.p5.1)\.
- \[10\]B\. Kim, M\. Wattenberg, J\. Gilmer, C\. Cai, J\. Wexler, F\. Viegas, and R\. Sayres\(2018\-07\-03\)Interpretability beyond feature attribution: quantitative testing with concept activation vectors \(TCAV\)\.InProceedings of the 35th International Conference on Machine Learning,pp\. 2668–2677\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v80/kim18d.html)Cited by:[§II](https://arxiv.org/html/2606.29069#S2.p6.1),[footnote 1](https://arxiv.org/html/2606.29069#footnote1)\.
- \[11\]P\. W\. Koh, T\. Nguyen, Y\. S\. Tang, S\. Mussmann, E\. Pierson, B\. Kim, and P\. Liang\(2020\-11\-21\)Concept bottleneck models\.InProceedings of the 37th International Conference on Machine Learning,pp\. 5338–5348\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v119/koh20a.html)Cited by:[§II](https://arxiv.org/html/2606.29069#S2.p6.1)\.
- \[12\]Y\. Li, Y\. Du, K\. Zhou, J\. Wang, X\. Zhao, and J\. Wen\(2023\-12\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 292–305\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.20/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.20)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p4.1)\.
- \[13\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InNeurIPS,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf)Cited by:[TABLE I](https://arxiv.org/html/2606.29069#S4.T1.1.1.1.1.1.1.1.1.2.1.1)\.
- \[14\]S\. Liu, Z\. Zeng, T\. Ren, F\. Li, H\. Zhang, J\. Yang, Q\. Jiang, C\. Li, J\. Yang, H\. Su, J\. Zhu, and L\. Zhang\(2025\)Grounding DINO: marrying DINO with grounded pre\-training for open\-set object detection\.InComputer Vision – ECCV 2024,A\. Leonardis, E\. Ricci, S\. Roth, O\. Russakovsky, T\. Sattler, and G\. Varol \(Eds\.\),pp\. 38–55\.External Links:ISBN 978\-3\-031\-72970\-6,[Document](https://dx.doi.org/10.1007/978-3-031-72970-6%5F3)Cited by:[§II](https://arxiv.org/html/2606.29069#S2.p7.1)\.
- \[15\]D\. Méndez, G\. Bontempo, E\. Ficarra, R\. Confalonieri, and N\. Díaz\-Rodríguez\(2025\-06\)CUBIC: concept embeddings for unsupervised bias identification using VLMs\.In2025 International Joint Conference on Neural Networks \(IJCNN\),pp\. 1–8\.Note:ISSN: 2161\-4407External Links:ISSN 2161\-4407,[Link](https://ieeexplore.ieee.org/document/11228477),[Document](https://dx.doi.org/10.1109/IJCNN64981.2025.11228477)Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p3.1)\.
- \[16\]Mistral small 3\.1 \| mistral AI\(Website\)External Links:[Link](https://mistral.ai/news/mistral-small-3-1)Cited by:[TABLE I](https://arxiv.org/html/2606.29069#S4.T1.3.3.3.3.3.3.3.3.2.1.1)\.
- \[17\]Z\. Nussbaum, J\. X\. Morris, B\. Duderstadt, and A\. Mulyar\(2025\-02\-03\)Nomic embed: training a reproducible long context text embedder\.arXiv\.External Links:[Link](http://arxiv.org/abs/2402.01613),[Document](https://dx.doi.org/10.48550/arXiv.2402.01613),2402\.01613 \[cs\]Cited by:[Figure 2](https://arxiv.org/html/2606.29069#S3.F2),[§III](https://arxiv.org/html/2606.29069#S3.p12.1),[§IV](https://arxiv.org/html/2606.29069#S4.p13.1)\.
- \[18\]E\. Poeta, G\. Ciravegna, E\. Pastor, T\. Cerquitelli, and E\. Baralis\(2023\-12\-20\)Concept\-based explainable artificial intelligence: a survey\.arXiv\.External Links:[Link](http://arxiv.org/abs/2312.12936),[Document](https://dx.doi.org/10.48550/arXiv.2312.12936),2312\.12936 \[cs\]Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p2.1),[§II](https://arxiv.org/html/2606.29069#S2.p12.1)\.
- \[19\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever\(2021\-02\-26\)Learning transferable visual models from natural language supervision\.arXiv\.External Links:[Link](http://arxiv.org/abs/2103.00020),[Document](https://dx.doi.org/10.48550/arXiv.2103.00020),2103\.00020 \[cs\]Cited by:[§I](https://arxiv.org/html/2606.29069#S1.p1.1),[§II](https://arxiv.org/html/2606.29069#S2.p3.1)\.
- \[20\]G\. Teamet al\.\(2025\-03\-25\)Gemma 3 technical report\.arXiv\.External Links:[Link](http://arxiv.org/abs/2503.19786),[Document](https://dx.doi.org/10.48550/arXiv.2503.19786),2503\.19786 \[cs\]Cited by:[TABLE I](https://arxiv.org/html/2606.29069#S4.T1.2.2.2.2.2.2.2.2.2.1.1)\.
- \[21\]B\. Zhou, H\. Zhao, X\. Puig, S\. Fidler, A\. Barriuso, and A\. Torralba\(2017\-07\)Scene parsing through ADE20k dataset\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 5122–5130\.External Links:ISBN 978\-1\-5386\-0457\-1,[Link](http://ieeexplore.ieee.org/document/8100027/),[Document](https://dx.doi.org/10.1109/CVPR.2017.544)Cited by:[TABLE II](https://arxiv.org/html/2606.29069#S4.T2.4.4.4.4.4.4.4.4.5.1.1),[§IV](https://arxiv.org/html/2606.29069#S4.p5.1)\.