Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval

# Can Frontier LLMs Match Natively Multimodal Embeddings? A Comparison on Hard-Negative Text-to-Image Retrieval
Source: [https://arxiv.org/html/2608.11343](https://arxiv.org/html/2608.11343)
###### Abstract

Multimodal retrieval and classification across different types of media, spanning text, images, video and audio, has traditionally relied on dual\-encoder models that align visual and textual representations through contrastive learning\. The March 2026 release of Gemini Embedding 2, Google’s first natively multimodal embedding model to map text, images, video, audio, and documents into a single shared space, raises competition among multimodal retrieval systems\. Simultaneously, frontier Large language models \(LLMs\) have also demonstrated strong visual understanding, raising the question of whether they can serve as effective zero\-shot rankers\. Our study provides the first direct comparison of native multimodal embeddings against LLM\-based visual ranking on Flickr30k\. We observe that GPT\-4\.1 and Claude Sonnet 4\.6 perform on par with Gemini Embedding 2\. Additionally, once embeddings are precomputed, multimodal embeddings are better suited for low\-latency applications\.

## 1Introduction

Text\-image retrieval, ranking a gallery of images by relevance to a natural language query, is a long\-standing task with applications in visual search, content recommendation, and retrieval\-augmented generation\. There have been two recent developments in this space\. First, LLMs with strong vision capabilities, such as GPT\-4\.1\([16](https://arxiv.org/html/2608.11343#bib.bib24)\)and Claude\([2](https://arxiv.org/html/2608.11343#bib.bib6)\), can now receive images and text jointly, enabling a fundamentally different retrieval paradigm: presenting all candidate images alongside the query in one prompt and ranking them directly\. Second, natively multimodal embedding models have recently become publicly available: Gemini Embedding 2 \(GE2\)\([8](https://arxiv.org/html/2608.11343#bib.bib5)\)and Amazon Nova 2 \(Nova 2\)\([1](https://arxiv.org/html/2608.11343#bib.bib1)\), released in March, 2026 and October 2025, respectively, both project multiple modalities through a single shared backbone into a unified embedding space, unlike dual\-encoder models that align independently encoded representations\. The comparison of native multimodal embedding \(GE2, Nova 2\) with direct LLM visual ranking \(GPT\-4\.1, Claude Sonnet 4\.6\) is timely and currently uncharacterized in the literature, specifically on \(1\)Retrieval Accuracyand \(2\)Ranking TimeOur contributions are as follows:

- •The first comparison of natively multimodal embedding models against frontier LLMs as zero\-shot direct visual rankers on Flickr30k\([23](https://arxiv.org/html/2608.11343#bib.bib20)\), using semantically challenging hard\-negative candidate sets without task\-specific fine\-tuning or intermediate caption generation\.
- •An empirical finding that GE2, GPT\-4\.1, and Claude Sonnet 4\.6 achieve statistically indistinguishable retrieval accuracy, and embedding models rank 1000 queries in under two seconds \(with pre\-computation\) compared to over three hours for LLMs\.

## 2Related Work

Dual\-encoder cross\-modal retrieval\.CLIP\([17](https://arxiv.org/html/2608.11343#bib.bib2)\)established the paradigm of learning aligned image and text encoders via large\-scale contrastive pretraining, achieving strong zero\-shot transfer on image\-text retrieval benchmarks\. ALIGN\([9](https://arxiv.org/html/2608.11343#bib.bib3)\)demonstrated that data scale can compensate for annotation quality\. More recent work, including BLIP\-2\([12](https://arxiv.org/html/2608.11343#bib.bib8)\)and EVA\-CLIP\([6](https://arxiv.org/html/2608.11343#bib.bib14)\), has extended these ideas with larger architectures and richer pretraining objectives, achieving state\-of\-the\-art performance on COCO\([13](https://arxiv.org/html/2608.11343#bib.bib18)\)and Flickr30k\.

LLMs for retrieval and re\-ranking\.[20](https://arxiv.org/html/2608.11343#bib.bib11)found that ChatGPT achieves competitive performance as a zero\-shot document ranker on text\-only benchmarks\.[14](https://arxiv.org/html/2608.11343#bib.bib12)showed that prompting LLMs to produce listwise rankings outperforms pointwise scoring in text retrieval\. In the multimodal setting, BLIP\-2\([12](https://arxiv.org/html/2608.11343#bib.bib8)\)and InstructBLIP\([3](https://arxiv.org/html/2608.11343#bib.bib9)\)use LLMs to generate textual image descriptions, which are then matched against queries by a text retrieval system\. Our work departs from this indirect approach by prompting LLMs to rank all candidate images together\. MM\-Embed\([10](https://arxiv.org/html/2608.11343#bib.bib23)\)fine\-tunes a multimodal LLM\-based retriever but focuses on learning retrieval representations rather than evaluating frontier models as zero\-shot rankers\. RagVL\([24](https://arxiv.org/html/2608.11343#bib.bib10)\)demonstrates that multimodal LLMs are strong rerankers in retrieval\-augmented generation settings, though in a document QA context rather than image retrieval\.

Natively multimodal embeddings\.ImageBind\([7](https://arxiv.org/html/2608.11343#bib.bib13)\)demonstrated joint embedding of six modalities using image\-paired data as a binding signal, though its underlying encoders remain modality\-specific\. VLM2Vec\([11](https://arxiv.org/html/2608.11343#bib.bib22)\)takes a step further by fine\-tuning a vision\-language model as a unified embedding backbone, leveraging deep integration of vision and language features within a single transformer architecture rather than late fusion of independently encoded representations\.

Hard negative mining\.VSE\+\+\([5](https://arxiv.org/html/2608.11343#bib.bib4)\)introduced hard negative mining as a key training\-time improvement, showing that selecting the most violating negatives substantially improves rank\-based metrics\.[19](https://arxiv.org/html/2608.11343#bib.bib15)also demonstrate that semantically hard negatives substantially change task difficulty\.

## 3Methodology

### 3\.1Dataset, Sampling and Evaluated Systems

Flickr30k\([23](https://arxiv.org/html/2608.11343#bib.bib20)\)provides 31,000 images each annotated with five independent captions\. Although full\-gallery retrieval on Flickr30k is largely saturated at the top of the leaderboard, our hard\-negative candidate sets create a non\-trivial task\. We sample a pool of 5,000 images from Flickr30k, each paired with five independent human\-written captions\. To obtain a single representative query caption per image while controlling for annotator\-style variance, we apply a*stratified*caption sampling: the 5,000 images are partitioned into five equally\-sized groups of 1,000, each assigned exclusively to one caption index \(0 to 4\)\. The ground truth for every query is the image from which its caption was originally drawn\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/pipeline_diagram.png)Figure 1:Overview of the three\-phase evaluation pipeline: hard negative dataset construction, ranking via embedding models and LLMs, and evaluation\.
### 3\.2Hard Negative Candidate Set Construction

For each of the 5,000 pool images, we constructed a candidate set of 25 images: the ground\-truth image and the 24 hardest negatives\. This candidate set size relatively balances task difficulty with the practical constraint of fitting all images within a single LLM prompt without exceeding context limits\. Hard negatives are selected by embedding all representative captions withsentence\-transformers/all\-mpnet\-base\-v2\([18](https://arxiv.org/html/2608.11343#bib.bib7)\), a general\-purpose text encoder that is intentionally selected to be decoupled from all evaluated retrieval systems\. We then compute a 5,000×\\times5,000 pairwise cosine similarity matrix, and retain the 24 most similar captions to the query \(excluding the query itself\)\. The images corresponding to these captions form the hard negative set\. From the full pool, we randomly sample 1,000 captions as queries\. Figure[1](https://arxiv.org/html/2608.11343#S3.F1)shows the pipeline diagram\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/Hard_Negative_6.png)

![Refer to caption](https://arxiv.org/html/2608.11343v1/Hard_Negative_5.png)

Figure 2:Examples of hard negative datasets showing candidate images for a given query\.We construct hard negatives in*caption*embedding space rather than image embedding space for two reasons\. First, grounding distractor difficulty in caption\-space similarity increases ambiguity, making images hard to distinguish*given the caption*, which is precisely the retrieval difficulty we wish to evaluate\. Second, since human perception of image similarity is naturally expressed through language, using caption similarity to construct candidate sets is a well\-motivated design choice \(details mentioned in[Limitations](https://arxiv.org/html/2608.11343#Sx1)\)\. We note that constructing hard negatives in caption space may favor embedding\-based retrieval methods\. All evaluated systems \(Gemini Embedding 2, Amazon Nova 2 Multimodal Embedding, GPT\-4\.1, and Claude Sonnet 4\.6\) are exposed to identical queries and candidate sets\. Details about system configurations are provided in Appendix[A](https://arxiv.org/html/2608.11343#A1)

### 3\.3Evaluation Metrics and Statistical Tests

We report three metrics over 1,000 queries:

- •Recall@1 \(R@1\): Fraction of queries where the ground\-truth image is ranked first\.
- •Recall@3 \(R@3\): Fraction of queries where the ground\-truth image appears in the top three\.
- •Mean Reciprocal Rank \(MRR\): Mean of the reciprocal rank of the ground\-truth image across all queries\.

Additionally, we report 95% bootstrap confidence intervals for all three metrics \(Appendix[B](https://arxiv.org/html/2608.11343#A2)\) using 1,000 resamples\([4](https://arxiv.org/html/2608.11343#bib.bib17)\)\. For pairwise significance testing of R@1 \(a binary outcome\), we apply McNemar’s test\([15](https://arxiv.org/html/2608.11343#bib.bib16)\)\. For MRR \(a continuous outcome\), we apply the Wilcoxon signed\-rank test\([22](https://arxiv.org/html/2608.11343#bib.bib19)\)\. We use a significance threshold ofα=0\.05\\alpha=0\.05with Bonferroni correction for three pairwise comparisons\.

## 4Results and Discussion

![Refer to caption](https://arxiv.org/html/2608.11343v1/ACL_Multimodal_Primary_Result.png)Figure 3:Evaluating the four systems on Recall@1, Recall@3 and MRR\.Figure[3](https://arxiv.org/html/2608.11343#S4.F3)visualizes the metric comparison\. GE2 achieves the highest score across all three metrics, followed closely by Claude Sonnet 4\.6 and GPT\-4\.1, with Amazon Nova 2 trailing by 13 percentage points\. However, pairwise significance testing \(Table[2](https://arxiv.org/html/2608.11343#S4.T2)\) reveals a clear separation: Gemini Embedding 2, GPT\-4\.1, and Claude Sonnet 4\.6 are statistically indistinguishable from one another on both R@1 \(McNemar, all p \> 0\.25\) and MRR \(Wilcoxon, all p \> 0\.09\), while the performance of Amazon Nova 2 is significantly different from every other system \(p < 0\.0001 on both tests\)\. The absence of a statistically significant gap between the embedding paradigm \(Gemini Embedding 2\) and the LLM reasoning paradigm \(GPT\-4\.1, Claude Sonnet 4\.6\) is the primary finding of this study \(given the dataset and candidate set\)\. Dedicated multimodal embedding and joint visual reasoning achieve competitive retrieval accuracy on hard\-negative candidates drawn from a large image pool\.

Table 1:Pairwise statistical significance tests\. McNemar’s test is applied to paired binary R@1 outcomes; Wilcoxon signed\-rank test is applied to paired MRR scores\. The upper block shows non\-significant pairs; the lower block shows pairs involving Amazon Nova 2, all significant atp<0\.0001p<0\.0001\.Precomputation TimeSystemQuery emb\.Image emb\.Total TimeRanking\-only TimeGemini Embedding 2∼2,000\{\\sim\}2\{,\}000s∼12,500\{\\sim\}12\{,\}500s∼14,500\{\\sim\}14\{,\}500s0\.66 s\(with precomp\.\)Amazon Nova 2∼1,800\{\\sim\}1\{,\}800s∼10,000\{\\sim\}10\{,\}000s∼11,800\{\\sim\}11\{,\}800s1\.16 s\(with precomp\.\)GPT\-4\.1N/A \(end\-to\-end per\-query API calls\)6,100 s \(no precomp\.\)Claude Sonnet 4\.6N/A \(end\-to\-end per\-query API calls\)9,415 s \(no precomp\.\)
Table 2:Execution time estimates for 1,000 queries with 25 candidates\.Precomputation Time: GE2:∼2\.0\{\\sim\}2\.0s/query,∼2\.5\{\\sim\}2\.5s/image; Amazon Nova 2:∼1\.8\{\\sim\}1\.8s/query,∼2\.0\{\\sim\}2\.0s/image; the image embedding time is aggregated over 5,000 unique candidate images\.With precomputation: embeddings are computed once and reused for ranking\.GE2’s accuracy parity with LLMs\-based rankers suggests that its embedding space is optimized for retrieval, whereas LLMs are generalist models that rank through prompting\. The accuracy parity could also be potentially attributed to embedding models’ similarity scores computed independently for each query\-candidate pair, making them robust to candidate set composition and free from position or distractor biases\([21](https://arxiv.org/html/2608.11343#bib.bib21)\)\. However, we minimized this positioning bias in LLMs by randomizing the order per query\. In contrast, LLMs have an advantage that they receive all 25 images simultaneously and can reason about relative visual content across candidates before producing a ranking\. This cross\-candidate reasoning might be an advantage for the LLMs\. As a qualitative example, in Figure[4](https://arxiv.org/html/2608.11343#S4.F4)we observe that GE2 retrieved the correct image despite it being dark, while Nova 2 and GPT\-4\.1 selected the same image as their top\-ranked image\. Additional examples are provided in Appendix[F](https://arxiv.org/html/2608.11343#A6)\.

Amazon Nova 2 trails GE2 by 13\.1 percentage points on Recall@1\. We attribute this gap to the use ofGENERIC\_INDEXas the embedding purpose, which is designed for general vector database creation rather than optimized text\-image matching\. While this choice reflects realistic vector database deployment scenarios, the Amazon Bedrock API offers a dedicatedIMAGE\_RETRIEVALpurpose that may yield stronger performance on text\-to\-image tasks; we leave this comparison to future work\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/GE2_correct_only.png)Figure 4:Gemini Embedding 2 model is correct; others incorrect\.#### Precomputation \- the practical case for embedding models\.

Embedding models are highly suitable for ranking large data with lower latency owing to precomputation, compared to using LLMs as visual rankers which are limited to a smaller candidate set\. Without precomputation, embedding model total time is broadly comparable to LLM end\-to\-end time \(depending on candidate pool overlap\)\. Table[2](https://arxiv.org/html/2608.11343#S4.T2)shows that once precomputation is complete, online ranking latency \(for 1000 queries and 25 candidates\) drops to under 2 seconds for both Gemini Embedding 2 and Amazon Nova 2, compared to 6,100 seconds and 9,415 seconds for GPT\-4\.1 and Claude Sonnet 4\.6, respectively\. This speedup is four orders of magnitude faster\. For applications requiring sub\-second retrieval over a large but fixed set of text and/or images, embedding models are the better choice \(especially GE2 because of high recall scores\)\. One critical distinction is that embedding model costs are incurredonce, when creating the embeddings, and amortized across all subsequent queries\. On the other hand, LLM costs scale linearly with every query\. LLM\-based rankers may be preferable when the image gallery is frequently updated or small, rendering precomputation impractical or its amortized benefit negligible\.

## 5Conclusion

We have presented a comparison of natively multimodal embedding models \(Gemini Embedding 2 and Amazon Nova 2\), and LLMs \(GPT\-4\.1 and Claude\-Sonnet\-4\.6\) for text\-to\-image retrieval, using hard\-negative candidates from Flickr30k\. Gemini Embedding 2, GPT\-4\.1, and Claude Sonnet 4\.6 achieve statistically indistinguishable retrieval accuracy on all three metrics \- Recall@1, Recall@3, and MRR\. The practical implication favors multimodal embedding models for low\-latency applications\. Once embeddings are precomputed, Gemini Embedding 2 and Amazon Nova 2 rank 1,000 queries in under two seconds, compared to 6,100 and 9,415 seconds for GPT\-4\.1 and Claude Sonnet 4\.6, respectively\. Extending this comparison to additional datasets and to video and audio modalities are natural directions for future work\.

## Limitations

#### Caption–image correspondence\.

Flickr30k captions describe images at varying levels of specificity, and a caption from one image may plausibly describe another image in the dataset, particularly among hard negatives selected for semantic similarity\. This creates ambiguous ground truth for some queries\. We argue this limitation affects all evaluated systems equally, as none is given privileged information about which image served as the caption’s source\.

#### Hard negatives defined in caption space\.

Candidate sets are hard with respect to caption\-level semantics but the "hardness" may change in image embedding space\. A distractor that is semantically close in caption space may or may not be visually confusable in the embedding or reasoning space of a given model\. This is a deliberate design choice, as discussed in Section[3](https://arxiv.org/html/2608.11343#S3)\. Another limitation of selecting hard negatives based on caption similarity is that it may inherently favor embedding\-based methods optimized for dense vector representations\.

#### Different inference conditions\.

The two paradigms operate under fundamentally different inference conditions: LLMs receive all 25 candidate images simultaneously and may exploit cross\-candidate comparisons before producing a ranking, while embedding models score each image independently without access to other candidates\.

#### Proprietary model access\.

All four evaluated systems are accessible only through paid APIs, and model weights may be updated by providers between our experiments and any replication attempt\.

#### Candidate set size and generalizability\.

With 25\-image candidate sets, our evaluation measures ranking performance rather than full\-gallery retrieval\. Conclusions about relative system performance may not generalize to large\-scale settings\.

#### LLM Ranking Prompt and Replicability\.

Changes to LLM prompt for GPT\-4\.1, and Claude\-Sonnet\-4\.6 may affect the ranking\. For reproducibility, we have reported the prompt we created, in Appendix[C](https://arxiv.org/html/2608.11343#A3)\. Additionally, candidates could be presented to LLMs in smaller batches rather than all 25 simultaneously, which may mitigate position bias\.

## References

- Amazon \(2025\)AmazonAmazon nova multimodal embeddings: technical report and model card\.Technical reportAmazon Web Services\.External Links:[Link](https://assets.amazon.science/de/d4/149300334682a464963f01553ffb/nova-mme-technical-report-10.pdf)Cited by:[§1](https://arxiv.org/html/2608.11343#S1.p1.1)\.
- Anthropic \(2024\)AnthropicClaude: a family of large language models\.Note:Available at[https://www\.anthropic\.com](https://www.anthropic.com/)Cited by:[Appendix A](https://arxiv.org/html/2608.11343#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.11343#S1.p1.1)\.
- Daiet al\.\(2023\)W\. Dai, J\. Li, D\. Li, A\. M\. H\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. Fung, and S\. HoiInstructBLIP: towards general\-purpose vision\-language models with instruction tuning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.
- Efron and Tibshirani \(1993\)B\. Efron and R\. J\. TibshiraniAn introduction to the bootstrap\.Chapman and Hall/CRC\.Cited by:[§3\.3](https://arxiv.org/html/2608.11343#S3.SS3.p1.2)\.
- Faghriet al\.\(2018\)F\. Faghri, D\. J\. Fleet, J\. R\. Kiros, and S\. FidlerVSE\+\+: improving visual\-semantic embeddings with hard negatives\.InBritish Machine Vision Conference \(BMVC\),Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p3.1)\.
- Fanget al\.\(2023\)Y\. Fang, Q\. Sun, X\. Wang, T\. Huang, X\. Wang, and Y\. CaoEVA\-CLIP: improved training techniques for CLIP at scale\.arXiv preprint arXiv:2303\.15389\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.1)\.
- Girdharet al\.\(2023\)R\. Girdhar, A\. El\-Nouby, Z\. Liu, M\. Singh, K\. V\. Alwala, A\. Joulin, and I\. MisraImageBind: one embedding space to bind them all\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 15180–15190\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p2.1)\.
- Google DeepMind \(2026\)Google DeepMindGemini Embedding 2: our first natively multimodal embedding model\.Note:Available at[https://blog\.google/innovation\-and\-ai/models\-and\-research/gemini\-models/gemini\-embedding\-2/](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-embedding-2/)\. Accessed March 2026\.Cited by:[§1](https://arxiv.org/html/2608.11343#S1.p1.1)\.
- Jiaet al\.\(2021\)C\. Jia, Y\. Yang, Y\. Xia, Y\. Chen, Z\. Parekh, H\. Pham, Q\. V\. Le, Y\. Sung, Z\. Li, and T\. DuerigALIGN: scaling up visual and vision\-language representation learning with noisy text supervision\.InProceedings of the 38th International Conference on Machine Learning,pp\. 4904–4916\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.1)\.
- Jianget al\.\(2024\)H\. Jiang T\. Lianet al\.MM\-Embed: universal multimodal retrieval with multimodal LLMs\.External Links:2411\.02571,[Link](https://arxiv.org/abs/2411.02571)Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.
- Jianget al\.\(2025\)Z\. Jiang, R\. Meng, X\. Yang, S\. Y\. Castillo, S\. Yavuz, and W\. ChenVLM2Vec: training vision\-language models for massive multimodal embedding tasks\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2410.05160)Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p2.1)\.
- Liet al\.\(2023\)J\. Li, D\. Li, S\. Savarese, and S\. HoiBLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational Conference on Machine Learning,pp\. 19730–19742\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.1),[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft COCO: common objects in context\.InEuropean Conference on Computer Vision \(ECCV\),pp\. 740–755\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.1)\.
- Maet al\.\(2023\)X\. Ma, X\. Zhang, R\. Pradeep, and J\. LinZero\-shot listwise document reranking with a large language model\.arXiv preprint arXiv:2305\.02156\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.
- McNemar \(1947\)Q\. McNemarNote on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§3\.3](https://arxiv.org/html/2608.11343#S3.SS3.p1.2)\.
- OpenAI \(2025\)OpenAIGPT\-4\.1 system card\.Technical reportOpenAI\.External Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[§1](https://arxiv.org/html/2608.11343#S1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using Siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,pp\. 3982–3992\.Cited by:[§3\.2](https://arxiv.org/html/2608.11343#S3.SS2.p1.1)\.
- Robinsonet al\.\(2021\)J\. Robinson, C\. Chuang, S\. Sra, and S\. JegelkaContrastive learning with hard negative samples\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p3.1)\.
- Sunet al\.\(2023\)W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. RenIs ChatGPT good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 14918–14937\.Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.
- Wanget al\.\(2023\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.Cited by:[Appendix A](https://arxiv.org/html/2608.11343#A1.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2608.11343#S4.p2.1)\.
- Wilcoxon \(1945\)F\. WilcoxonIndividual comparisons by ranking methods\.Biometrics Bulletin1\(6\),pp\. 80–83\.Cited by:[§3\.3](https://arxiv.org/html/2608.11343#S3.SS3.p1.2)\.
- Younget al\.\(2014\)P\. Young, A\. Lai, M\. Hodosh, and J\. HockenmaierFrom image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions\.Transactions of the Association for Computational Linguistics2,pp\. 67–78\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00166)Cited by:[1st item](https://arxiv.org/html/2608.11343#S1.I1.i1.p1.1),[§3\.1](https://arxiv.org/html/2608.11343#S3.SS1.p1.1)\.
- Zhuet al\.\(2024\)W\. Zhuet al\.MLLM is a strong reranker: advancing multimodal retrieval\-augmented generation via knowledge\-enhanced reranking and noise\-injected training\.External Links:2407\.21439,[Link](https://arxiv.org/abs/2407.21439)Cited by:[§2](https://arxiv.org/html/2608.11343#S2.p1.2)\.

## Appendix ADetails of Evaluated Systems

#### System 1: Gemini Embedding 2 \(Gemini Embedding 2\)\.

We embed each query caption using thegemini\-embedding\-2\-previewmodel withtask\_type=RETRIEVAL\_QUERYand embed each candidate image withtask\_type=RETRIEVAL\_DOCUMENT, using 3,072\-dimensional output\. We compute cosine similarity between the caption embedding and each of the 25 candidate image embeddings, and rank the candidates from highest to lowest similarity\.

#### System 2: Amazon Nova 2 \- Multimodal Embedding 2\.

We embed each query caption using theamazon\.nova\-2\-multimodal\-embeddings\-v1:0model withembedding\_purpose=GENERIC\_INDEXand also embed each candidate image withembedding\_purpose=GENERIC\_INDEX, using 3,072\-dimensional output\. We compute cosine similarity between the caption embedding and each of the 25 candidate image embeddings and rank the candidates from highest to lowest similarity\.

#### System 3: GPT\-4\.1

We send the query caption and all 25 candidate images in a single prompt \(zero\-shot\) togpt\-4\.1\-2025\-04\-14\(latest snapshot at time of experiment\)\. Images are presented in randomized order per query to control for position bias\([21](https://arxiv.org/html/2608.11343#bib.bib21)\), numbered 1 through 25\. The prompt instructs the model to return a ranking of all 25 images by their relevance to the caption, formatted as a list of image numbers\. Randomization is applied independently per query, and the same random seed is used across both LLM conditions for reproducibility\.

#### System 4: Claude Sonnet 4\.6

We replicate the same procedure as GPT\-4\.1 for theclaude\-sonnet\-4\-6 model[2](https://arxiv.org/html/2608.11343#bib.bib6)\.

## Appendix BMetrics with Confidence Intervals

Table[3](https://arxiv.org/html/2608.11343#A2.T3)reports the complete metric results for all four systems with 95% bootstrap confidence intervals \(1,000 resamples\) and total ranking execution time for 1,000 queries\.

Table 3:Full results for all systems on 1,000 queries with 25 hard\-negative candidates per query\. Bootstrap CIs are 95% intervals computed over 1,000 resamples\. Random baseline values are theoretical \(1/25 for R@1, 3/25 for R@3, harmonic mean for MRR\)\.
## Appendix CLLM Ranking Prompt

Here’s the LLM Ranking prompt used:

"system": "You are a precise image retrieval system\. Given a text query and a set of candidate images, rank ALL images from most to least relevant\. Relevance means how well the image matches the specific subjects, actions, setting, and details described in the query\. You must return a JSON array containing every image ID exactly once\. Output ONLY the JSON array — no explanation, no commentary, nothing else\."

"intro\_template": "Text query: query\_caption\. You are given n\_candidates candidate images, each labeled with an image ID shown before it\. For each image, consider: \(1\) Does it show the same subjects or people described in the query? \(2\) Does it show the same action or activity? \(3\) Does it match the setting, location, or scene? \(4\) Does it contain the specific objects or details mentioned? Images:"

"outro": "Now return a JSON array of ALL n\_candidates image IDs ranked from most relevant \(index 0\) to least relevant \(last index\)\. Every ID must appear exactly once\. If two images seem equally relevant, use your best judgment to order them — do not omit any ID\. Provide JSON array only:",

## Appendix DMore Hard Negative Dataset Examples

Figures[5](https://arxiv.org/html/2608.11343#A4.F5)–[7](https://arxiv.org/html/2608.11343#A4.F7)illustrate representative hard negative candidate sets\. Each figure shows the query caption, the correct image \(green border\), and the top\-kkmost semantically similar images as selected byall\-mpnet\-base\-v2caption similarity\. These examples demonstrate that the hard negatives are visually and semantically confusable with the correct image, validating the difficulty of the evaluation task\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/Hard_Negative_7.png)

Figure 5:Hard negative example 1\. The query caption describes a specific scene; candidate images share overlapping visual and semantic elements that make the retrieval task non\-trivial\.![Refer to caption](https://arxiv.org/html/2608.11343v1/Hard_Negative_5.png)

Figure 6:Hard negative example 2\.![Refer to caption](https://arxiv.org/html/2608.11343v1/Hard_Negative_3.png)

Figure 7:Hard negative example 3\.
## Appendix EContingency Tables

Table[4](https://arxiv.org/html/2608.11343#A5.T4)shows the R@1 contingency table for GE2 versus GPT\-4\.1\. Of the 1,000 queries, 716 are answered correctly by both systems, while 88 are answered correctly by GE2 alone and 73 by GPT\-4\.1 alone\.

Table 4:R@1 contingency table for Gemini Embedding 2 vs\. GPT\-4\.1 \(1,000 queries\)\. Off\-diagonal cells represent qualitatively distinct failure modes analysed in Section[4](https://arxiv.org/html/2608.11343#S4)\. Contingency tables for all other system pairs are in Appendix[E](https://arxiv.org/html/2608.11343#A5)\.Tables[5](https://arxiv.org/html/2608.11343#A5.T5)–[9](https://arxiv.org/html/2608.11343#A5.T9)report R@1 contingency tables for the remaining system pairs not shown in the main text\. Off\-diagonal cells indicate queries where the two systems disagree, forming the basis for qualitative error analysis\.

Table 5:R@1 contingency table: GE2 vs\. Claude Sonnet 4\.6 \(1,000 queries\)\.Table 6:R@1 contingency table: GPT\-4\.1 vs\. Claude Sonnet 4\.6 \(1,000 queries\)\.Table 7:R@1 contingency table: GE2 vs\. Nova2 \(1,000 queries\)\.Table 8:R@1 contingency table: GPT\-4\.1 vs\. Nova2 \(1,000 queries\)\.Table 9:R@1 contingency table: Claude Sonnet 4\.6 vs\. Nova2 \(1,000 queries\)\.
## Appendix FQualitative Observation

Each figure shows the query caption, the correct image, and the top\-ranked image returned by each system\. Green text indicate a correct rank\-1 prediction; red text indicate an incorrect rank\-1 prediction\. Figure[8](https://arxiv.org/html/2608.11343#A6.F8)is another example where GE2 selected the correct image despite it being dark\. Figure[9](https://arxiv.org/html/2608.11343#A6.F9)shows how close the systems are to each other in terms of retrieval for certain queries\. Even though GE2 was technically correct, the other systems also picked "reasonably correct" images \(based on human vision\)\. This illustrates that the differences in retrieval capability of each system may be lesser than what the quantitative metrics convey\.

Interestingly, Figure[10](https://arxiv.org/html/2608.11343#A6.F10)shows an example when Nova 2 selected the correct image while all others were incorrect\. This indicates that Nova 2 may be better for certain queries, which can be explored further\. Figure[11](https://arxiv.org/html/2608.11343#A6.F11)and[12](https://arxiv.org/html/2608.11343#A6.F12)show where only the Multimodal Embedding models were correct and only LLMs were correct, respectively\. The following figures provide more examples on the performance of evaluated systems\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/GE2_correct_only_close_with_Claude.png)Figure 8:Another example where Gemini Embedding 2 model is correct; others incorrect\.![Refer to caption](https://arxiv.org/html/2608.11343v1/GE2_correct_only_3.png)Figure 9:An example where all models’ retrieval reasonably close to the query\.![Refer to caption](https://arxiv.org/html/2608.11343v1/Nova2_correct_only.png)Figure 10:Amazon Nova 2 results are correct; Other models’ results are incorrect\.![Refer to caption](https://arxiv.org/html/2608.11343v1/Multimodal_correct_only.png)Figure 11:Multimodal embedding models are correct; LLM Rankers incorrect\.![Refer to caption](https://arxiv.org/html/2608.11343v1/LLMs_correct_only.png)Figure 12:LLM models’ results are correct; Embedding models’ results are incorrect\.### F\.1GE2 correct, all LLMs wrong

Figure[13](https://arxiv.org/html/2608.11343#A6.F13)shows queries where GE2 ranked the correct image first while GPT\-4\.1 and Claude Sonnet 4\.6 both failed\. These cases tend to involve fine\-grained visual similarity where the correct image shares compositional structure or low\-level visual properties with the distractors that GE2’s embedding captures but LLMs miss when reasoning jointly over 25 candidates\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/GE2_correct_only_1.png)\(a\)Example 1
![Refer to caption](https://arxiv.org/html/2608.11343v1/GE2_correct_only_2.png)\(b\)Example 2

Figure 13:Queries where GE2 is correct and both LLMs \(GPT\-4\.1 and Claude Sonnet\) are wrong on R@1\. Green = correct rank\-1, red = incorrect rank\-1\.
### F\.2Nova2 correct, all other systems wrong

Figure[14](https://arxiv.org/html/2608.11343#A6.F14)shows the rare queries where Nova2 ranked correctly while GE2, GPT\-4\.1, and Claude Sonnet 4\.6 all failed\. Given Nova2’s overall lower performance, these cases are uncommon but analytically interesting as they may reflect query types where GENERIC\_INDEX embeddings capture complementary signal\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/Nova2_correct_only_1.png)\(a\)Example 1
![Refer to caption](https://arxiv.org/html/2608.11343v1/Nova2_correct_only.png)\(b\)Example 2

Figure 14:Queries where Nova2 is correct and all other systems are wrong on R@1\.
### F\.3Both LLMs correct, both embedding models wrong

Figure[15](https://arxiv.org/html/2608.11343#A6.F15)shows queries where both GPT\-4\.1 and Claude Sonnet ranked correctly while both GE2 and Nova2 failed\. These cases illustrate the advantage of cross\-candidate joint reasoning: the LLM can compare all 25 images simultaneously and apply semantic or relational understanding that independent embedding scoring cannot replicate\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/LLMs_correct_only.png)\(a\)Example 1
![Refer to caption](https://arxiv.org/html/2608.11343v1/LLMs_correct_only_1.png)\(b\)Example 2

Figure 15:Queries where both LLMs \(GPT\-4\.1 and Claude Sonnet 4\.6\) are correct and both embedding models \(GE2 and Nova2\) are wrong on R@1\.
### F\.4Both embedding models correct, both LLMs wrong

Figure[16](https://arxiv.org/html/2608.11343#A6.F16)shows queries where both GE2 and Nova2 ranked correctly while both LLMs failed\. These cases are particularly informative as they suggest that dense cross\-modal embedding captures visual correspondence that joint visual reasoning consistently misses, even when the pattern holds across model families\.

![Refer to caption](https://arxiv.org/html/2608.11343v1/Multimodal_correct_only_1.png)\(a\)Example 1
![Refer to caption](https://arxiv.org/html/2608.11343v1/Multimodal_correct_only_2.png)\(b\)Example 2

Figure 16:Queries where both embedding models \(GE2 and Nova2\) are correct and both LLMs \(GPT\-4\.1 and Claude Sonnet 4\.6\) are wrong on R@1\.