SIMMER: Cross-Modal Food Image–Recipe Retrieval via MLLM-Based Embedding

# Cross-Modal Food Image–Recipe Retrieval via MLLM-Based Embedding

Source: https://arxiv.org/html/2604.15628

Keisuke Gomi, Keiji Yanai

[email protected], [email protected]

The University of Electro-Communications
1-5-1 Chofugaoka, Chofu-shi, Tokyo 182-8585, Japan

## Abstract

Cross-modal retrieval between food images and recipe texts is an important task with applications in nutritional management, dietary logging, and cooking assistance. Existing methods predominantly rely on dual-encoder architectures with separate image and text encoders, requiring complex alignment strategies and task-specific network designs to bridge the semantic gap between modalities. In this work, we propose SIMMER (Single Integrated Multimodal Model for Embedding Recipes), which applies Multimodal Large Language Model (MLLM)-based embedding models, specifically VLM2Vec, to this task, replacing the conventional dual-encoder paradigm with a single unified encoder that processes both food images and recipe texts. We design prompt templates tailored to the structured nature of recipes, which consist of a title, ingredients, and cooking instructions, enabling effective embedding generation by the MLLM. We further introduce a component-aware data augmentation strategy that trains the model on both complete and partial recipes, improving robustness to incomplete inputs. Experiments on the Recipe1M dataset demonstrate that SIMMER achieves state-of-the-art performance across both the 1k and 10k evaluation settings, substantially outperforming all prior methods. In particular, our best model improves the 1k image-to-recipe R@1 from 81.8% to 87.5% and the 10k image-to-recipe R@1 from 56.5% to 65.5% compared to the previous best method.

![Figure 1: (a) Existing methods and (b) SIMMER (proposed)](caption: Overview of the existing methods and SIMMER)

## 1 Introduction

Food is an essential part of daily life, and interest in healthy eating habits continues to grow worldwide. In recent years, recipe-sharing platforms and social media have made vast quantities of food images and recipe texts publicly available, driving an increasing demand for techniques that can effectively leverage such multimodal data. In this context, cross-modal retrieval between food images and recipe texts has emerged as an important research problem. This task aims to enable bidirectional retrieval: given a food image, the system retrieves the corresponding cooking recipe, and conversely, given a recipe text, it retrieves the matching food image. Such capabilities have broad practical applications, including nutritional management, dietary logging, and cooking assistance.

Meanwhile, in the fields of computer vision and natural language processing, Multimodal Large Language Models (MLLMs) such as LLaVA and Qwen2-VL have advanced rapidly, demonstrating remarkable abilities in the integrated understanding of images and text. More recently, a line of research has explored repurposing MLLMs as embedding models. Methods such as VLM2Vec and MM-Embed have shown that the rich multimodal knowledge acquired during large-scale pre-training can be effectively transferred to produce high-quality multimodal embeddings.

However, existing methods for cross-modal food image–recipe retrieval predominantly rely on dual-encoder architectures, in which separate image and text encoders are independently designed and trained, as illustrated in Figure 1(a). To our knowledge, MLLMs have not yet been applied to this task. Moreover, dual-encoder approaches typically require elaborate alignment strategies and task-specific network designs to bridge the semantic gap between the visual and textual modalities. For instance, ACME and R2GAN employ adversarial training to align the feature distributions across modalities. Meanwhile, H-T and T-Food introduce hierarchical Transformer architectures specifically tailored to the structured nature of recipe texts comprising titles, ingredients, and instructions.

In this work, we propose SIMMER (Single Integrated Multimodal Model for Embedding Recipes), which applies MLLM-based embedding models—specifically VLM2Vec-V1 and VLM2Vec-V2—to cross-modal food image–recipe retrieval, as depicted in Figure 1(b). Since MLLMs are pre-trained on large-scale multimodal corpora, they inherently possess the ability to process both images and text within a unified framework. By leveraging this pre-trained knowledge, SIMMER eliminates the need for the complex alignment mechanisms and specialized architectures that have been essential in prior work, while still obtaining effective cross-modal embeddings.

We validate the effectiveness of SIMMER through extensive experiments on Recipe1M, a large-scale benchmark dataset for food image–recipe retrieval. The main contributions of this paper are summarized as follows:

- We are the first to apply MLLM-based embedding models to cross-modal food image–recipe retrieval, demonstrating that a single unified encoder can replace the conventional dual-encoder paradigm.
- We propose a method for encoding structured recipe data—consisting of a title, ingredients, and cooking instructions—into a format suitable for MLLM-based embedding generation, along with a component-aware data augmentation strategy to improve robustness to incomplete recipes.
- SIMMER achieves state-of-the-art performance on the Recipe1M dataset, substantially outperforming all existing methods across both image-to-recipe and recipe-to-image retrieval tasks.

## 2 Related Work

### 2.1 Cross-Modal Image–Text Embeddings

Cross-modal embedding of images and text has long been an active area of research. Early work on visual–semantic embeddings pioneered the idea of projecting images and text into a shared vector space. DeViSE trained a linear mapping from CNN image features to a word-embedding space learned by a skip-gram language model, enabling zero-shot recognition of unseen object categories. Kiros et al. extended this idea by jointly learning image and sentence encoders with a contrastive ranking objective, establishing the Visual Semantic Embedding (VSE) framework widely adopted in subsequent research. VSE++ improved training efficiency by introducing hard-negative mining into the ranking loss. SCAN further advanced fine-grained matching by employing stacked cross-attention between image regions detected by object detectors and individual words, achieving strong performance on image–text retrieval benchmarks.

Beyond dual-encoder designs, a line of work explored early-fusion Transformer architectures for vision–language understanding. ViLBERT introduced a two-stream Transformer with co-attentional layers pre-trained on image–text pairs, while UNITER adopted a single-stream Transformer to learn universal image–text representations via masked language/region modeling and image–text matching. These pre-trained multimodal Transformers substantially improved downstream vision–language tasks but required costly cross-attention at inference time, limiting their use in large-scale retrieval.

Building on these foundations, a new generation of models scaled up contrastive dual-encoder pre-training to web-scale data. Representative methods include CLIP, ALIGN, BLIP, and SigLIP, which encode images and text through independent encoders and align them in a shared representation space via contrastive learning. While these models have achieved remarkable success across a wide range of vision–language tasks, their dual-encoder design inherently limits the degree of cross-modal interaction, making it difficult to capture fine-grained correspondences between visual and linguistic content.

With the rapid advancement of Multimodal Large Language Models (MLLMs), a new paradigm has emerged that repurposes MLLMs as embedding models. VLM2Vec fine-tunes pre-trained MLLMs—such as Phi-3.5, LLaVA, and Qwen2-VL—on a large-scale multimodal embedding dataset, substantially outperforming conventional CLIP-based models on multimodal embedding benchmarks. Building upon this, VLM2Vec-V2 further expands the training data to encompass not only images and text but also videos and visual documents, broadening the applicability of MLLM-based embeddings. These methods demonstrate that the rich world knowledge and deep cross-modal understanding acquired during MLLM pre-training can be effectively distilled into dense vector representations. Motivated by these advances, SIMMER applies MLLM-based embedding models to the cross-modal retrieval of food images and recipe texts—a domain in which only dual-encoder approaches have been explored to date.

### 2.2 Cross-Modal Food Image–Recipe Retrieval

Cross-modal retrieval between food images and recipe texts has become an actively studied task since the release of the Recipe1M dataset. Compared to generic image–text retrieval, a key characteristic of this problem is that recipe texts are inherently structured, typically consisting of a title, a list of ingredients, and a sequence of cooking instructions. Accordingly, progress in this area has been closely tied to (i) how each modality is encoded and (ii) how cross-modal alignment objectives exploit this structured text.

Early approaches largely followed a CNN–RNN recipe encoding pipeline: image encoders were typically CNNs pre-trained on ImageNet, such as VGG or ResNet, while recipe encoders used Word2Vec embeddings followed by LSTMs. Within this paradigm, Salvador et al. introduced a joint embedding model trained with a cosine-similarity-based ranking objective augmented by semantic regularization. AdaMine proposed a joint retrieval-and-classification framework with a double-triplet learning scheme and adaptive triplet mining. Several subsequent works improved alignment or robustness by adding adversarial and generative auxiliary objectives, modeling cooking procedures more explicitly, introducing noise-robust and sentence-level recipe modeling, or incorporating stochastic latent variables to capture cross-modal interactions while keeping inference efficient. While they were effective, these early recipe encoders generally relied on limited language pre-training (e.g., Word2Vec) and struggled to capture richer compositional semantics of long, structured recipes.

A second wave of work emphasized richer recipe modeling and finer-grained image–recipe correspondences. This includes structure-aware recipe encoders, such as tree-structured LSTMs that capture hierarchical relationships among recipe components, as well as Transformer-based multilingual recipe encoders with translation-based regularization as in X-MRS. In parallel, many methods focused on designing stronger cross-modal interaction and alignment mechanisms (e.g., hybrid fusion and intra-/inter-modality attention, multi-subspace implicit alignment, modality-alignment regularizations and improved metric learning objectives, semantic consistency with attention for discriminative recipe representations, and disentanglement-based generation as an auxiliary signal). With the rise of the Transformer architecture, stronger backbones became increasingly common on both modalities. On the recipe side, the Hierarchical Transformer (H-T) became influential by encoding recipe components (title, ingredients, instructions) and introducing a self-supervised objective over recipe components, enabling the use of both paired image–recipe data and recipe-only samples. On top of such modern recipe encoders, some methods explicitly model cross-modal interactions during training while retaining efficient unimodal encoding at inference, e.g., Transformer decoders used as a multimodal regularizer in T-Food.

Around the same time, other complementary directions were explored: non-parametric retrieval on top of precomputed embeddings with cross-modal kNN-type alignment (DaC), event-oriented modality alignment for event-dense text–image retrieval validated in the food domain (EOMA), TF-IDF-enhanced recipe representations for joint embedding learning (MSJE), and structured "cooking program" representations to enrich procedural semantics. More recent Transformer-based pipelines further refine recipe encoders and training regimes (e.g., large-batch training and improved component-level learning). In recent years, large-scale vision–language pre-training, most notably CLIP, has become a powerful backbone for food retrieval when adapted to structured recipes. Representative examples include component-aware prompt learning to leverage vision–language pre-trained models without full