搜索帕累托前沿:面向异构科学模态的多目标检索

arXiv cs.AI 论文

摘要

本文比较了多模态基础模型嵌入与通过基于帕累托的多目标优化对分别训练的模态嵌入进行后期融合这两种方法在跨模态星系检索中的表现,发现多目标方法保留了各模态特有的结构,且检索到的近邻在原始数据空间中的距离缩短了8%。

arXiv:2611.23639v1 Announce Type: new Abstract: In cosmology, surveys span different sky regions and wavelength ranges, resulting in uneven modality coverage across observational datasets. Such heterogeneity can be addressed through either multimodal foundation models (FMs), which learn shared embedding spaces, or independently learned modality embeddings combined via late fusion. We investigate whether the latter provides an adaptive alternative to FM-based scientific retrieval. We evaluate this approach on a galaxy retrieval task spanning diverse modalities--history of star formation, spectra of light emitted, and multi-wavelength observations from two telescopes--comparing an FM baseline with separately trained embeddings fused through multi-objective optimization based on Pareto optimality. We find that the FM baseline achieves strong embedding-space proximity, but our method preserves modality-specific distance structure that allows scientists to inspect which physical signals drive similarity. Our method also achieves improved fidelity in raw-data space: across six measured properties, its nearest neighbor is 8% closer to the query on average than that retrieved by kNN in the FM embedding space (spread 0.128 vs. 0.139). These results suggest that multi-objective retrieval can surface astrophysically coherent candidates that single-objective search alone may overlook.
查看原文
查看缓存全文

缓存时间: 2026/11/20 09:07

# Searching the Pareto Front: Multi-Objective Retrieval for Heterogeneous Scientific Modalities
Source: [https://arxiv.org/html/2611.23639](https://arxiv.org/html/2611.23639)
Song Young OhEmail:[so27@uchicago\.edu](mailto:[email protected])Affiliation:Department of Computer Science, University of Chicago, 5730 S Ellis Ave, Chicago, 60637, IL, USANesar RamachandraEmail:[nramachandra@anl\.gov](mailto:[email protected])Affiliation:Computational Science Division, Argonne National Laboratory, 9700 S Cass Ave, Lemont, 60439, IL, USABin XiaEmail:[xiabin@gatech\.edu](mailto:[email protected])Affiliation:School of Physics, Georgia Institute of Technology, 837 State St NW, Atlanta, 30332, GA, USAIan FosterEmail:[foster@anl\.gov](mailto:[email protected])Affiliation:Department of Computer Science, University of Chicago, 5730 S Ellis Ave, Chicago, 60637, IL, USAAffiliation:Data Science and Learning Division, Argonne National Laboratory, 9700 S Cass Ave, Lemont, 60439, IL, USAKyle ChardEmail:[chard@uchicago\.edu](mailto:[email protected])Affiliation:Department of Computer Science, University of Chicago, 5730 S Ellis Ave, Chicago, 60637, IL, USAAffiliation:Data Science and Learning Division, Argonne National Laboratory, 9700 S Cass Ave, Lemont, 60439, IL, USA

###### Abstract

In cosmology, surveys span different sky regions and wavelength ranges, resulting in uneven modality coverage across observational datasets\. Such heterogeneity can be addressed through either multimodal foundation models \(FMs\), which learn shared embedding spaces, or independently learned modality embeddings combined via late fusion\. We investigate whether the latter provides an adaptive alternative to FM\-based scientific retrieval\. We evaluate this approach on a galaxy retrieval task spanning diverse modalities—history of star formation, spectra of light emitted, and multi\-wavelength observations from two telescopes—comparing an FM baseline with separately trained embeddings fused through multi\-objective optimization based on Pareto optimality\. We find that the FM baseline achieves strong embedding\-space proximity, but our method preserves modality\-specific distance structure that allows scientists to inspect which physical signals drive similarity\. Our method also achieves improved fidelity in raw\-data space: across six measured properties, its nearest neighbor is 8% closer to the query on average than that retrieved by kNN in the FM embedding space \(spread 0\.128 vs\. 0\.139\)\. These results suggest that multi\-objective retrieval can surface astrophysically coherent candidates that single\-objective search alone may overlook\.

###### keywords

Multimodal Retrieval, Late Fusion, Multi\-Objective Optimization, Foundation Models, Scientific Data

### 1Introduction

Cross\-modal retrieval is often conducted using nearest\-neighbor search in a unified embedding space learned by a multimodal foundation model \(FM\)[Lin et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib49);[Jia et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib3)\. In this paradigm, heterogeneous modalities are mapped to a shared representation, where retrieval can be performed with similarity metrics[Wang et al\. \(2016\)](https://arxiv.org/html/2611.23639#bib.bib45);[Wang et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib61)\. This approach has been highly effective for domains in which modalities are well aligned and jointly available during training[Radford et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib2);[Li et al\. \(2022\)](https://arxiv.org/html/2611.23639#bib.bib4);[Girdhar et al\. \(2023\)](https://arxiv.org/html/2611.23639#bib.bib5)\. Scientific data, however, often violate these assumptions\. Observations are incomplete, irregular, and shaped by modality\-specific measurement processes\. In astronomy, for example, telescope surveys differ in sky coverage, wavelength range, spatial resolution, and observational depth[Tyson \(2002\)](https://arxiv.org/html/2611.23639#bib.bib60);[Shanks et al\. \(2015\)](https://arxiv.org/html/2611.23639#bib.bib54)\. As a result, the same galaxy may be observed through only a subset of possible measurements, making globally aligned representations challenging to learn and difficult to maintain as new instruments or derived products become available\.

Scientific retrieval is challenging because similarity may be defined differently across modalities\. General\-purpose FMs are designed to support broad downstream use, including missing\-modality prediction and fine\-tuning on limited data[Li et al\. \(2024\)](https://arxiv.org/html/2611.23639#bib.bib50);[Bommasani et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib46); however, retrieval imposes a more specific requirement: the induced neighborhood structure should preserve scientifically meaningful similarity relationships[Wu et al\. \(2024\)](https://arxiv.org/html/2611.23639#bib.bib47)\. This requirement is non\-trivial to satisfy when modalities differ substantially in scale, noise, coverage and physical interpretation\. Moreover, each modality may have its own embedding geometry, making direct comparison or aggregation across modalities poorly defined[Jha et al\. \(2026\)](https://arxiv.org/html/2611.23639#bib.bib55);[Poklukar et al\. \(2022\)](https://arxiv.org/html/2611.23639#bib.bib53)\. Collapsing these modality\-specific distances into a single distance measure can obscure scientifically relevant structure, while full retraining or re\-alignment may be impractical when modalities evolve[Xu et al\. \(2024\)](https://arxiv.org/html/2611.23639#bib.bib48)\. These challenges call for retrieval methods that can operate over available modalities while preserving their distinct geometric contributions\.

To address this gap, we propose ParetoFuse, a multimodal retrieval framework that treats per\-modality similarities as complementary signals rather than forcing them into a single global representation[Fonseca and Fleming \(1995\)](https://arxiv.org/html/2611.23639#bib.bib59);[Miettinen \(1999\)](https://arxiv.org/html/2611.23639#bib.bib51)\. Each modality produces its own distance estimate, and candidate objects are compared in the resulting multi\-dimensional distance space\. Pareto dominance is then used to identify candidates that are not uniformly worse than others across the available modalities[Censor \(1977\)](https://arxiv.org/html/2611.23639#bib.bib56)\. Our design applies a lightweight late fusion mechanism for heterogeneous scientific data: it preserves modality\-specific geometry while still producing a coherent candidate set for retrieval\.

In this paper, we make the following contributions\. We introduce a multi\-objective formulation of heterogeneous scientific retrieval that retains modality\-specific distances\. We apply a Pareto\-based reranking, including a post\-processing step that selects precise neighbors from the non\-dominated candidate set\. We evaluate our proposed approach on a galaxy retrieval task against MOSAIC\-lite, a lightweight variant of the MOSAIC multimodal FM[Xia et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib1), using both embedding\-space and raw\-data metrics, and find that our method preserves modality\-specific distance structure while reducing average nearest neighbor spread in raw\-data space by 8%\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2611.23639#S2)reviews related work on multimodal representation learning and late fusion for retrieval\. Section[3](https://arxiv.org/html/2611.23639#S3)presents ParetoFuse, including independent embedding generation, one\-dimensional distance projection, and multi\-objective reranking methods\. Section[4](https://arxiv.org/html/2611.23639#S4)describes the galaxy retrieval task, baselines, and evaluation metrics\. Section[5](https://arxiv.org/html/2611.23639#S5)reports the empirical comparison of FM retrieval and multi\-objective retrieval using independent embeddings\. Section[6](https://arxiv.org/html/2611.23639#S6)discusses the implications for scientific retrieval, Section[7](https://arxiv.org/html/2611.23639#S7)summarizes limitations, and Section[8](https://arxiv.org/html/2611.23639#S8)concludes the paper\. Appendix[9](https://arxiv.org/html/2611.23639#S9)provides additional details on MOSAIC and MOSAIC\-lite\.

### 2Related Work

#### 2\.1Multimodal Representation Learning

Multimodal representation learning seeks to align heterogeneous data types in a shared embedding space[Guo et al\. \(2019\)](https://arxiv.org/html/2611.23639#bib.bib62)\. Contrastive learning has become a central paradigm for this goal, using paired observations to bring semantically related samples closer while separating unrelated ones[Singh et al\. \(2023\)](https://arxiv.org/html/2611.23639#bib.bib29)\. Large\-scale models such as CLIP[Radford et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib2)and ALIGN[Jia et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib3)demonstrated the effectiveness of this approach for vision–language alignment\. Subsequent models have extended the paradigm through richer training objectives and broader modality coverage\. For example, BLIP[Li et al\. \(2022\)](https://arxiv.org/html/2611.23639#bib.bib4)combines contrastive and generative objectives, while ImageBind[Girdhar et al\. \(2023\)](https://arxiv.org/html/2611.23639#bib.bib5)extends joint embedding learning across multiple sensory modalities\. These systems establish a common retrieval design: modality\-specific encoders map inputs into a shared latent space, and retrieval is then performed by nearest\-neighbor search under a single similarity metric\.

Scientific domains have increasingly adopted this shared\-embedding paradigm, but under substantially different data conditions[Leung and Bovy \(2024\)](https://arxiv.org/html/2611.23639#bib.bib32)\. In science, modalities often correspond to distinct instruments, measurement processes, spatial or temporal resolutions, and physical quantities\. Recent work on omnimodal foundation models aims to address this heterogeneity by learning unified representations across diverse scientific signals[Luo et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib43);[Tyagi and Sharma \(2025\)](https://arxiv.org/html/2611.23639#bib.bib7)\. In astronomy, models such as AION\-1[Parker et al\. \(2026\)](https://arxiv.org/html/2611.23639#bib.bib44), AstroCLIP[Parker et al\. \(2024\)](https://arxiv.org/html/2611.23639#bib.bib6)and MOSAIC[Xia et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib1)learn joint embeddings for domain\-specific modalities, supporting cross\-modal retrieval at large dataset scale\. These approaches are effective when modalities can be aligned through paired supervision\. Our work instead considers a different setting in which joint training is not feasible, and retrieval must instead operate flexibly over independently learned modality representations\.

#### 2\.2Late Fusion for Scientific Retrieval

Late fusion provides a natural alternative when modalities are incomplete and independently maintained\. Instead of learning a single joint representation, late\-fusion methods compute modality\-specific similarity scores and combine them at inference time[Ye et al\. \(2012\)](https://arxiv.org/html/2611.23639#bib.bib34);[Zhao et al\. \(2024\)](https://arxiv.org/html/2611.23639#bib.bib33)\. This separation between representation learning and decision\-level integration is attractive in scientific workflows, where data sources may be distributed, updated asynchronously, or available only for subsets of samples[Li and Tang \(2024\)](https://arxiv.org/html/2611.23639#bib.bib12);[Jin et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib13);[Oh et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib8)\. It also allows new modalities or revised encoders to be incorporated without retraining an entire multimodal model[Lahat et al\. \(2015\)](https://arxiv.org/html/2611.23639#bib.bib40);[Atrey et al\. \(2010\)](https://arxiv.org/html/2611.23639#bib.bib9)\.

A central challenge in late fusion is score heterogeneity\. Similarity scores from different feature spaces may differ in scale, density, and discriminative behavior[Snoek et al\. \(2005\)](https://arxiv.org/html/2611.23639#bib.bib14)\. Classical methods address this issue through normalization and weighted combinations, including linear score fusion and related ensemble methods[Jain et al\. \(2005\)](https://arxiv.org/html/2611.23639#bib.bib35);[Escalante et al\. \(2008\)](https://arxiv.org/html/2611.23639#bib.bib30)\. Other approaches model score distributions probabilistically[Nandakumar et al\. \(2008\)](https://arxiv.org/html/2611.23639#bib.bib37), learn supervised fusion weights[Terrades et al\. \(2008\)](https://arxiv.org/html/2611.23639#bib.bib39);[Shoukat et al\. \(2022\)](https://arxiv.org/html/2611.23639#bib.bib10);[Gialampoukidis et al\. \(2016\)](https://arxiv.org/html/2611.23639#bib.bib31);[Hori et al\. \(2017\)](https://arxiv.org/html/2611.23639#bib.bib11), or project modalities into shared latent subspaces[Wright et al\. \(2009\)](https://arxiv.org/html/2611.23639#bib.bib38);[Dong et al\. \(2018\)](https://arxiv.org/html/2611.23639#bib.bib36)\.

Multi\-objective optimization \(MOO\) offers a complementary view\. Rather than resolving heterogeneous evidence through calibration, weighting, or projection, MOO treats each objective separately and identifies non\-dominated trade\-offs[Fonseca and Fleming \(1995\)](https://arxiv.org/html/2611.23639#bib.bib59);[Diwekar \(2003\)](https://arxiv.org/html/2611.23639#bib.bib52);[Marler and Arora \(2004\)](https://arxiv.org/html/2611.23639#bib.bib25);[Deb \(2011\)](https://arxiv.org/html/2611.23639#bib.bib15);[Giagkiozis and Fleming \(2015\)](https://arxiv.org/html/2611.23639#bib.bib42)\. A candidate is Pareto\-optimal when no other candidate improves one objective without degrading another[Miettinen \(1999\)](https://arxiv.org/html/2611.23639#bib.bib51);[Qu et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib18);[Gunantara \(2018\)](https://arxiv.org/html/2611.23639#bib.bib41)\. This principle has been applied in retrieval and recommendation settings that balance criteria such as relevance, diversity, and fairness[Dai et al\. \(2011\)](https://arxiv.org/html/2611.23639#bib.bib19);[Arevalillo\-Herráez et al\. \(2013\)](https://arxiv.org/html/2611.23639#bib.bib20);[Lerher et al\. \(2013\)](https://arxiv.org/html/2611.23639#bib.bib21);[van Doorn et al\. \(2016\)](https://arxiv.org/html/2611.23639#bib.bib22);[Borovinšek et al\. \(2017\)](https://arxiv.org/html/2611.23639#bib.bib23);[Rodriguez and Carver \(2020\)](https://arxiv.org/html/2611.23639#bib.bib24)\. In scientific retrieval, this perspective is useful because disagreement across modalities may reflect meaningful physical variation rather than noise or calibration error\. We adopt this approach for multimodal retrieval by preserving modality\-specific distances as separate retrieval objectives\. Candidates are then selected as Pareto\-optimal trade\-offs across modalities\.

### 3Methods

In this section, we first define the multimodal retrieval setting and notation of ParetoFuse\. We then describe how modality\-specific embeddings are learned independently, how these embeddings are converted into one\-dimensional distance signals, and how candidates are ranked using multi\-objective retrieval\.

#### 3\.1Problem Formulation

We consider retrieval over objects with heterogeneous modality observations\. Our goal is to rank corpus objects by their modality\-specific similarity to a query, without reducing all modalities to a single shared embedding space\.

LetMMdenote the number of modalities\. Each modalitym∈\{1,…,M\}m\\in\\\{1,\\ldots,M\\\}has an input space𝒳\(m\)\\mathcal\{X\}^\{\(m\)\}and an encoder

f\(m\):𝒳\(m\)→ℝdm\.f^\{\(m\)\}:\\mathcal\{X\}^\{\(m\)\}\\rightarrow\\mathbb\{R\}^\{d\_\{m\}\}\.\(1\)The encoder maps observations from modalitymmto admd\_\{m\}\-dimensional embedding space\. An objectxxmay contain observations from several modalities and is written asx=\(x\(1\),…,x\(M\)\)x=\\left\(x^\{\(1\)\},\\ldots,x^\{\(M\)\}\\right\), wherex\(m\)∈𝒳\(m\)x^\{\(m\)\}\\in\\mathcal\{X\}^\{\(m\)\}when modalitymmis available\. In scientific datasets, not every object is observed in every modality\. We therefore letΩ⁡\(q,x\)⊆\{1,…,M\}\\Omega\(q,x\)\\subseteq\\\{1,\\ldots,M\\\}denote the set of modalities available for both the queryqqand corpus itemxx\. For each available modality, we define a modality\-specific distance

Δm​\(q,x\)=δm​\(f\(m\)​\(q\(m\)\),f\(m\)​\(x\(m\)\)\),m∈Ω⁡\(q,x\),\\Delta\_\{m\}\(q,x\)=\\delta\_\{m\}\\left\(f^\{\(m\)\}\(q^\{\(m\)\}\),f^\{\(m\)\}\(x^\{\(m\)\}\)\\right\),\\quad m\\in\\Omega\(q,x\),\(2\)whereδm\\delta\_\{m\}is the distance function used in the embedding space of modalitymm\. Smaller values ofΔm​\(q,x\)\\Delta\_\{m\}\(q,x\)indicate greater similarity betweenqqandxxunder modalitymm\.

Rather than collapsing modality\-specific distances into one global score, we treat each shared modality as a separate retrieval objective\. For queryqqand candidatexx, we define the distance profile

𝚫⁡\(q,x\)=\(Δm​\(q,x\)\)m∈Ω⁡\(q,x\)\.\\mathbf\{\\Delta\}\(q,x\)=\\left\(\\Delta\_\{m\}\(q,x\)\\right\)\_\{m\\in\\Omega\(q,x\)\}\.\(3\)This vector records the candidate’s distance to the query in each modality\. Given a retrieval corpus𝒞\\mathcal\{C\}, the goal is to rank candidatesx∈𝒞x\\in\\mathcal\{C\}using these modality\-specific distance profiles\.

#### 3\.2Independent Modality Embeddings

Table 1:Summary of the four autoencoder modalities\. Encoder structures are fully connected neural networks; dimensions denote layer widths from input to bottleneck\. The first dimension corresponds to the dimensionality of each modality\.To obtain independent modality embeddings, we learn a separate representation per modality\. For each modalitymm, we train an autoencoder with encoderf\(m\)f^\{\(m\)\}and decoderh\(m\)h^\{\(m\)\}[Rumelhart et al\. \(1986\)](https://arxiv.org/html/2611.23639#bib.bib57);[Hinton and Salakhutdinov \(2006\)](https://arxiv.org/html/2611.23639#bib.bib58)\. The decoder is used only during training; after training, we use the bottleneck representationf\(m\)​\(x\(m\)\)f^\{\(m\)\}\(x^\{\(m\)\}\)as the modality\-specific embedding\.[Table 1](https://arxiv.org/html/2611.23639#S3.T1.fig1)summarizes the four modalities and encoder architectures we use in this work\.

The reconstruction objective for modalitymmis

ℒAE\(m\)=1Nm​∑i=1Nmℓβ​\(xi\(m\),h\(m\)​\(f\(m\)​\(xi\(m\)\)\)\),\\mathcal\{L\}^\{\(m\)\}\_\{\\mathrm\{AE\}\}=\\frac\{1\}\{N\_\{m\}\}\\sum\_\{i=1\}^\{N\_\{m\}\}\\ell\_\{\\beta\}\\left\(x\_\{i\}^\{\(m\)\},h^\{\(m\)\}\(f^\{\(m\)\}\(x\_\{i\}^\{\(m\)\}\)\)\\right\),\(4\)whereNmN\_\{m\}is the number of training samples for modalitymm, andℓβ\\ell\_\{\\beta\}is the SmoothL1 loss withβ=1\\beta=1\. SmoothL1 is used instead of mean\-squared error because it is less affected by extreme residuals, making training more stable across heterogeneous modalities\.

Each decoder mirrors its corresponding encoder\. Hidden layers use ReLU activations; bottleneck and output layers are linear\. We train all models with Adam, using learning rate10−310^\{\-3\}, batch size 1024, and 50 epochs, and retain the checkpoint with the lowest validation loss for each modality\. The resulting training and validation reconstruction curves are shown in[Fig\. 1](https://arxiv.org/html/2611.23639#S3.F1)\. The loss corresponding to star formation history shows the least optimization because it is less informative compared to other modalities, and has considerably lower signal\-to\-noise ratio\. While the hyper\-parameters and architectures of the individual embedding networks are tuned by\-hand, automated searches can be utilized as well\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_loss_curves_log.png)Figure 1:Training and validation reconstruction losses for the independently trained autoencoders\. Each model is trained only on its corresponding modality, and the bottleneck representation is used as the modality\-specific embedding\.To qualitatively compare the latent geometries induced by independent modality training and the shared FM,[Fig\. 2](https://arxiv.org/html/2611.23639#S3.F2)visualizes the corresponding𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}embedding spaces\. The distinct UMAP structures suggest that the two representations organize the data differently, although the projected coordinates and interpoint distances do not have direct physical interpretations\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_umap.png)Figure 2:UMAP visualization of𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}embeddings for 5,000 galaxy objects\. Left: independently trained embeddings\. Right: foundation model embeddings\. Points are colored by redshift values\.
#### 3\.3Projection to One\-Dimensional Signals

We reduce each modality embedding to a one\-dimensional representation, allowing embeddings of different dimensionalities to serve as comparable retrieval objectives\. Query–candidate distances are then computed separately for each modality, yielding one scalar distance per modality while preserving them as distinct objectives for subsequent multi\-objective reranking\. For modalitymm, let

g\(m\):ℝdm→ℝg^\{\(m\)\}:\\mathbb\{R\}^\{d\_\{m\}\}\\rightarrow\\mathbb\{R\}\(5\)denote a modality\-specific projector\. The projected coordinate of itemxxis

zm​\(x\)=g\(m\)​\(f\(m\)​\(x\(m\)\)\),z\_\{m\}\(x\)=g^\{\(m\)\}\\left\(f^\{\(m\)\}\(x^\{\(m\)\}\)\\right\),\(6\)and the corresponding query–candidate distance is

dm​\(q,x\)=\|zm​\(q\)−zm​\(x\)\|\.d\_\{m\}\(q,x\)=\\left\|z\_\{m\}\(q\)\-z\_\{m\}\(x\)\\right\|\.\(7\)The projected distance profile used for reranking is

𝐝⁡\(q,x\)=\(dm​\(q,x\)\)m∈Ω⁡\(q,x\)\.\\mathbf\{d\}\(q,x\)=\\left\(d\_\{m\}\(q,x\)\\right\)\_\{m\\in\\Omega\(q,x\)\}\.\(8\)
In this work, we evaluate two projectors with different modeling assumptions\. The first is a neural projector, defined as a nonlinear mapping implemented by a multilayer perceptron with hidden dimensions 128 and 64\. It is trained independently for each modality to preserve pairwise distances from the original embedding space after projection\. The second is a PCA projector, defined by the leading principal component of each modality embedding space, and provides a linear baseline\. This comparison assesses whether the retrieval stage benefits from nonlinear projection or whether a linear dominant\-direction projection is sufficient\.

#### 3\.4Multi\-Objective Reranking

Multi\-objective reranking retains modality\-specific similarity information during candidate selection\. Because different modalities may encode complementary structure, converting them into a single score can obscure meaningful trade\-offs among candidates\. We therefore rerank candidates using their projected distance profiles, treating each modality as a separate objective\.

Given a queryqq, each candidatex∈𝒞x\\in\\mathcal\{C\}is represented by the projected distance vector𝐝⁡\(q,x\)\\mathbf\{d\}\(q,x\)\. We evaluate two reranking families: Pareto\-based methods, which preserve the vector\-valued distance profile, and scalarized methods, which reduce it to a scalar value\.

##### 3\.4\.1Pareto Reranking

Pareto reranking identifies candidates whose modality\-wise distance profiles are not dominated by any other candidate[Fonseca and Fleming \(1995\)](https://arxiv.org/html/2611.23639#bib.bib59);[Miettinen \(1999\)](https://arxiv.org/html/2611.23639#bib.bib51);[Censor \(1977\)](https://arxiv.org/html/2611.23639#bib.bib56)\. For a queryqq, candidatexax\_\{a\}Pareto\-dominates candidatexbx\_\{b\}, written asxa≺qxbx\_\{a\}\\prec\_\{q\}x\_\{b\}, if

dm​\(q,xa\)≤dm​\(q,xb\)∀m∈Ω⁡\(q,xa,xb\),d\_\{m\}\(q,x\_\{a\}\)\\leq d\_\{m\}\(q,x\_\{b\}\)\\quad\\forall m\\in\\Omega\(q,x\_\{a\},x\_\{b\}\),\(9\)and there exists at least one modalitym⋆∈Ω⁡\(q,xa,xb\)m^\{\\star\}\\in\\Omega\(q,x\_\{a\},x\_\{b\}\)such that

dm⋆​\(q,xa\)<dm⋆​\(q,xb\)\.d\_\{m^\{\\star\}\}\(q,x\_\{a\}\)<d\_\{m^\{\\star\}\}\(q,x\_\{b\}\)\.\(10\)Here,Ω⁡\(q,xa,xb\)\\Omega\(q,x\_\{a\},x\_\{b\}\)denotes the modalities available for the query and both candidates\. Thus,xax\_\{a\}dominatesxbx\_\{b\}when it is no farther from the query in every modality and strictly closer in at least one\. The Pareto front for queryqqis the set of non\-dominated candidates:

𝒫\(q\)=\{x∈𝒞:∄x′∈𝒞such thatx′≺qx\}\.\\mathcal\{P\}\(q\)=\\left\\\{x\\in\\mathcal\{C\}:\\nexists x^\{\\prime\}\\in\\mathcal\{C\}\\text\{ such that \}x^\{\\prime\}\\prec\_\{q\}x\\right\\\}\.\(11\)
The resulting front contains candidates for which no other candidate provides an unambiguous improvement across the available modalities\. As shown in[Fig\. 3](https://arxiv.org/html/2611.23639#S3.F3), the size of this non\-dominated candidate set varies across queries\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_pareto_front.png)Figure 3:Distribution of Pareto front sizes across 1,000 queries over a corpus of 5,000 items\. Front sizes are computed in the four\-dimensional projected distance space\.
##### 3\.4\.2Scalarized Reranking

We include scalarized reranking as a comparison to Pareto retrieval\. Scalarized methods reduce the projected distance vector𝐝⁡\(q,x\)\\mathbf\{d\}\(q,x\)to a single score and rank candidates by increasing score\. To reduce scale effects, we use normalized distancesd~m​\(q,x\)\\tilde\{d\}\_\{m\}\(q,x\), obtained by dividingdm​\(q,x\)d\_\{m\}\(q,x\)by the empirical mean distance for modalitymmover the evaluation corpus\. We consider three methods:

\(1\) Weighted sum\.This method ranks candidates by

swsum​\(q,x\)=∑m∈Ω⁡\(q,x\)d~m​\(q,x\),s^\{\\mathrm\{wsum\}\}\(q,x\)=\\sum\_\{m\\in\\Omega\(q,x\)\}\\tilde\{d\}\_\{m\}\(q,x\),\(12\)favoring candidates with small average distance across modalities[Marler and Arora \(2010\)](https://arxiv.org/html/2611.23639#bib.bib26)\.

\(2\) Chebyshev\.This method ranks candidates by

scheb​\(q,x\)=maxm∈Ω⁡\(q,x\)⁡d~m​\(q,x\),s^\{\\mathrm\{cheb\}\}\(q,x\)=\\max\_\{m\\in\\Omega\(q,x\)\}\\tilde\{d\}\_\{m\}\(q,x\),\(13\)penalizing the largest modality\-specific mismatch[Boyd \(2001\)](https://arxiv.org/html/2611.23639#bib.bib27)\.

\(3\)pp\-norm\.This method ranks candidates by

sp​\-norm​\(q,x\)=\(∑m∈Ω⁡\(q,x\)d~m​\(q,x\)p\)1/p\.s^\{p\\text\{\-norm\}\}\(q,x\)=\\left\(\\sum\_\{m\\in\\Omega\(q,x\)\}\\tilde\{d\}\_\{m\}\(q,x\)^\{p\}\\right\)^\{1/p\}\.\(14\)withp=2p=2in our experiments\. Largerppvalues make the score more sensitive to large modality\-specific mismatches[Gentile and Littlestone \(1999\)](https://arxiv.org/html/2611.23639#bib.bib28)\.

#### 3\.5Pareto Gold Refinement

The Pareto front has query\-dependent size and therefore does not directly produce a fixed\-KKretrieval list\. We define*Pareto Gold*as a two\-stage refinement: first compute the Pareto front𝒫⁡\(q\)\\mathcal\{P\}\(q\), then rank only the non\-dominated candidates by nearest\-neighbor distance in the concatenated independent embedding space\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_pareto_log.png)Figure 4:Pairwise 2D views of modality\-specific distances from a query galaxy across four modalities:𝐟sph\\mathbf\{f\}\_\{\\rm sph\},𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\},𝝍\\boldsymbol\{\\psi\}, and𝐟λ\\mathbf\{f\}\_\{\\lambda\}\. Each axis shows a log\-scaled distance in the corresponding modality units\. Each panel shows one modality pair\. Gray points are corpus items, and the query appears at the origin because its self\-distance is zero\. Red circles indicate items on the first Pareto front, while the dashed orange curve shows the second Pareto front for reference\. Green triangles denote multimodal FMkkNN results, whereas yellow diamonds denote results from the*Pareto Gold*refinement of ParetoFuse, obtained by applyingkkNN \(k=10k=10\) within the Pareto set\. Because Pareto retrieval is computed in the full 4D distance space, Pareto\-optimal items need not form a visible front in 2D projection\.For fixed\-size output,*Pareto Gold*returns

ℛKPG​\(q\)=TopKx∈𝒫⁡\(q\)\(−‖f⊕​\(q\)−f⊕​\(x\)‖2\),\\mathcal\{R\}^\{\\mathrm\{PG\}\}\_\{K\}\(q\)=\\operatorname\*\{TopK\}\_\{x\\in\\mathcal\{P\}\(q\)\}\\left\(\-\\left\\\|f\_\{\\oplus\}\(q\)\-f\_\{\\oplus\}\(x\)\\right\\\|\_\{2\}\\right\),\(15\)wheref⊕​\(⋅\)f\_\{\\oplus\}\(\\cdot\)denotes the concatenation of the modality\-specific embeddings\. Thus, candidates must first be non\-dominated across projected modality distances and are then ordered by local proximity in the concatenated independent embedding space\.[Fig\. 4](https://arxiv.org/html/2611.23639#S3.F4)illustrates this refinement for an example query, contrasting the full Pareto front with the resulting fixed\-size retrieval set\.

### 4Experimental Design

This section describes the evaluation dataset, retrieval protocol, methods under comparison, and metrics used to assess retrieval quality\.

#### 4\.1Dataset

We evaluate our method using a galaxy retrieval corpus built from the Last Journey simulation[Heitmann et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib17)\. The corpus contains\|𝒞\|=5,000\|\\mathcal\{C\}\|=5\{,\}000galaxies, and the query set contains\|𝒬\|=1,000\|\\mathcal\{Q\}\|=1\{,\}000held\-out galaxies disjoint from the corpus\. Each query is evaluated against the same corpus, enabling paired comparisons across retrieval methods\.

Every galaxy contains four vector modalities, summarized in[Table 1](https://arxiv.org/html/2611.23639#S3.T1.fig1)\. These modalities differ in dimensionality and physical meaning, making them a suitable testbed for heterogeneous scientific retrieval\. We also use redshift and stellar mass as scalar attributes for raw\-space evaluation, as described later in[Section 4\.4](https://arxiv.org/html/2611.23639#S4.SS4)\.

#### 4\.2Retrieval Protocol

All comparison methods use the same candidate generation protocol\. For each queryqq, we retrieve the top\-kknearest neighbors in each modality using the projected distancedm​\(q,x\)d\_\{m\}\(q,x\), then take the union of these modality\-specific neighborhoods as the reranking pool\. This preserves candidates that are close in at least one modality before applying cross\-modal reranking\.

We usek=50k=50for candidate generation and report the main results at retrieval depthK=25K=25\. The candidate depth is larger than the reporting depth so that reranking can select from a sufficiently broad pool while keeping computation fixed across methods\. The main depthK=25K=25provides a moderate top\-list size for analyzing neighborhood quality and is not tuned separately for any method\. For overlap analyses, we also evaluateK∈\{1,5,10,25\}K\\in\\\{1,5,10,25\\\}to characterize sensitivity to retrieval depth\.

#### 4\.3Comparison Methods

We compare late fusion methods over independently trained embeddings against standardkkNN retrieval in a shared multimodal FM embedding space\.

Independent\-embedding methods\.These methods use the independently trained embeddings described in[Section 3\.2](https://arxiv.org/html/2611.23639#S3.SS2), denoted as IndepEmb in the result table\. We evaluate:\(i\) Pareto fusion, with both neural and PCA projectors, reported as Pareto \(NN\) and Pareto \(PCA\);\(ii\) Scalar fusion, including weighted\-sum, Chebyshev, andpp\-norm scalarization using the neural projector; and\(iii\)kkNN, which performs nearest\-neighbor search over concatenated independent embeddings\.

Shared\-embedding FM baseline\.As a shared\-embedding baseline, we use MOSAIC\-lite, a lightweight multimodal FM trained at the same data scale as our independent encoders\. Retrieval is performed usingkkNN search in its unified embedding space\.

#### 4\.4Evaluation Metrics

Because multimodal galaxy retrieval has no canonical ground\-truth ranking, we evaluate retrieval quality using query\-relative diagnostics that assess retrieved items against a corpus\-level reference\. The four metrics capture different aspects of retrieval quality:*relative proximity*\(𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\) measures overall closeness in the projected embedding space,*cross\-modal balance*\(𝐶𝑀𝐵\\mathit\{CMB\}\) measures how evenly that closeness is distributed across modalities,*Pareto coverage*\(𝑃𝐶𝑜𝑣\\mathit\{PCov\}\) measures whether the retrieved list preserves non\-dominated trade\-offs, and*raw\-data spread*\(R​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}\) measures fidelity in the original raw\-feature space\.

LetℛK​\(q\)⊂𝒞∖\{q\}\\mathcal\{R\}\_\{K\}\(q\)\\subset\\mathcal\{C\}\\setminus\\\{q\\\}be the top\-KKitems returned for queryq∈𝒬q\\in\\mathcal\{Q\}\. For modalitymm, we define the corpus reference distance as

d¯m​\(q\)=1\|𝒞\|−1​∑x∈𝒞∖\{q\}dm​\(q,x\)\.\\bar\{d\}\_\{m\}\(q\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\-1\}\\sum\_\{x\\in\\mathcal\{C\}\\setminus\\\{q\\\}\}d\_\{m\}\(q,x\)\.\(16\)
This reference normalizes distances by the average corpus distance for the same query and modality, making scores comparable across queries and modalities\. All metrics are computed per query and then averaged over𝒬\\mathcal\{Q\}\.

##### 4\.4\.1Relative proximity \(𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\)

Relative proximity measures the overall compactness of the retrieved neighborhood in the projected embedding space\. It asks whether the returned items are, on average, closer to the query than a typical corpus item\. For each modalitymm,

𝑟𝑃𝑟𝑜𝑥m​\(q\)=1K​∑x∈ℛK​\(q\)dm​\(q,x\)d¯m​\(q\),𝑟𝑃𝑟𝑜𝑥⁡\(q\)=1\|ΩK​\(q\)\|​∑m∈ΩK​\(q\)𝑟𝑃𝑟𝑜𝑥m​\(q\),\\mathit\{rProx\}\_\{m\}\(q\)=\\frac\{1\}\{K\}\\sum\_\{x\\in\\mathcal\{R\}\_\{K\}\(q\)\}\\frac\{d\_\{m\}\(q,x\)\}\{\\bar\{d\}\_\{m\}\(q\)\},\\qquad\\mathit\{rProx\}\(q\)=\\frac\{1\}\{\|\\Omega\_\{K\}\(q\)\|\}\\sum\_\{m\\in\\Omega\_\{K\}\(q\)\}\\mathit\{rProx\}\_\{m\}\(q\),\(17\)where

ΩK​\(q\)=⋂x∈ℛK​\(q\)Ω⁡\(q,x\)\\Omega\_\{K\}\(q\)=\\bigcap\_\{x\\in\\mathcal\{R\}\_\{K\}\(q\)\}\\Omega\(q,x\)\(18\)is the set of modalities shared by the query and all retrieved items\. Values below 1 indicate retrievals closer than the corpus average; lower is better\.

##### 4\.4\.2Cross\-modal balance \(𝐶𝑀𝐵\\mathit\{CMB\}\)

Relative proximity \(𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\) summarizes average closeness, but it does not indicate whether that closeness comes from all modalities or only a subset of them\. Cross\-modal balance therefore measures how evenly retrieval quality is distributed across modalities\. We compute the per\-query coefficient of variation of the per\-modality relative proximities,

cv𝑟𝑃𝑟𝑜𝑥⁡\(q\)=stdm∈ΩK​\(q\)⁡\(𝑟𝑃𝑟𝑜𝑥m​\(q\)\)meanm∈ΩK​\(q\)⁡\(𝑟𝑃𝑟𝑜𝑥m​\(q\)\),\\operatorname\{cv\}\_\{\\mathit\{rProx\}\}\(q\)=\\frac\{\\operatorname\{std\}\_\{m\\in\\Omega\_\{K\}\(q\)\}\\\!\\bigl\(\\mathit\{rProx\}\_\{m\}\(q\)\\bigr\)\}\{\\operatorname\{mean\}\_\{m\\in\\Omega\_\{K\}\(q\)\}\\\!\\bigl\(\\mathit\{rProx\}\_\{m\}\(q\)\\bigr\)\},\(19\)and define cross\-modal balance as

𝐶𝑀𝐵=1−1\|𝒬\|​∑q∈𝒬cv𝑟𝑃𝑟𝑜𝑥⁡\(q\)\.\\mathit\{CMB\}=1\\,\-\\,\\frac\{1\}\{\|\\mathcal\{Q\}\|\}\\sum\_\{q\\in\\mathcal\{Q\}\}\\operatorname\{cv\}\_\{\\mathit\{rProx\}\}\(q\)\.\(20\)Thus𝐶𝑀𝐵=1\\mathit\{CMB\}=1when all per\-modality𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}values are identical, and𝐶𝑀𝐵\\mathit\{CMB\}decreases as the retrieved neighborhood becomes more uneven across modalities\.

##### 4\.4\.3Pareto coverage \(𝑃𝐶𝑜𝑣\\mathit\{PCov\}\)

We use Pareto coverage to assess how many candidates in the top\-KKlist remain non\-dominated relative to the other retrieved items\. This metric quantifies how much non\-dominated structure is preserved after retrieval\. Let𝒫K​\(q\)⊆ℛK​\(q\)\\mathcal\{P\}\_\{K\}\(q\)\\subseteq\\mathcal\{R\}\_\{K\}\(q\)be the retrieved items that are not Pareto\-dominated by any other item inℛK​\(q\)\\mathcal\{R\}\_\{K\}\(q\)under the dominance relation≺q\\prec\_\{q\}from Eqs\. \([9](https://arxiv.org/html/2611.23639#S3.E9)\)–\([10](https://arxiv.org/html/2611.23639#S3.E10)\)\. Pareto coverage is

𝑃𝐶𝑜𝑣​@​K​\(q\)=\|𝒫K​\(q\)\|K\.\\mathit\{PCov\}@K\(q\)=\\frac\{\|\\mathcal\{P\}\_\{K\}\(q\)\|\}\{K\}\.\(21\)In our experiments, we truncate each retrieved list atK=10K=10\. Higher values mean that a larger fraction of the top\-KKlist remains non\-dominated\. Unlike the corpus\-level front𝒫⁡\(q\)\\mathcal\{P\}\(q\)in Eq\. \([11](https://arxiv.org/html/2611.23639#S3.E11)\),𝒫K​\(q\)\\mathcal\{P\}\_\{K\}\(q\)is computed only within the retrieved set;𝑃𝐶𝑜𝑣\\mathit\{PCov\}therefore characterizes the ranking rather than the full corpus structure\.

##### 4\.4\.4Raw\-data spread \(R​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}\)

The preceding metrics are computed in the embedding space used for retrieval\. R\-spread provides a complementary check in the original feature space, measuring whether retrieved galaxies remain close to the query in raw physical units\. For each modalitymm, we use

dmraw​\(q,x\)=\{‖q\(m\)−x\(m\)‖2,m​is vector\-valued,\|q\(m\)−x\(m\)\|,m​is scalar\.d^\{\\mathrm\{raw\}\}\_\{m\}\(q,x\)=\\begin\{cases\}\\\|q^\{\(m\)\}\-x^\{\(m\)\}\\\|\_\{2\},&m\\text\{ is vector\-valued\},\\\\\[2\.0pt\] \|q^\{\(m\)\}\-x^\{\(m\)\}\|,&m\\text\{ is scalar\}\.\\end\{cases\}\(22\)Letd¯mraw​\(q\)\\bar\{d\}^\{\\mathrm\{raw\}\}\_\{m\}\(q\)be defined as in Eq\. \([16](https://arxiv.org/html/2611.23639#S4.E16)\) usingdmrawd^\{\\mathrm\{raw\}\}\_\{m\}\. We define

R​\-​𝑠𝑝𝑟𝑒𝑎𝑑m​\(q\)=1K​∑x∈ℛK​\(q\)dmraw​\(q,x\)d¯mraw​\(q\)\.\\mathit\{R\\text\{\-\}spread\}\_\{m\}\(q\)=\\frac\{1\}\{K\}\\sum\_\{x\\in\\mathcal\{R\}\_\{K\}\(q\)\}\\frac\{d^\{\\mathrm\{raw\}\}\_\{m\}\(q,x\)\}\{\\bar\{d\}^\{\\mathrm\{raw\}\}\_\{m\}\(q\)\}\.\(23\)Lower values indicate closer retrievals in the original feature space\. We reportR​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}for the four vector modalities \(described in[Table 1](https://arxiv.org/html/2611.23639#S3.T1.fig1)\) and for two scalar modalities, redshift and stellar mass, which are not used to generate independent embeddings\.

### 5Results

We organize the results around how each method shapes retrieved neighborhoods, preserves modality\-specific trade\-offs, and behaves in raw physical space\.

#### 5\.1Embedding\-space proximity favors shared geometry

Table 2:Retrieval performance under embedding\-space metrics\.Note\.Lower𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}is better: values below 1 mean that retrieved items are closer to the query object than a randomly drawn corpus item, measured in the method’s own embedding space\. Higher𝐶𝑀𝐵\\mathit\{CMB\}indicates more balanced per\-modality proximity, but is not a universal optimality criterion\. Higher𝑃𝐶𝑜𝑣\\mathit\{PCov\}indicates greater coverage of non\-dominated items in the top\-10 results\.*Pareto Gold*is reported atK=10K\{=\}10to match the𝑃𝐶𝑜𝑣\\mathit\{PCov\}truncation\. Bold marks the best value per column\.

Under an embedding\-space proximity criterion, the shared FM geometry provides the strongest retrieval neighborhoods\. As summarized in[Table 2](https://arxiv.org/html/2611.23639#S5.T2), the FM baseline achieves the lowest𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\(0\.153\), corresponding to a 38% relative reduction compared with the strongest independent\-embedding baseline,kkNN \[IndepEmb\] \(0\.247\), and larger reductions compared with Pareto and scalar fusion methods\. This result is expected: the FM embedding space maps the four modalities into a single 384\-dimensional representation, so EuclideankkNN directly optimizes neighborhoods in the same geometry learned by the model\.

Independent\-embedding methods are less competitive under𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}because they do not optimize a single shared distance function\. Late fusion instead ranks candidates using modality\-specific distance signals, which can surface items that are strong along particular modality axes but not uniformly close under the averaged proximity measured by𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\.

Table 3:Comparison of*Pareto Gold*and the FM baseline\.Note\.Coverage probability reports whether the two methods share at least one retrieved item at depthKK; recall measures the item\-level fraction of*Pareto Gold*results recovered by the FM\. Intervals are 95% confidence intervals over 1,000 queries\.

#### 5\.2Multi\-objective retrieval preserves modality\-specific structure

We next examine whether the retrieved neighborhoods retain modality\-specific trade\-offs\. The𝐶𝑀𝐵\\mathit\{CMB\}results exhibit a clear contrast between shared and independent representations\. The FM baseline obtains the highest𝐶𝑀𝐵\\mathit\{CMB\}\(0\.810\) in[Table 2](https://arxiv.org/html/2611.23639#S5.T2), whereas all independent\-embedding methods are substantially lower: Pareto \(NN\) and Pareto \(PCA\) achieve 0\.109 and 0\.115, scalar methods range from 0\.068 to 0\.109, andkkNN \[IndepEmb\] reaches 0\.291\. Since lower𝐶𝑀𝐵\\mathit\{CMB\}reflects greater variation among per\-modality proximity values, these results indicate that multi\-objective retrieval preserves heterogeneous modality profiles, whereas the FM embedding makes modality\-specific distances more uniform\.

Pareto coverage \(𝑃𝐶𝑜𝑣\\mathit\{PCov\}\) further reveals how much trade\-off structure remains in the retrieved set\. Even without the*Pareto Gold*refinement reported later \([Section 5\.3](https://arxiv.org/html/2611.23639#S5.SS3)\), the direct Pareto rerankers retain high coverage: Pareto \(PCA\) reaches 0\.448 and Pareto \(NN\) reaches 0\.440, above both scalar baselines \(0\.364–0\.388\) and the FM baseline \(0\.382\)\. However, high𝑃𝐶𝑜𝑣\\mathit\{PCov\}is not limited to explicit Pareto selection:kkNN \[IndepEmb\] attains𝑃𝐶𝑜𝑣=0\.417\\mathit\{PCov\}=0\.417, also exceeding the FM baseline\. This suggests that𝑃𝐶𝑜𝑣\\mathit\{PCov\}is driven not only by Pareto selection, but also by whether the retrieval space preserves modality\-specific geometry\. When that geometry is preserved, non\-dominated candidates remain visible even under standard nearest\-neighbor retrieval\. Scalar fusion and shared\-space retrieval, in contrast, more often compress cross\-modal trade\-offs into a single ranking signal\.

Together, the𝐶𝑀𝐵\\mathit\{CMB\}and𝑃𝐶𝑜𝑣\\mathit\{PCov\}results show that the main benefit of multi\-objective retrieval is structural, not simply proximity\-based\. ParetoFuse better preserves the non\-dominated candidates that arise from uneven, modality\-specific relevance\. This makes the retrieved set less uniform, but more faithful to the cross\-modal trade\-offs that scientific retrieval must expose\.

#### 5\.3Pareto Gold is compatible with the FM yet more selective

*Pareto Gold*substantially refines plain Pareto retrieval \([Table 2](https://arxiv.org/html/2611.23639#S5.T2)\):𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}falls from 0\.346 for Pareto \(NN\) to 0\.249, while𝑃𝐶𝑜𝑣\\mathit\{PCov\}rises from 0\.440 to 0\.480, the highest value achieved\. ThekkNN refinement step concentrates the Pareto front on candidates that remain close in the embedding space, improving average proximity while retaining non\-dominated structure\. The main trade\-off appears in𝐶𝑀𝐵\\mathit\{CMB\}\(0\.109→0\.0440\.109\\rightarrow 0\.044\), indicating that per\-modality proximity becomes less uniform\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_raw_spread.png)Figure 5:Raw\-data spread \(R​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}\) for*Pareto Gold*and the FM baseline atK=1K\{=\}1andK=5K\{=\}5\.R​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}measures the average raw\-space distance of the retrieved items to a given query; lower values indicate closer retrieved items in the original feature space\. Dotted lines indicate per\-method averages across the six modalities\.We then examine whether this stricter modality\-aware selection remains compatible with the similarity structure learned by the FM baseline\. At matched retrieval depthKK,[Table 3](https://arxiv.org/html/2611.23639#S5.T3)shows that overlap is common but selective\. Coverage rises from 0\.556 atK=1K\{=\}1to 0\.942 atK=25K\{=\}25, so the two methods usually share at least one object as the retrieval set grows\. However,*Pareto Gold*recall falls from 0\.556 to 0\.110, meaning that most of its retrieved items are not recovered by the FM at larger depths\. Thus,*Pareto Gold*remains aligned with the broad FM neighborhood while prioritizing different candidates within it\.

![Refer to caption](https://arxiv.org/html/2611.23639v1/fig_pdf_stmass.png)Figure 6:Log stellar mass distributions for six example queries\. Each panel shows the corpus density, the Pareto\-front density, and the query value\. The Pareto\-front density is more concentrated near the query than the corpus background, indicating that non\-dominated retrieval over observed modalities can align with a held\-out scalar modality\.
#### 5\.4Pareto Gold improves physical fidelity

Raw\-data spread \(R​\-​𝑠𝑝𝑟𝑒𝑎𝑑\\mathit\{R\\text\{\-\}spread\}\) reveals a different ranking than embedding\-space proximity\. AtK=1K\{=\}1,*Pareto Gold*achieves a lower average spread across the six modalities than the FM baseline \(0\.128 versus 0\.139;[Fig\. 5](https://arxiv.org/html/2611.23639#S5.F5)\), with the largest improvement on𝝍\\boldsymbol\{\\psi\}\(0\.285 to 0\.169, a 39% relative reduction\)\. This result is notable because the underlying Pareto front has roughly twice the embedding\-space𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}of the FM baseline \(0\.346 versus 0\.153;[Table 2](https://arxiv.org/html/2611.23639#S5.T2)\)\. Thus, the method that is closest under the learned embedding metric is not necessarily the method that retrieves the closest neighbor in the original feature space\. AtK=5K\{=\}5, the aggregate average spread slightly favors the FM baseline \(0\.213 versus 0\.249\), but*Pareto Gold*remains closer on𝐟sph\\mathbf\{f\}\_\{\\rm sph\},𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}, and𝐟λ\\mathbf\{f\}\_\{\\lambda\}, showing that its advantage persists for several modalities\. These raw\-space gains are noteworthy because Pareto retrieval uses only independently trained encoders, without cross\-modal supervision or shared parameters\.

The query\-level examples provide a qualitative illustration of the same effect\. In[Fig\. 6](https://arxiv.org/html/2611.23639#S5.F6), the stellar\-mass densities of Pareto\-front items are more concentrated around the query value than the corpus background\. These examples suggest that non\-domination across observed modalities can serve as a useful proxy for similarity along physical attributes not directly used in retrieval\.

### 6Discussion

We frame retrieval over heterogeneous modalities as a multi\-objective optimization problem\. Rather than projecting modality\-specific embeddings into a single learned geometry, ParetoFuse preserves each modality as a separate objective and uses Pareto dominance to retain candidates that remain competitive across all of them\. The empirical question is whether this late fusion view can recover scientifically meaningful neighbors that are not surfaced by a unified FM embedding\. Our results show that the FM baseline provides a strong global representation, whereas Pareto\-based late fusion exposes trade\-offs among the underlying physical measurements\.

The main implication is that embedding\-space proximity and raw\-space fidelity are not equivalent\. FM embeddings produce compact neighborhoods in its own learned space, as reflected by its lower𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}\. However, this intrinsic compactness does not always translate into proximity in the original data spaces\. At the head of the ranking,*Pareto Gold*retrieves neighbors with lower raw\-space spread than the FM baseline on average\. This indicates that a unified embedding can be highly effective as a semantic compression of the data while still obscuring modality\-specific variation that remains scientifically meaningful\.

This discrepancy reflects a fundamental difference in how the two retrieval strategies define similarity\. A unified FM embedding induces one composite distance, so local neighborhoods are determined by the geometry learned across all modalities jointly\. This is valuable when the goal is a reusable cross\-modal representation or when downstream tasks benefit from smooth alignment across observed and unobserved modalities\. Pareto\-based retrieval imposes a different criterion: a candidate need not be nearest under a single aggregate distance, but must remain competitive across the preserved modality axes\. This makes the ranking less compact in embedding space, but more explicit about which modalities support a neighbor relationship and where the trade\-offs occur\. In scientific settings, that distinction is important because similarity is often not a scalar property; two galaxies may be close in photometry, star\-formation history, or emitted spectrum for different physical reasons\.

The overlap analyses in[Section 5\.3](https://arxiv.org/html/2611.23639#S5.SS3)support this interpretation\. Pareto\-based retrieval and FM\-basedkkNN do not select unrelated regions of the corpus; instead, they often identify a similar candidate neighborhood and then impose different local orderings within it\. The shared candidates tend to be high\-fidelity cases, while the method\-specific candidates reveal where the two geometries diverge\. This suggests a complementary division of labor: the FM embedding is well suited for broad candidate localization, whereas Pareto\-based reranking can refine that neighborhood using modality\-aware criteria\. In this sense, the proposed method should not be understood as replacing FM retrieval, but as making explicit the modality\-wise structure that a single learned distance may suppress\.

Our results should not be read as a general critique of multimodal FMs\. They instead identify a retrieval regime in which Pareto\-based late fusion is most useful: modality\-specific distance must remain inspectable and labeled data are too limited for task\-specific reranking\. In this setting, Pareto dominance provides a principled way to refine retrieval without reducing all modalities to a single shared representation\. For discovery\-oriented retrieval, where the meaning of scientific similarity depends on the measurements being compared, preserving this trade\-off structure is a methodological advantage rather than a secondary diagnostic\.

### 7Limitations

This study has three main limitations\. First, the evaluation considers a single FM baseline, MOSAIC\-lite, and defines Pareto dominance using one\-dimensional neural and PCA projections\. The observed gap between embedding\-space proximity and raw\-space fidelity may therefore depend on the encoder, training objective, and projection quality\. For PCA, the low explained variance ratio—the fraction of variance retained by the first principal component—observed for modalities such as𝝍\\boldsymbol\{\\psi\}suggests that the projection may discard meaningful structure and weaken the dominance relation\. Higher\-dimensional or more expressive projections could preserve more information, but would increase the cost of Pareto reranking\.

Second, dominance\-based retrieval becomes more expensive as the corpus size and number of objectives grow\. The four\-modality setting studied here is tractable, but larger deployments may require approximate dominance pruning or other scalable retrieval mechanisms\. Although our methods can be parallelized across corpus partitions and combined through a final ranking stage, we do not yet evaluate such an implementation with a larger number of modalities\.

Third, our evaluation metrics partially reflect the retrieval objective in the dataset\. Embedding\-space metrics such as𝑟𝑃𝑟𝑜𝑥\\mathit\{rProx\}are useful for analyzing geometry, but they may not fully represent scientific relevance in the original measurements\. This mismatch helps explain why our methods appear weaker under aggregated embedding\-space scores while retrieving neighbors with stronger physical fidelity\. Future work should develop evaluation criteria that better align embedding\-space structure with raw\-space agreement\.

### 8Conclusion

We introduced ParetoFuse, a multi\-objective late fusion framework for scientific modalities, and evaluated it on a galaxy retrieval task against an FM embedding baseline\. Rather than collapsing all modalities into a single shared representation, the framework preserves modality\-specific distance signals and combines them at retrieval time through Pareto dominance\. The*Pareto Gold*refinement further converts the resulting non\-dominated set into a fixed\-size ranked shortlist while retaining the multi\-objective structure of the candidate pool\.

Our results indicate that FM\-basedkkNN retrieval and Pareto\-based retrieval over independent embeddings capture different notions of similarity\. The FM provides stronger proximity in embedding space but produces rankings that are more homogeneous across modalities\. In contrast, ParetoFuse retains modality\-specific trade\-offs and identifies neighbors that are closer to the query in the raw data space, particularly for informative modalities whose structure may be compressed in a shared embedding space\. The two approaches therefore do not retrieve from entirely disjoint regions of the corpus; rather, they operate within similar neighborhoods while imposing different orderings on the candidates\.

These findings suggest that late fusion of independent embeddings offers a practical complement to FM\-based retrieval for scientific datasets with heterogeneous modality coverage\. ParetoFuse does not require retraining a unified model and provides a lightweight way to surface astrophysically coherent candidates whose relevance may be diluted in a single\-objective ranking\.

##### Acknowledgements

This material is based upon work supported by the U\.S\. Department of Energy \(DOE\), Office of Science, Office of Advanced Scientific Computing Research, including the TIDES project and Argonne Leadership Computing Facility at Argonne National Laboratory, under Contract No\. DE\-AC02\-06CH11357\.

### Declarations

The authors declare no conflict of interest\.

### 9MOSAIC and MOSAIC\-lite foundation models for Cosmology

MOSAIC[Xia et al\. \(2025\)](https://arxiv.org/html/2611.23639#bib.bib1), short for Multimodal Observation\-Simulation Integration for Cosmology, is a multimodal foundation model \(FM\) designed to connect simulation\-derived galaxy properties with synthetic observational quantities\. It uses an encoder\-only Transformer, inspired by BERT[Devlin et al\. \(2019\)](https://arxiv.org/html/2611.23639#bib.bib16), to operate on heterogeneous galaxy modalities, including scalar quantities such as redshift, halo mass, and stellar mass, as well as vector modalities such as𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\},𝐟sph\\mathbf\{f\}\_\{\\rm sph\},𝝍\\boldsymbol\{\\psi\}, and𝐟λ\\mathbf\{f\}\_\{\\lambda\}\. MOSAIC was trained on a subset of galaxies derived from the Last Journey simulation[Heitmann et al\. \(2021\)](https://arxiv.org/html/2611.23639#bib.bib17), where galaxy properties and synthetic observables are assigned to dark\-matter structures through galaxy–halo connection and stellar\-population modeling\.

Figure 7:Schematic of the MOSAIC architecture\. Each modality is masked and projected into modality\-specific embeddings, concatenated into a shared token sequence, combined with positional embeddings, and processed by an encoder\-only Transformer\. The output sequence is split into modality\-specific segments and decoded with dedicated regression heads\. MOSAIC\-lite follows the same shared\-embedding principle and is used in this work as a lightweight multimodal FM baseline for retrieval\.MOSAIC represents each modality as a sequence of continuous\-valued tokens rather than using discrete tokenization\. Scalar quantities are grouped into a “scalar” vector, while vector modalities naturally define token sequences over their sampled dimensions\. As shown in[Fig\. 7](https://arxiv.org/html/2611.23639#S9.F7), each token is projected into a shared hidden dimension using a modality\-specific linear embedding layer\. The embedded modality sequences are concatenated along the sequence dimension, augmented with positional embeddings, and passed through a shared Transformer encoder\. The output sequence is then split back into modality\-specific segments and decoded using separate regression heads\. This design allows the model to learn a shared latent representation while preserving modality\-specific structure\.

The model is trained using a masked\-regression objective\. During training, each modality may be partially or fully masked, simulating missing observations or unmeasured galaxy properties\. The model is then asked to reconstruct both masked and unmasked values across modalities\. This training strategy supports both in\-modality reconstruction, such as recovering missing segments of𝐟λ\\mathbf\{f\}\_\{\\lambda\}or𝝍\\boldsymbol\{\\psi\}, and cross\-modality prediction, such as predicting redshift, stellar mass, or spectra from photometric inputs\. In the original MOSAIC study, combining𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}and𝐟sph\\mathbf\{f\}\_\{\\rm sph\}improved redshift estimation relative to𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}alone, while combining late\-time𝝍\\boldsymbol\{\\psi\}with𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}improved stellar\-mass inference relative to early\-time𝝍\\boldsymbol\{\\psi\}with𝐟lsst\\mathbf\{f\}\_\{\\rm lsst\}\.

In this work, we use MOSAIC\-lite as an FM baseline\. Because the full MOSAIC model was trained on a substantially larger dataset, using it directly would conflate retrieval performance with differences in data scale and model capacity\. We therefore train MOSAIC\-lite on the same 20,000 galaxy samples as our evaluation, using the same multimodal inputs described above together with scalar modalities grouped into a scalar vector\. MOSAIC\-lite follows the same general principle as MOSAIC: it learns a shared embedding space through joint multimodal training\. For the experiments in this paper, MOSAIC\-lite uses an encoder\-only Transformer with hidden size 384, 8 hidden layers, 12 attention heads, intermediate size 3072, and dropout probability 0\.1 for both hidden states and attention weights\. Retrieval with MOSAIC\-lite is performed by nearest\-neighbor search in its shared 384\-dimensional embedding space\.

### Tool Use

ChatGPT was used to improve the grammar and clarity of the manuscript\. All suggested revisions were reviewed by the authors, who take full responsibility for the final content\. The research ideas, methods, and results are the authors’ own\.

## References

- M\. Arevalillo\-Herráez, F\. J\. Ferri, and S\. Moreno\-PicotA hybrid multi\-objective optimization algorithm for content based image retrieval\.Applied Soft Computing13\(11\),pp\. 4358–4369\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Atreyet al\.\(2010\)P\. K\. Atrey, M\. A\. Hossain, A\. El Saddik, and M\. S\. KankanhalliMultimodal fusion for multimedia analysis: a survey\.Multimedia systems16\(6\),pp\. 345–379\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Bommasaniet al\.\(2021\)R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill,et al\.On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Borovinšeket al\.\(2017\)M\. Borovinšek, B\. Y\. Ekren, A\. Burinskienė, and T\. LerherMulti\-objective optimisation model of shuttle\-based storage and retrieval system\.Transport32\(2\),pp\. 120–137\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Boyd \(2001\)J\. P\. BoydChebyshev and Fourier spectral methods\.Second edition,Dover Books on Mathematics,Dover Publications,Mineola, NY\.External Links:ISBN 0486411834 9780486411835Cited by:[§3\.4\.2](https://arxiv.org/html/2611.23639#S3.SS4.SSS2.p3.2)\.
- Censor \(1977\)Y\. CensorPareto optimality in multiobjective problems\.Applied Mathematics and Optimization4\(1\),pp\. 41–59\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p3.1),[§3\.4\.1](https://arxiv.org/html/2611.23639#S3.SS4.SSS1.p1.1)\.
- Daiet al\.\(2011\)N\. Dai, M\. Shokouhi, and B\. D\. DavisonMulti\-objective optimization in learning to rank\.InProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval,pp\. 1241–1242\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Deb \(2011\)K\. DebMulti\-objective optimisation using evolutionary algorithms: an introduction\.InMulti\-objective Evolutionary Optimisation for Product Design and Manufacturing,L\. Wang, A\. H\. C\. Ng, and K\. Deb \(Eds\.\),pp\. 3–34\.External Links:ISBN 978\-0\-85729\-652\-8Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§9](https://arxiv.org/html/2611.23639#S9.p1.1)\.
- Diwekar \(2003\)U\. M\. DiwekarMulti\-objective optimization\.InIntroduction to Applied Optimization,pp\. 209–253\.External Links:ISBN 978\-1\-4757\-3745\-5Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Donget al\.\(2018\)X\. Dong, Y\. Yan, M\. Tan, Y\. Yang, and I\. W\. TsangLate fusion via subspace search with consistency preservation\.IEEE Transactions on Image Processing28\(1\),pp\. 518–528\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Escalanteet al\.\(2008\)H\. J\. Escalante, C\. A\. Hérnadez, L\. E\. Sucar, and M\. MontesLate fusion of heterogeneous methods for multimedia image retrieval\.InProceedings of the 1st ACM international conference on Multimedia information retrieval,pp\. 172–179\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Fonseca and Fleming \(1995\)C\. M\. Fonseca and P\. J\. FlemingAn overview of evolutionary algorithms in multiobjective optimization\.Evolutionary computation3\(1\),pp\. 1–16\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p3.1),[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1),[§3\.4\.1](https://arxiv.org/html/2611.23639#S3.SS4.SSS1.p1.1)\.
- Gentile and Littlestone \(1999\)C\. Gentile and N\. LittlestoneThe robustness of the p\-norm algorithms\.InProceedings of the twelfth annual conference on Computational learning theory,pp\. 1–11\.Cited by:[§3\.4\.2](https://arxiv.org/html/2611.23639#S3.SS4.SSS2.p4.2)\.
- Giagkiozis and Fleming \(2015\)I\. Giagkiozis and P\. J\. FlemingMethods for multi\-objective optimization: an analysis\.Information Sciences293,pp\. 338–350\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Gialampoukidiset al\.\(2016\)I\. Gialampoukidis, A\. Moumtzidou, D\. Liparas, S\. Vrochidis, and I\. KompatsiarisA hybrid graph\-based and non\-linear late fusion approach for multimedia retrieval\.In2016 14th International workshop on content\-based multimedia indexing \(CBMI\),pp\. 1–6\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Girdharet al\.\(2023\)R\. Girdhar, A\. El\-Nouby, Z\. Liu, M\. Singh, K\. V\. Alwala, A\. Joulin, and I\. MisraImagebind: one embedding space to bind them all\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15180–15190\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Gunantara \(2018\)N\. GunantaraA review of multi\-objective optimization: methods and its applications\.Cogent Engineering5\(1\),pp\. 1502242\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Guoet al\.\(2019\)W\. Guo, J\. Wang, and S\. WangDeep multimodal representation learning: a survey\.Ieee Access7,pp\. 63373–63394\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Heitmannet al\.\(2021\)K\. Heitmann, N\. Frontiere, E\. Rangel, P\. Larsen, A\. Pope, I\. Sultan, T\. Uram, S\. Habib, H\. Finkel, D\. Korytov,et al\.The last journey\. i\. an extreme\-scale simulation on the mira supercomputer\.The Astrophysical Journal Supplement Series252\(2\),pp\. 19\.Cited by:[§4\.1](https://arxiv.org/html/2611.23639#S4.SS1.p1.1),[§9](https://arxiv.org/html/2611.23639#S9.p1.1)\.
- Hinton and Salakhutdinov \(2006\)G\. E\. Hinton and R\. R\. SalakhutdinovReducing the dimensionality of data with neural networks\.science313\(5786\),pp\. 504–507\.Cited by:[§3\.2](https://arxiv.org/html/2611.23639#S3.SS2.p1.1)\.
- Horiet al\.\(2017\)C\. Hori, T\. Hori, T\. Lee, Z\. Zhang, B\. Harsham, J\. R\. Hershey, T\. K\. Marks, and K\. SumiAttention\-based multimodal fusion for video description\.InProceedings of the IEEE international conference on computer vision,pp\. 4193–4202\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Jainet al\.\(2005\)A\. Jain, K\. Nandakumar, and A\. RossScore normalization in multimodal biometric systems\.Pattern recognition38\(12\),pp\. 2270–2285\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Jhaet al\.\(2026\)R\. Jha, C\. Zhang, V\. Shmatikov, and J\. MorrisHarnessing the universal geometry of embeddings\.Advances in Neural Information Processing Systems38,pp\. 45963–45987\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Jiaet al\.\(2021\)C\. Jia, Y\. Yang, Y\. Xia, Y\. Chen, Z\. Parekh, H\. Pham, Q\. Le, Y\. Sung, Z\. Li, and T\. DuerigScaling up visual and vision\-language representation learning with noisy text supervision\.InInternational conference on machine learning,pp\. 4904–4916\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Jinet al\.\(2025\)Q\. Jin, E\. Ge, Y\. Xie, H\. Luo, J\. Song, Z\. Bi, C\. X\. Liang, J\. Guan, J\. Yeong, X\. Song,et al\.Multimodal representation learning and fusion\.arXiv preprint arXiv:2506\.20494\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Lahatet al\.\(2015\)D\. Lahat, T\. Adali, and C\. JuttenMultimodal data fusion: an overview of methods, challenges, and prospects\.Proceedings of the IEEE103\(9\),pp\. 1449–1477\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Lerheret al\.\(2013\)T\. Lerher, M\. Šraml, M\. Borovinšek, and I\. PotrčMULTI\-objective optimization of automated storage and retrieval systems\.\.Annals of the Faculty of Engineering Hunedoara\-International Journal of Engineering11\(1\)\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Leung and Bovy \(2024\)H\. W\. Leung and J\. BovyTowards an astronomical foundation model for stars with a transformer\-based model\.Monthly Notices of the Royal Astronomical Society527\(1\),pp\. 1494–1520\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1)\.
- Liet al\.\(2024\)C\. Li, Z\. Gan, Z\. Yang, J\. Yang, L\. Li, L\. Wang, and J\. GaoMultimodal foundation models: from specialists to general\-purpose assistants\.Foundations and Trends in Computer Graphics and Vision16\(1\-2\),pp\. 1–214\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Liet al\.\(2022\)J\. Li, D\. Li, C\. Xiong, and S\. HoiBLIP: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InInternational Conference on Machine Learning,pp\. 12888–12900\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Li and Tang \(2024\)S\. Li and H\. TangMultimodal alignment and fusion: a survey\.arXiv preprint arXiv:2411\.17040\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Linet al\.\(2025\)S\. Lin, C\. Lee, M\. Shoeybi, J\. Lin, B\. Catanzaro, and W\. PingMm\-embed: universal multimodal retrieval with multimodal llms\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 44215–44234\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1)\.
- Luoet al\.\(2025\)R\. Luo, X\. Xia, L\. Wang, L\. Chen, R\. Shan, J\. Luo, M\. Yang, and T\. ChuaNext\-omni: towards any\-to\-any omnimodal foundation models with discrete flow matching\.arXiv preprint arXiv:2510\.13721\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1)\.
- Marler and Arora \(2004\)R\. T\. Marler and J\. S\. AroraSurvey of multi\-objective optimization methods for engineering\.Structural and multidisciplinary optimization26\(6\),pp\. 369–395\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Marler and Arora \(2010\)R\. T\. Marler and J\. S\. AroraThe weighted sum method for multi\-objective optimization: new insights\.Structural and multidisciplinary optimization41\(6\),pp\. 853–862\.Cited by:[§3\.4\.2](https://arxiv.org/html/2611.23639#S3.SS4.SSS2.p2.2)\.
- Miettinen \(1999\)K\. MiettinenNonlinear multiobjective optimization\.Vol\.12,Springer Science & Business Media,New York, NY\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p3.1),[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1),[§3\.4\.1](https://arxiv.org/html/2611.23639#S3.SS4.SSS1.p1.1)\.
- Nandakumaret al\.\(2008\)K\. Nandakumar, Y\. Chen, S\. C\. Dass, and A\. JainLikelihood ratio\-based biometric score fusion\.IEEE transactions on pattern analysis and machine intelligence30\(2\),pp\. 342–347\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Ohet al\.\(2025\)S\. Y\. Oh, A\. Khan, I\. Foster, and K\. ChardSMURF: federated multimodal retrieval for scientific data via embedding alignment\.In2025 IEEE International Conference on eScience \(eScience\),pp\. 452–459\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Parkeret al\.\(2024\)L\. Parker, F\. Lanusse, S\. Golkar, L\. Sarra, M\. Cranmer, A\. Bietti, M\. Eickenberg, G\. Krawezik, M\. McCabe, R\. Morel,et al\.AstroCLIP: a cross\-modal foundation model for galaxies\.Monthly Notices of the Royal Astronomical Society531\(4\),pp\. 4990–5011\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1)\.
- Parkeret al\.\(2026\)L\. Parker, F\. Lanusse, J\. Shen, O\. Liu, T\. Hehir, L\. Sarra, L\. Meyer, M\. Bowles, S\. Wagner\-Carena, H\. Qu,et al\.AION\-1: omnimodal foundation model for astronomical sciences\.Advances in Neural Information Processing Systems38,pp\. 95386–95428\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1)\.
- Poklukaret al\.\(2022\)P\. Poklukar, M\. Vasco, H\. Yin, F\. S\. Melo, A\. Paiva, and D\. KragicGeometric multimodal contrastive representation learning\.InInternational Conference on Machine Learning,pp\. 17782–17800\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Quet al\.\(2021\)Q\. Qu, Z\. Ma, A\. Clausen, and B\. N\. JørgensenA comprehensive review of machine learning in multi\-objective optimization\.In2021 IEEE 4th International Conference on Big Data and Artificial Intelligence \(BDAI\),pp\. 7–14\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1),[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Rodriguez and Carver \(2020\)D\. V\. Rodriguez and D\. L\. CarverMulti\-objective information retrieval\-based nsga\-ii optimization for requirements traceability recovery\.In2020 IEEE International Conference on Electro Information Technology \(EIT\),pp\. 271–280\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Rumelhartet al\.\(1986\)D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. WilliamsLearning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[§3\.2](https://arxiv.org/html/2611.23639#S3.SS2.p1.1)\.
- Shankset al\.\(2015\)T\. Shanks, N\. Metcalfe, B\. Chehade, J\. Findlay, M\. Irwin, E\. Gonzalez\-Solares, J\. Lewis, A\. K\. Yoldas, R\. Mann, M\. Read,et al\.The vlt survey telescope atlas\.Monthly Notices of the Royal Astronomical Society451\(4\),pp\. 4238–4252\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1)\.
- Shoukatet al\.\(2022\)M\. Shoukat, K\. Ahmad, N\. Said, N\. Ahmad, M\. Hassanuzaman, and K\. AhmadA late fusion framework with multiple optimization methods for media interestingness\.arXiv preprint arXiv:2207\.04762\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Singhet al\.\(2023\)R\. Singh, S\. Sledzieski, B\. Bryson, L\. Cowen, and B\. BergerContrastive learning in protein language space predicts interactions between drugs and protein targets\.Proceedings of the National Academy of Sciences120\(24\),pp\. e2220778120\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p1.1)\.
- Snoeket al\.\(2005\)C\. G\. Snoek, M\. Worring, and A\. W\. SmeuldersEarly versus late fusion in semantic video analysis\.InProceedings of the 13th annual ACM international conference on Multimedia,pp\. 399–402\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Terradeset al\.\(2008\)O\. R\. Terrades, E\. Valveny, and S\. TabboneOptimal classifier fusion in a non\-bayesian probabilistic framework\.IEEE transactions on pattern analysis and machine intelligence31\(9\),pp\. 1630–1644\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Tyagi and Sharma \(2025\)S\. Tyagi and P\. SharmaOmniLearn: a framework for distributed deep learning over heterogeneous clusters\.IEEE Transactions on Parallel and Distributed Systems\.Cited by:[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1)\.
- Tyson \(2002\)J\. A\. TysonLarge synoptic survey telescope: overview\.Survey and Other Telescope Technologies and Discoveries4836,pp\. 10–20\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1)\.
- van Doornet al\.\(2016\)J\. van Doorn, D\. Odijk, D\. M\. Roijers, and M\. de RijkeBalancing relevance criteria through multi\-objective optimization\.InProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval,pp\. 769–772\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p3.1)\.
- Wanget al\.\(2016\)K\. Wang, Q\. Yin, W\. Wang, S\. Wu, and L\. WangA comprehensive survey on cross\-modal retrieval\.arXiv preprint arXiv:1607\.06215\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1)\.
- Wanget al\.\(2025\)T\. Wang, F\. Li, L\. Zhu, J\. Li, Z\. Zhang, and H\. T\. ShenCross\-modal retrieval: a systematic review of methods and future directions\.Proceedings of the IEEE112\(11\),pp\. 1716–1754\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p1.1)\.
- Wrightet al\.\(2009\)J\. Wright, A\. Ganesh, S\. Rao, Y\. Peng, and Y\. MaRobust principal component analysis: exact recovery of corrupted low\-rank matrices via convex optimization\.Advances in neural information processing systems22\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p2.1)\.
- Wuet al\.\(2024\)S\. Wu, Y\. Li, K\. Zhu, G\. Zhang, Y\. Liang, K\. Ma, C\. Xiao, H\. Zhang, B\. Yang, W\. Chen,et al\.Scimmir: benchmarking scientific multi\-modal information retrieval\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12560–12574\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Xiaet al\.\(2025\)B\. Xia, N\. Ramachandra, A\. I\. Wells, S\. Habib, and J\. WiseMulti\-modal foundation model for cosmological simulation data\.arXiv preprint arXiv:2510\.07684\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p4.1),[§2\.1](https://arxiv.org/html/2611.23639#S2.SS1.p2.1),[§9](https://arxiv.org/html/2611.23639#S9.p1.1)\.
- Xuet al\.\(2024\)M\. Xu, W\. Yin, D\. Cai, R\. Yi, D\. Xu, Q\. Wang, B\. Wu, Y\. Zhao, C\. Yang, S\. Wang,et al\.A survey of resource\-efficient llm and multimodal foundation models\.arXiv preprint arXiv:2401\.08092\.Cited by:[§1](https://arxiv.org/html/2611.23639#S1.p2.1)\.
- Yeet al\.\(2012\)G\. Ye, D\. Liu, I\. Jhuo, and S\. ChangRobust late fusion with rank minimization\.In2012 IEEE conference on computer vision and pattern recognition,pp\. 3021–3028\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.
- Zhaoet al\.\(2024\)F\. Zhao, C\. Zhang, and B\. GengDeep multimodal data fusion\.ACM computing surveys56\(9\),pp\. 1–36\.Cited by:[§2\.2](https://arxiv.org/html/2611.23639#S2.SS2.p1.1)\.

相似文章

通用多模态基础模型

arXiv cs.LG

本文介绍了一种通用多模态基础模型,能够处理任意模态组合与预测任务,通过在具有多样化因果结构的大规模合成数据集上训练,实现了具有竞争力的性能。

面向模态异质性下的鲁棒联邦多模态图学习

arXiv cs.LG

本文提出FedMPO,一种鲁棒的联邦多模态图学习方法,通过拓扑感知的跨模态生成、缺失感知的专家路由和可靠性感知的聚合来解决模态异质性和缺失模态问题,在多个数据集上实现了性能提升。

从碎片化到融合:全模态嵌入综述与路线图

Hugging Face Daily Papers

一篇综述与立场论文,认为不同模态的嵌入研究正通过对比学习和 Transformer 架构走向融合,并提出一个涵盖覆盖性、可比性和组合性的框架,将变换、对齐和联合嵌入视为通向全模态统一的路径。