Out-Of-The-Loop Multi-Fidelity Bayesian Optimization

# Out-Of-The-Loop Multi-Fidelity Bayesian Optimization
Source: [https://arxiv.org/html/2608.04113](https://arxiv.org/html/2608.04113)
Gustavo Sutter1,2,†, Hao Wang3, Luis Ricardez\-Sandoval3,4, Pascal Poupart1,2, Agustinus Kristiadi2,5

###### Abstract

Black\-box optimization is a ubiquitous problem in science and engineering, often dealing with expensive objective functions with cheaper lower\-fidelity proxies available\. Multi\-fidelity Bayesian optimization \(MF\-BO\) is a principled approach to this problem, leveraging correlations across different fidelities when querying the objective\. However, for many important MF\-BO tasks, the true highest\-fidelity function is prohibitively expensive to be part of the optimization loop\. Nevertheless, practitioners often have gold standard data \(observations of the highest\-fidelity function\) obtained from previous experiments that might provide information for the current task\. For instance, in molecular optimization, chemists often pick the top\-kkcandidate molecules using various computer simulations, and later reveal their*true*objective function values\. In this work, we demonstrate the suboptimality of standard MF\-BO algorithms in the real\-world scenarios above, even under ideal assumptions\. Next, we mitigate this problem by incorporating historical high\-fidelity data accompanied by task descriptors—which can be explicitly given or extracted from unstructured metadata\. We demonstrate the effectiveness of our methods on synthetic functions, as well as real\-world problems in chemistry and hyperparameter optimization\.

$\\dagger$$\\dagger$footnotetext:Corresponding author:gsutterp@uwaterloo\.ca## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.04113v1/x1.png)Figure 1:Visualization of the out\-of\-the\-loop multi\-fidelity setting, emphasizing the distinction between thein\-the\-loopandout\-of\-the\-loopfidelities\.\(I\)TheonlineBayesian optimization loop actively queries lower\-fidelity functions\.\(II\)Theofflinestage takes the final top\-kksuggestionsout of the loopto be observed on the highest\-fidelity objective\.A wide array of problems in chemistry\(Griffiths and Hernández\-Lobato[2020](https://arxiv.org/html/2608.04113#bib.bib50); Greenawayet al\.[2023](https://arxiv.org/html/2608.04113#bib.bib14); Muthyalaet al\.[2025](https://arxiv.org/html/2608.04113#bib.bib72)\), biology\(Romeroet al\.[2013](https://arxiv.org/html/2608.04113#bib.bib23); Ruberget al\.[2023](https://arxiv.org/html/2608.04113#bib.bib15); Martenset al\.[2025](https://arxiv.org/html/2608.04113#bib.bib64)\), and engineering\(Amentet al\.[2023](https://arxiv.org/html/2608.04113#bib.bib52); Feureret al\.[2022](https://arxiv.org/html/2608.04113#bib.bib16); Lamet al\.[2018](https://arxiv.org/html/2608.04113#bib.bib65)\)consists of optimizing black\-box functions\. Such objectives often have prohibitively large input domains and are expensive to compute\. This creates the need for data\-efficient black\-box optimization algorithms that are able to find the optimal input point by querying the objective function as few times as possible\. Bayesian optimization\(BO; Močkus[1975](https://arxiv.org/html/2608.04113#bib.bib20); Garnett[2023](https://arxiv.org/html/2608.04113#bib.bib18)\)is a principled way to solve this problem following the principles of Bayesian decision theory\.

One solution to expensive objective function evaluations is the use of functions with different cost\-fidelity trade\-offs as proxies for the true, prohibitive objective\. This setting is addressed by multi\-fidelity Bayesian optimization\(MF\-BO; Kennedy and O’Hagan[2000](https://arxiv.org/html/2608.04113#bib.bib71); Huanget al\.[2006](https://arxiv.org/html/2608.04113#bib.bib73)\)algorithms, where costs and correlations between proxies and the true objective are considered when selecting the next point to query in the BO loop\. For example, one can employ numerical simulators or machine learning models instead of performing wet\-lab experiments\(Guanet al\.[2022](https://arxiv.org/html/2608.04113#bib.bib74)\)\.

However, in many applications, the highest fidelity level is prohibitively expensive, so practitioners often do not include it in the BO loop\. Instead, the highest level available during online exploration is not the exact gold standard that is being optimized\. Usually, practitioners run optimization on the best available fidelity and evaluate the obtained maximizers on the real objective function of interest\. Notably, the results of previous experiments are saved for future reference and experiments, which can be exploited by future optimization campaigns\. For example, the highest fidelity in material design is the physical experiment in the laboratory, but often BO is run considering density functional theory\(DFT; Argaman and Makov[2000](https://arxiv.org/html/2608.04113#bib.bib63)\)as the objective, with lower fidelities corresponding to lighter DFT configurations and/or machine learning interatomic potential models, such as UMA\(Woodet al\.[2025](https://arxiv.org/html/2608.04113#bib.bib62)\)\. Only once the BO algorithm is done, its top suggestions are used in lab experiments, the real objective functions are computed, and the results are saved\.

In this work, we first characterize the suboptimality of standard MF\-BO under this setting, even assuming knowledge of the true correlation between fidelities—which is not available in real\-world applications\. This highlights the need for extra information when the highest fidelity is not available in the loop\. Next, we propose a solution based on transferring information from previous optimization tasks to address the suboptimality problem\. We incorporate information from offline data collected at the end of past experiments via a multi\-task multi\-fidelity kernel\. Importantly, the proposed deep kernel can leverage task\-specific features that are either explicitly provided or extracted from unstructured metadata using modern foundation models\(Bommasaniet al\.[2021](https://arxiv.org/html/2608.04113#bib.bib80)\), making use of the rich textual context available in real\-world applications\. We validate our approach on synthetic benchmarks as well as real\-world applications in chemistry and hyperparameter tuning, demonstrating the importance of knowledge transfer in the OOL\-MF\-BO setting\.

![Refer to caption](https://arxiv.org/html/2608.04113v1/x2.png)\(a\)Latent processes
![Refer to caption](https://arxiv.org/html/2608.04113v1/x3.png)\(b\)Objective functions
![Refer to caption](https://arxiv.org/html/2608.04113v1/x4.png)\(c\)Target decomposition

Figure 2:Illustration of the pitfalls of out\-of\-the\-loop multi\-fidelity optimization under the LMC kernel, withM=2M=2\.\(a\)Three latent functionsu1,u2,u3u\_\{1\},u\_\{2\},u\_\{3\}sampled from GPs with different lengthscales\.\(b\)Higher and lower fidelity functions, withf\(1\)=u1\+u2f^\{\(1\)\}=u\_\{1\}\+u\_\{2\}andf\(2\)=2​u1\+u2\+u3f^\{\(2\)\}=2\\,u\_\{1\}\+u\_\{2\}\+u\_\{3\}\. That is, the processu3u\_\{3\}is present in the target but not in the observable fidelity\.\(c\)The observable componentf∥\(2\)\\smash\{f^\{\(2\)\}\_\{\\\|\}\}does not have the same maximizer as the targetf\(2\)\\smash\{f^\{\(2\)\}\}due to the shift imposed byf⟂\(2\)\\smash\{f^\{\(2\)\}\_\{\\perp\}\}\.The contributions of this work are as follows:

- •We are, to the best of our knowledge, the first to formalize the OOL\-MF\-BO problem\.
- •We introduce theoretical results characterizing the suboptimality of OOL\-MF\-BO even if the true correlations between fidelities are known\.
- •We introduce a method that leverages historical experimental data and task\-specific features—extracted from either structured metadata or foundation models—to escape the aforementioned suboptimality, leading to a reduction in cumulative regret in multiple benchmarks\.
- •We show empirical results in the OOL\-MF\-BO setting across standard synthetic functions and on real\-world chemistry and hyperparameter optimization benchmarks\.

## 2Preliminaries

Letf:𝒳→ℝf:\{\\mathcal\{X\}\}\\rightarrow\{\\mathbb\{R\}\}denote an unknown objective function on a space𝒳⊂ℝd\{\\mathcal\{X\}\}\\subset\{\\mathbb\{R\}\}^\{d\}\. The goal of black\-box optimization is to find an optimal point𝒙∗∈arg​max𝒙∈𝒳⁡f​\(𝒙\)\{\\bm\{x\}\}\_\{\*\}\\in\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}f\(\{\\bm\{x\}\}\)while assuming thatffis expensive to compute and the input domain cannot be explored exhaustively\. The objective function is accessed via noisy observationsy=f​\(𝒙\)\+ϵy=f\(\{\\bm\{x\}\}\)\+\\epsilonwithϵ∼𝒩​\(0,σn2\)\{\\epsilon\}\\sim\{\\mathcal\{N\}\}\(0,\\sigma\_\{n\}^\{2\}\), stored in the observation dataset𝒟t=\{\(𝒙i,yi\)\}i=1t\\mathcal\{D\}\_\{t\}=\\smash\{\\\{\(\{\\bm\{x\}\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{t\}\}\.

### Bayesian Optimization

Bayesian optimization\(Močkus[1975](https://arxiv.org/html/2608.04113#bib.bib20); Garnett[2023](https://arxiv.org/html/2608.04113#bib.bib18)\)provides a principled solution for black\-box optimization problems based on two main components: a surrogate modelp​\(f\|𝒟t\)p\(f\|\\mathcal\{D\}\_\{t\}\)and an acquisition functionα​\(𝒙;𝒟t\)\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\)\. The surrogate model represents the current posterior belief over the unknown objective function, which is used by the acquisition function to evaluate candidate input locations and select the next query point according to𝒙t\+1=arg​max𝒳⁡α​\(𝒙;𝒟t\)\{\\bm\{x\}\}\_\{t\+1\}=\\operatorname\*\{arg\\,max\}\_\{\{\\mathcal\{X\}\}\}\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\)\. The optimization loop terminates after a predefined number of iterationsTTis reached\. Then, the algorithm returns its final choice𝒙^T=arg​max𝒙i∈𝒟T⁡yi\\hat\{\{\\bm\{x\}\}\}\_\{T\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\_\{i\}\\in\\mathcal\{D\}\_\{T\}\}y\_\{i\}, given by the observed point with the highest objective value\.

#### Multi\-Fidelity Bayesian Optimization

The multi\-fidelity extension of the Bayesian optimization framework assumes that we have access to several functions in\{f\(m\):𝒳→ℝ\}m=1M\\smash\{\\\{f^\{\(m\)\}:\{\\mathcal\{X\}\}\\to\\mathbb\{R\}\\\}\_\{m=1\}^\{M\}\}with the highest fidelity functionf\(M\)f^\{\(M\)\}being the objective to be maximized, i\.e\., we aim to find

𝒙∗∈arg​max𝒙∈𝒳⁡f\(M\)​\(𝒙\)\{\\bm\{x\}\}\_\{\*\}\\in\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}f^\{\(M\)\}\(\{\\bm\{x\}\}\)The observations dataset is a collection of triples\(𝒙i,yi,mi\)\(\{\\bm\{x\}\}\_\{i\},y\_\{i\},m\_\{i\}\), wheremi∈\{1,…,M\}m\_\{i\}\\in\\\{1,\.\.\.,M\\\}indicates the fidelity queried on stepii\. For each fidelity, there is an associated querying costc\(m\)≥0c^\{\(m\)\}\\geq 0\(assumed to be independent of𝒙\{\\bm\{x\}\}\)\. In the multi\-fidelity regime, the acquisition function is given byα​\(𝒙,m;𝒟t\)\\alpha\(\{\\bm\{x\}\},m;\\mathcal\{D\}\_\{t\}\), which also incorporates the fidelity as an input\. The algorithm runs until the optimization budgetΛ\\Lambdais exhausted\. Both the surrogate model and the acquisition function are adapted to deal with a set of correlated functions\. Analogous to the single\-fidelity case, at termination, the algorithm makes a final suggestion\. Usually, this is done by suggesting the observed point from the highest fidelity with the highest objective value, i\.e\.𝒙^T=arg​max𝒙i∈𝒟T,m⁡yi\\hat\{\{\\bm\{x\}\}\}\_\{T\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\_\{i\}\\in\\mathcal\{D\}\_\{T,m\}\}y\_\{i\}where𝒟T,m=\{\(𝒙i,yi,mi\):mi=M\}\\mathcal\{D\}\_\{T,m\}=\\\{\(\{\\bm\{x\}\}\_\{i\},y\_\{i\},m\_\{i\}\):m\_\{i\}=M\\\}\. Another option is to suggest𝒙^T=arg​max𝒙∈𝒳⁡𝔼​\[f\(M\)​\(𝒙\)\|𝒟T\]\\hat\{\{\\bm\{x\}\}\}\_\{T\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\)\}\(\{\\bm\{x\}\}\)\|\\mathcal\{D\}\_\{T\}\], allowing for points that were not observed in the highest fidelity level or not observed at all\.

### Gaussian Processes

Gaussian processes\(GPs; Rasmussen and Williams[2006](https://arxiv.org/html/2608.04113#bib.bib19)\)are the standard choice for surrogate modelling in BO\. They define a distribution over functions specified by a mean functionμ:𝒳→ℝ\\mu:\{\\mathcal\{X\}\}\\to\\mathbb\{R\}and a covariance kernelk:𝒳×𝒳→ℝk:\{\\mathcal\{X\}\}\\times\{\\mathcal\{X\}\}\\to\\mathbb\{R\}\. Given a dataset𝒟t\\mathcal\{D\}\_\{t\}with inputs and noisy observations, a GP priorf∼𝒢​𝒫​\(μ,k\)f\\sim\\mathcal\{GP\}\(\\mu,k\)induces a multivariate Gaussian distribution over the unseen function values and observed data𝒟t\\mathcal\{D\}\_\{t\}\. Conditioning on the data yields a GP posterior whose predictive distribution at a test point𝒙\{\\bm\{x\}\}is Gaussian with closed\-form meanμt​\(𝒙\)\\mu\_\{t\}\(\{\\bm\{x\}\}\)and varianceσt​\(𝒙\)\\sigma\_\{t\}\(\{\\bm\{x\}\}\)\.

#### Vector\-valued Gaussian Process

When dealing with vector\-valued functionsf:𝒳→ℝMf:\{\\mathcal\{X\}\}\\to\\mathbb\{R\}^\{M\}, it is still possible to use GPs\. Under this setting, it is assumed thatf​\(x\)=\(f1​\(x\),…,fM​\(x\)\)⊤f\(x\)=\(f\_\{1\}\(x\),\\dots,f\_\{M\}\(x\)\)^\{\\top\}follows a vector\-valued GP\(Álvarezet al\.[2012](https://arxiv.org/html/2608.04113#bib.bib66)\), fully characterized by a mean function𝝁:𝒳→ℝM\\bm\{\\mu\}:\{\\mathcal\{X\}\}\\to\\mathbb\{R\}^\{M\}and a matrix\-valued kernel𝐊:𝒳×𝒳→ℝM×M\\mathbf\{K\}:\{\\mathcal\{X\}\}\\times\{\\mathcal\{X\}\}\\to\\mathbb\{R\}^\{M\\times M\}\. The entries\(𝐊​\(𝒙,𝒙′\)\)m,m′\(\\mathbf\{K\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\)\_\{m,m^\{\\prime\}\}in the matrix𝐊​\(𝒙,𝒙′\)\\mathbf\{K\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)correspond to the covariance betweenf\(m\)​\(𝒙\)f^\{\(m\)\}\(\{\\bm\{x\}\}\)andf\(m′\)​\(𝒙′\)f^\{\(m^\{\\prime\}\)\}\(\{\\bm\{x\}\}^\{\\prime\}\)\.

A popular choice is the linear model of coregionalization\(LMC; Journel and Huijbregts[1978](https://arxiv.org/html/2608.04113#bib.bib82); Goovaerts[1997](https://arxiv.org/html/2608.04113#bib.bib81)\):

\[𝐊​\(𝒙,𝒙′\)\]m,m′=∑q=1Q𝑩m,m′\(q\)​kq​\(𝒙,𝒙′\),\[\\mathbf\{K\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\]\_\{m,m^\{\\prime\}\}=\\sum\_\{q=1\}^\{Q\}\{\\bm\{B\}\}^\{\(q\)\}\_\{m,m^\{\\prime\}\}k\_\{q\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\),\(1\)where𝑩\(q\)∈ℝM×M\{\\bm\{B\}\}^\{\(q\)\}\\in\\mathbb\{R\}^\{M\\times M\}is a positive definite coregionalization matrix andkqk\_\{q\}is a kernel forq∈\{1,…,Q\}q\\in\\\{1,\\dots,Q\\\}\. This corresponds to modelling each output as a linear combination of latent processes,f\(m\)​\(𝒙\)=∑q=1Qaq\(m\)​uq​\(𝒙\)f^\{\(m\)\}\(\{\\bm\{x\}\}\)=\\sum\_\{q=1\}^\{Q\}a^\{\(m\)\}\_\{q\}u\_\{q\}\(\{\\bm\{x\}\}\), whereuq∼𝒢​𝒫​\(0,kq\)u\_\{q\}\\sim\\mathcal\{GP\}\(0,k\_\{q\}\)are i\.i\.d\. latent processes\.

When all latent processes have the same kernel, the resulting model is called an intrinsic coregionalization model\(ICM; Goovaerts[1997](https://arxiv.org/html/2608.04113#bib.bib81)\), for which the covariance is:

\[𝐊​\(𝒙,𝒙′\)\]m,m′=𝑩m,m′​k​\(𝒙,𝒙′\),\[\\mathbf\{K\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\]\_\{m,m^\{\\prime\}\}=\{\\bm\{B\}\}\_\{m,m^\{\\prime\}\}k\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\),\(2\)
Another popular kernel, the multi\-information source \(MISO\) model ofPoloczeket al\.\([2017](https://arxiv.org/html/2608.04113#bib.bib83)\), can be written in the LMC form by takingQ=MQ=Mand settingf\(M\)=u1f^\{\(M\)\}=u\_\{1\}andf\(m\)=u1\+um\+1f^\{\(m\)\}=u\_\{1\}\+u\_\{m\+1\}form=1,…,M−1m=1,\\dots,M\-1\. Note that this requires distinct kernels for each process, placing it in the LMC family but outside ICM\.

## 3Pitfalls of Out\-of\-the\-Loop Fidelity

![Refer to caption](https://arxiv.org/html/2608.04113v1/x5.png)Figure 3:Proposed solution for the OOL\-MF\-BO setting based on data from historical tasks for which the highest fidelity was observed\. The surrogate model has access to a dataset of historical tasks alongside the data being collected from the available fidelities of the current task\.We start by characterizing what happens when we only have access to the lower fidelities in the BO loop\. As noted before, in many complex scientific applications the true objective function is prohibitively expensive—both computationally and financially—to be accessed within the loop\.

Consider a multi\-fidelity optimization problem with functions\{f\(m\)\}m=1M\\smash\{\\\{f^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\}drawn from a joint Gaussian process with zero mean and an LMC kernel, that is, each fidelity is a linear combination of the same set of latent processes\{uq\}q=1Q\\smash\{\\\{u\_\{q\}\\\}\_\{q=1\}^\{Q\}\}\. Let the costs\{c\(m\)\}m=1M\\smash\{\\\{c^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\}be such thatc\(M\)=∞c^\{\(M\)\}=\\infty, and let the budget be finite,Λ\>0\\Lambda\>0\. Therefore, we havemi<Mm\_\{i\}<Mfor all elements in𝒟t=\{\(𝒙i,yi,mi\)\}i=1N\\mathcal\{D\}\_\{t\}=\\\{\(\{\\bm\{x\}\}\_\{i\},y\_\{i\},m\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. The final suggestion of the algorithm is given by

𝒙^T=arg​max𝒙∈𝒳⁡𝔼​\[f\(M\)​\(𝒙\)∣𝒟T\],\\hat\{\{\\bm\{x\}\}\}\_\{T\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\)\}\(\{\\bm\{x\}\}\)\\mid\\mathcal\{D\}\_\{T\}\],\(3\)since we are unable to query our target fidelity\.

First, we partition the set of latent functions based on their presence in the lower fidelities\. Next, using the two sets in this partition, we decompose the objective function into observable and unobservable components\.

###### Definition 3\.1\.

Let𝒮obs=\{q\|aq\(m\)≠0​for some​m<M\}\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}=\\\{q\\;\|\\;\\smash\{a^\{\(m\)\}\_\{q\}\}\\neq 0\\;\\text\{for some\}\\;m<M\\\}denote the set of indices of latent processes that are observable in at least one of the lower fidelities\. In addition, define its complement𝒮obs𝖼=\{q\|aq\(m\)=0​for all​m<M\}\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}=\\\{q\\;\|\\;\\smash\{a^\{\(m\)\}\_\{q\}\}=0\\;\\text\{for all\}\\;m<M\\\}, capturing the latent processes that are not present in any observable fidelity\.

###### Definition 3\.2\.

Let the target fidelity bef\(M\)​\(𝒙\)=∑q=1Qaq\(M\)​uq​\(𝒙\)f^\{\(M\)\}\(\{\\bm\{x\}\}\)=\\smash\{\\sum\_\{q=1\}^\{Q\}a^\{\(M\)\}\_\{q\}u\_\{q\}\(\{\\bm\{x\}\}\)\}\. Splitting this sum according to the partition\{𝒮obs,𝒮obs𝖼\}\\\{\{\\mathcal\{S\}\}\_\{\\text\{obs\}\},\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}\\\}of\{1,…,Q\}\\\{1,\\dots,Q\\\}from[Definition˜3\.1](https://arxiv.org/html/2608.04113#S3.Thmtheorem1)yields

f\(M\)\\displaystyle f^\{\(M\)\}=f∥\(M\)\+f⟂\(M\),where\\displaystyle=\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\+\\smash\{f^\{\(M\)\}\_\{\\perp\}\},\\quad\\text\{where\}f∥\(M\)\\displaystyle\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}:=∑q∈𝒮obsaq\(M\)​uq,f⟂\(M\):=∑q∈𝒮obs𝖼aq\(M\)​uq\.\\displaystyle=\\sum\_\{q\\in\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}\}a^\{\(M\)\}\_\{q\}u\_\{q\},\\quad\\smash\{f^\{\(M\)\}\_\{\\perp\}\}=\\sum\_\{q\\in\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}\}a^\{\(M\)\}\_\{q\}u\_\{q\}\.

This decomposition of the objective function allows us to reason about what can be reconstructed from out\-of\-the\-loop observations\. The key object to study under these conditions is the posterior meanμt\(M\)=𝔼​\[f\(M\)∣𝒟t\]\\mu^\{\(M\)\}\_\{t\}=\\mathbb\{E\}\[f^\{\(M\)\}\\mid\\mathcal\{D\}\_\{t\}\], which is used to make suggestions\.

###### Proposition 3\.3\.

Under the LMC kernel defined in \([1](https://arxiv.org/html/2608.04113#S2.E1)\), the highest\-fidelity posterior mean equals the posterior mean of the observable component:μt\(M\)=𝔼​\[f∥\(M\)∣𝒟t\]\\mu^\{\(M\)\}\_\{t\}=\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\\mid\\mathcal\{D\}\_\{t\}\]\. The unobservable componentf⟂\(M\)\\smash\{f^\{\(M\)\}\_\{\\perp\}\}contributes nothing\.

Crucially, this inability to capture information fromf⟂\(M\)\\smash\{f^\{\(M\)\}\_\{\\perp\}\}is not resolved as more low\-fidelity data is collected\. As a result, whenever the unobserved component shifts the location of the optimum, the algorithm incurs irreducible regret\.

###### Theorem 3\.4\.

Assume𝔼​\[f∥\(M\)∣𝒟t\]→f∥\(M\)\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\\mid\\mathcal\{D\}\_\{t\}\]\\to\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}uniformly ast→∞t\\to\\inftyand that the lower\-fidelity observations become dense in𝒳\{\\mathcal\{X\}\}\. Given the candidate𝐱^t\\hat\{\{\\bm\{x\}\}\}\_\{t\}selected via \([3](https://arxiv.org/html/2608.04113#S3.E3)\), wheneverarg​max⁡f\(M\)≠arg​max⁡f∥\(M\)\\operatorname\*\{arg\\,max\}f^\{\(M\)\}\\neq\\operatorname\*\{arg\\,max\}\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}the regret converges to a strictly positive constant:limt→∞\(f\(M\)​\(𝐱∗\)−f\(M\)​\(𝐱^t\)\)\>0\\lim\\limits\_\{t\\to\\infty\}\(f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\\hat\{\{\\bm\{x\}\}\}\_\{t\}\)\)\>0\.

A concrete instance of the LMC problem is presented in[Figure˜2](https://arxiv.org/html/2608.04113#S1.F2)\. In this bi\-fidelity example, the model has direct access to the true linear coefficients\. However, since the lower\-fidelity function does not include the higher\-frequency component, the posterior collapses to the observed component and the regret converges to a positive value\.

The suboptimality of the LMC just presented relies on the set of non\-observable latents being non\-empty\. For the ICM setting, however, we show that the regret is likely irreducible even when𝒮obs𝖼=∅\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}=\\emptyset, as the target posterior mean converges to a fixed combination of the lower fidelities that in general does not share the optimum location with the objective\.

###### Corollary 3\.6\.

Under the ICM kernel defined in \([2](https://arxiv.org/html/2608.04113#S2.E2)\), the highest\-fidelity posterior mean corresponds to a fixed linear combination of the lower\-fidelity posterior means, i\.e\.,μt\(M\)​\(𝐱\)=∑m=1M−1wm​μt\(m\)​\(𝐱\)\\mu^\{\(M\)\}\_\{t\}\(\{\\bm\{x\}\}\)=\\sum\_\{m=1\}^\{M\-1\}w\_\{m\}\\mu^\{\(m\)\}\_\{t\}\(\{\\bm\{x\}\}\), where the weights𝐰\{\\bm\{w\}\}do not depend on the input location𝐱\{\\bm\{x\}\}\. Furthermore, if𝐱∗≠arg​max𝐱∈𝒳​∑m=1M−1wm​f\(m\)​\(𝐱\)\{\\bm\{x\}\}\_\{\*\}\\neq\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\sum\_\{m=1\}^\{M\-1\}w\_\{m\}f^\{\(m\)\}\(\{\\bm\{x\}\}\), the simple regret converges to a strictly positive constant:limt→∞\(f\(M\)​\(𝐱∗\)−f\(M\)​\(𝐱^t\)\)\>0\\lim\_\{t\\to\\infty\}\(f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\\hat\{\{\\bm\{x\}\}\}\_\{t\}\)\)\>0\.

This effect is clear in the bi\-fidelity setting, which reduces to the high\-fidelity posterior mean being a simple rescaling of the lower\-fidelity one\. Therefore, whenever the lower\-fidelity maximizer or minimizer is not aligned with the true optima, the model will have irreducible regret\.

## 4Handling Out\-of\-the\-Loop Fidelity

The root of the suboptimality stated in the previous section is the absence of the target fidelity information within the loop\. Therefore, to escape this regime, it is necessary to incorporate a higher\-fidelity signal in the surrogate model\. As, by definition, the highest fidelity of the current objective is not available, the solution is to incorporate data from*outside*the current task\. Fortunately, in many science and engineering problems, practitioners do not start their optimization campaign in a vacuum\. Instead, they often possess a repository of historical data from previous experiments—datasets where the highest\-fidelity outcomes from those experiments have already been observed and recorded\. Note, however, that the relationship between those previous experiments and the current experiment is often ambiguous, as past experiments may come from partially relevant configurations or settings\.

Based on this observation, we propose a solution for the aforementioned problem in OOL\-MF\-BO\. Formally, considerNNsets of functions\{fn\(m\)\}m=1M\\smash\{\\smash\{\\\{f^\{\(m\)\}\_\{n\}\\\}\_\{m=1\}^\{M\}\}\}forn=1​…​Nn=1\.\.\.N, where the subscriptnnindicates the task index \(i\.e\., previous experiments\), and we are interested in optimizing taskNN\(i\.e\., current experiment\)\. Assume we have historical observationsℋn=\{\(𝒙t,y\(mt\),mt\)\}t=1Tn\\mathcal\{H\}\_\{n\}=\\smash\{\\\{\(\{\\bm\{x\}\}\_\{t\},y^\{\(m\_\{t\}\)\},m\_\{t\}\)\\\}\_\{t=1\}^\{T\_\{n\}\}\}, for which the highest fidelity is available\. We denote byℋ:=\{ℋn\}n=1N−1\\mathcal\{H\}:=\\smash\{\\\{\\mathcal\{H\}\_\{n\}\\\}\_\{n=1\}^\{N\-1\}\}the set of all historical data\. In addition, assume each task has a task context vectorene\_\{n\}that the optimization algorithm can access\. We assume that the previous tasks cannot be queried; only their past observations are available\.

### Surrogate modelling

To incorporate the information from the previous tasks, we replace the fidelity kernel to also include tasks:

Cov\(fn\(m\)\(𝒙\),\\displaystyle\\mathrm\{Cov\}\(f^\{\(m\)\}\_\{n\}\(\{\\bm\{x\}\}\),fn′\(m′\)\(𝒙′\)\)=k\(\(𝒙,m,n\),\(𝒙′,m′,n′\)\)\\displaystyle f^\{\(m^\{\\prime\}\)\}\_\{n^\{\\prime\}\}\(\{\\bm\{x\}\}^\{\\prime\}\)\)=k\(\(\{\\bm\{x\}\},m,n\),\(\{\\bm\{x\}\}^\{\\prime\},m^\{\\prime\},n^\{\\prime\}\)\)=kdata​\(𝒙,𝒙′\)⋅kθ​\(\(m,n\),\(m′,n′\)\)\.\\displaystyle=k\_\{\\text\{data\}\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\\cdot k\_\{\\theta\}\(\(m,n\),\(m^\{\\prime\},n^\{\\prime\}\)\)\.The newly introduced termkθ​\(\(m,n\),\(m′,n′\)\)k\_\{\\theta\}\(\(m,n\),\(m^\{\\prime\},n^\{\\prime\}\)\)is a deep kernel\(Wilsonet al\.[2016](https://arxiv.org/html/2608.04113#bib.bib78)\)that operates on fidelity indices and task features:

kθ​\(\(m,n\),\(m′,n′\)\)=kRBF​\(ℰ​\(m,en;θ\),ℰ​\(m′,en′;θ\)\),k\_\{\\theta\}\(\(m,n\),\(m^\{\\prime\},n^\{\\prime\}\)\)=k\_\{\\text\{RBF\}\}\(\{\\mathcal\{E\}\}\(m,e\_\{n\};\\theta\),\{\\mathcal\{E\}\}\(m^\{\\prime\},e\_\{n^\{\\prime\}\};\\theta\)\),whereℰ\{\\mathcal\{E\}\}is a neural network parametrized byθ\\theta\.

We use a deep kernel as it can flexibly learn complex, nonlinear relationships between tasks and fidelities\. This is particularly relevant in our setting, as the task features—whether manually defined or extracted via foundation models—can interact with fidelity indices in nontrivial ways\. This allows the model to reliably identify which information from the previous tasks is relevant to the current objective, as not all previous data is equally relevant\.

[Figure˜3](https://arxiv.org/html/2608.04113#S3.F3)illustrates how offline data from previous tasks can improve the modelling of the incumbent function and break the linear combination effect previously stated\. Data from historical tasks is taken into account in the modelling of the current objective and its lower\-fidelity functions\.

### Acquisition function

The proposed method does not impose restrictions on the choice of acquisition function\. Once the historical datasetℋ\\mathcal\{H\}is considered in the surrogate model, any multi\-fidelity acquisition function such as MF\-MES\(Takenoet al\.[2020](https://arxiv.org/html/2608.04113#bib.bib79)\)or MF\-EI\(Huanget al\.[2006](https://arxiv.org/html/2608.04113#bib.bib73)\)can be used\. Importantly, the selected acquisition function should remain formulated to optimize the unobserved highest\-fidelity objective\. At each step of the optimization loop, the next point and fidelity queried are given by

𝒙t\+1,mt\+1=arg​max𝒙∈𝒳,m∈\[M−1\]⁡α​\(𝒙,m;𝒟t,ℋ\)\.\{\\bm\{x\}\}\_\{t\+1\},m\_\{t\+1\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\},m\\in\[M\-1\]\}\\alpha\(\{\\bm\{x\}\},m;\\mathcal\{D\}\_\{t\},\\mathcal\{H\}\)\.The main change from the usual MF\-BO is the exclusion of the highest fidelity level from the optimization domain\.

### Task Features

Although some optimization problems may have natural task features, this is not the case for all problems\. In fact, many real\-world problems can be better explained in words than through a hand\-crafted feature vector\. A natural solution to this is to make use of foundation models trained on a wide variety of modalities and domains\. An example of a task description that we use in our experiments is provided in[Figure˜4](https://arxiv.org/html/2608.04113#S4.F4)\. Notably, we extract only one vector per task, representing a negligible cost\.

Hydrocarbon task \| Solvation energy benchmarkContext: Non\-polar, hydrophobic solutes\. Dominant interactions: weak Van der Waals dispersive forces\. Features: Zero dipole moment, lack of hydrogen bond donors or acceptors\. Chemical space: Alkanes, alkenes, and aromatics\. Solvation driven by cavity formation energy\.Figure 4:Task description example from solvation energy benchmark\. To generate the task embeddings, the description is fed to an LLM that maps it to a fixed\-dimensional vector\.

## 5Related work

![Refer to caption](https://arxiv.org/html/2608.04113v1/x6.png)Figure 5:Performance comparison of our proposed methods on synthetic benchmark functions\. Results are averaged over 20 independent trials with shading indicating the standard error\.#### Multi\-task Bayesian Optimization

Multi\-task BO was originally proposed bySwerskyet al\.\([2013](https://arxiv.org/html/2608.04113#bib.bib57)\)as a solution to transfer knowledge across tasks to speed up the optimization of a related task\. In their work, they use an ICM kernel, learning the correlation between tasks directly by maximum likelihood estimation\. Alternatively,Fenget al\.\([2020](https://arxiv.org/html/2608.04113#bib.bib56)\)proposed the latent embedding multi\-output kernel, in which the external task embeddings are fed to an RBF kernel function\. When the task embeddings are extracted by a neural network, this is closely related to deep kernel methods\(Wilsonet al\.[2016](https://arxiv.org/html/2608.04113#bib.bib78); Zhanget al\.[2025a](https://arxiv.org/html/2608.04113#bib.bib59)\); however, in this case, the network weights are frozen\. More recently, BOLT\(Zenget al\.[2025](https://arxiv.org/html/2608.04113#bib.bib54)\)was introduced as a framework for using large language models \(LLMs\) to improve BO transfer across tasks\. In their framework, an LLM is fine\-tuned to produce a good initialization from task\-specific context\.

Zhanget al\.\([2025d](https://arxiv.org/html/2608.04113#bib.bib55)\)is the only work to combine multi\-fidelity and multi\-task BO\. Their work differs from ours in three main ways: \(i\) the highest fidelity is always observable, \(ii\) the acquisition function used also weighs exploration for future tasks when selecting the next query point, and \(iii\) correlations among tasks arise from the i\.i\.d\. assumption over task parameters rather than an explicit task kernel acting on the task contexts\.

#### Bayesian Optimization Without Access to the Objective Function

The central characteristic of our problem setting is not having access to the highest fidelity in the BO loop\. In contrast, prior work has examined other settings, characterized by different constraints and structural assumptions on the objective function\.Zhanget al\.\([2025b](https://arxiv.org/html/2608.04113#bib.bib58)\)investigated the setting where, instead of directly selecting a point, one selects an action that is linked to the input space via a conditional distribution\. Related to the multi\-fidelity setting,Mikkolaet al\.\([2023](https://arxiv.org/html/2608.04113#bib.bib53)\)explored the problem of unreliable fidelities and proposed a robust algorithm for MF\-BO\. Furthermore, another related area of work is offline model\-based optimization\(Kimet al\.[2026](https://arxiv.org/html/2608.04113#bib.bib60)\), which leverages powerful surrogate and generative models to optimize a function in a completely offline setting\. In contrast, we study the distinct challenge of OOL\-MF\-BO, in which the highest fidelity of the objective function is removed from the optimization loop, and lower fidelities are queried directly\.

## 6Experiments

![Refer to caption](https://arxiv.org/html/2608.04113v1/x7.png)Figure 6:\(Top\)Performance comparison of all methods on three molecular optimization benchmarks: Xe/Kr selectivity, solvation energy, and polarizability\. Results are averaged over 10 trials\.\(Bottom\)Cumulative regret curves for logistic regression and SVM design spaces\. Curves are averaged across 80 trials \(8 datasets, each with 10 trials\)\.We perform OOL\-MF\-BO experiments on both synthetic functions, molecular benchmarks, and hyperparameter optimization tasks\. To evaluate performance, we visualize the cumulative regret at each step, i\.e\.,Rt=∑τ=1tf\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙^τ\)R\_\{t\}=\\sum\_\{\\tau=1\}^\{t\}f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\\hat\{\{\\bm\{x\}\}\}\_\{\\tau\}\)\. We consider the following methods:

- •Random Search: Suggestions are made randomly by picking𝒙^t∼Uniform​\(𝒳\)\\hat\{\{\\bm\{x\}\}\}\_\{t\}\\sim\\text\{Uniform\}\(\{\\mathcal\{X\}\}\)\.
- •Next\-Best: The surrogate models the current task up to the highest observable fidelity\. The suggested point is given by𝒙^t=arg​max𝒙∈𝒳⁡𝔼​\[f\(M−1\)​\(𝒙\)\|𝒟t\]\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\-1\)\}\(\{\\bm\{x\}\}\)\|\\mathcal\{D\}\_\{t\}\]\.
- •Single\-Task: Surrogate models all fidelities of the current task\. The suggested point is given by𝒙^t=arg​max𝒙∈𝒳⁡𝔼​\[f\(M\)​\(𝒙\)\|𝒟t\]\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\)\}\(\{\\bm\{x\}\}\)\|\\mathcal\{D\}\_\{t\}\]\.
- •Multi\-Task: Our proposed method in which the out\-of\-the\-loop fidelity is maximized\. The suggested point is given by𝒙^t=arg​max𝒙∈𝒳⁡𝔼​\[f\(M\)​\(𝒙\)\|𝒟t,ℋ\]\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\)\}\(\{\\bm\{x\}\}\)\|\\mathcal\{D\}\_\{t\},\\mathcal\{H\}\]\.
- •Multi\-Task Next\-Best: Variation of our proposed method in which the next\-best fidelity is maximized\. The suggested point is given by𝒙^t=arg​max𝒙∈𝒳⁡𝔼​\[f\(M−1\)​\(𝒙\)\|𝒟t,ℋ\]\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[f^\{\(M\-1\)\}\(\{\\bm\{x\}\}\)\|\\mathcal\{D\}\_\{t\},\\mathcal\{H\}\]\.

For all methods, exceptRandom Search, we use a GP with a constant mean and an RBF covariance kernel scaled by the dimension\(Hvarfneret al\.[2024](https://arxiv.org/html/2608.04113#bib.bib67)\)as the data kernel\. ForSingle\-TaskandNext\-Best, we show results with ICM, LMC, and MISO, covering popular multi\-fidelity kernels used in practice\. For theNext\-Bestvariants, Appendix[C](https://arxiv.org/html/2608.04113#A3)repeats these experiments using the observed incumbent rather than the posterior\-mean maximizer, with no qualitative change\. The deep kernel networkℰ\{\\mathcal\{E\}\}is parameterized as a two\-layer network with 16 hidden units and an 8\-dimensional output\. The acquisition function used is MF\-MES\(Takenoet al\.[2020](https://arxiv.org/html/2608.04113#bib.bib79)\)\. All experiments are implemented using the BoTorch package\(Balandatet al\.[2020](https://arxiv.org/html/2608.04113#bib.bib28)\)\.

### Synthetic Functions

We now evaluate the methods on synthetic functions commonly used in the MF\-BO literature\. Specifically, given a set ofMMmulti\-fidelity objective functionsg\(m\):ℝd→ℝg^\{\(m\)\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}form=1,…,Mm=1,\\dots,M, we generate a family of related tasks by sectioning the original functions along a specific input dimension\. Let the input space be decomposed such that the firstd′=d−1d^\{\\prime\}=d\-1dimensions represent the design variables𝒙∈ℝd−1\{\\bm\{x\}\}\\in\\mathbb\{R\}^\{d\-1\}, and thedd\-th dimension represents a task\-defining parameter\. By selecting a discrete set of task values𝒵=\{z1,…,zN\}\\mathcal\{Z\}=\\\{z\_\{1\},\\dots,z\_\{N\}\\\}, we define thenn\-th task at themm\-th fidelity level,fn\(m\):ℝd−1→ℝf^\{\(m\)\}\_\{n\}:\\mathbb\{R\}^\{d\-1\}\\to\\mathbb\{R\}, asfn\(m\)​\(𝒙\)=g\(m\)​\(x1,…,xd−1,zn\)\\smash\{f^\{\(m\)\}\_\{n\}\}\(\{\\bm\{x\}\}\)=\\smash\{g^\{\(m\)\}\}\(x\_\{1\},\\dots,x\_\{d\-1\},z\_\{n\}\)\. In this framework,znz\_\{n\}serves as a contextual feature\. This construction ensures that tasks are naturally correlated, as they are derived from slices of the same underlyingdd\-dimensional response surfaceg\(m\)g^\{\(m\)\}\.

Under this setting, we perform experiments on multi\-fidelity versions of eight synthetic functions111https://www\.sfu\.ca/˜ssurjano/optimization\.html: Branin, Michalewicz, Park, Styblinski\-Tang, Levy, Ackley, Rosenbrock, and Hartmann\. For all functions, we perform 20 trials, in each trial samplingM∼Uniform​\(\{2,3,4,5\}\)M\\sim\\text\{Uniform\}\(\\\{2,3,4,5\\\}\),N∼Uniform​\(\{2,3,4,5\}\)N\\sim\\text\{Uniform\}\(\\\{2,3,4,5\\\}\),\|Hn\|∼Uniform​\(\{20,30,40,50\}\)\|H\_\{n\}\|\\sim\\text\{Uniform\}\(\\\{20,30,40,50\\\}\)\. Given the number of tasks, the task\-defining parameters are sampled uniformly\. The costs are set toc\(m\)=2m−1c^\{\(m\)\}=2^\{m\-1\}form≤M−1m\\leq M\-1andc\(M\)=∞c^\{\(M\)\}=\\infty, capturing a situation where lowering the fidelity corresponds to halving the data size or tolerance parameter\(Eggenspergeret al\.[2021](https://arxiv.org/html/2608.04113#bib.bib84)\)\. Points in the historical observations are sampled uniformly with fidelity ratio2−\(m−1\)\{2^\{\-\(m\-1\)\}\}\. More information is presented in Appendix[B](https://arxiv.org/html/2608.04113#A2.SSx1)\.

The results are presented in Figure[5](https://arxiv.org/html/2608.04113#S5.F5)\. Across all functions,Multi\-Taskapproaches improve the results and are able to perform better thanNext\-BestandSingle\-Taskmethods\. Interestingly, theMulti\-Task Next\-Bestmethod is consistently better than its single\-task counterpart\. This gap in performance shows the clear advantage of incorporating previous tasks in the OOL\-MF\-BO setting\. As expected, theSingle\-Taskmethod shows performance comparable toRandom Searchand, at best, on par withNext\-best, highlighting the impossibility of optimizing the unavailable fidelity directly without auxiliary data\. For further analysis of the different components of the setting, we refer to the ablation study exploring the effects of number of tasks, task relevance, and historical data size provided in Appendix[D](https://arxiv.org/html/2608.04113#A4)\.

### Real\-World Applications

#### Molecular Optimization

We experiment with three molecular optimization benchmarks, adapting the MF\-BO problems used inSabanza\-Gilet al\.\([2025](https://arxiv.org/html/2608.04113#bib.bib61)\)to our OOL\-MF\-BO setting\. The problems are \(i\) Xe/Kr selectivity\(Gantzleret al\.[2023](https://arxiv.org/html/2608.04113#bib.bib68)\), \(ii\) solvation energy\(Mobley and Guthrie[2014](https://arxiv.org/html/2608.04113#bib.bib69)\), and \(iii\) polarizability\(Ghahremanpouret al\.[2018](https://arxiv.org/html/2608.04113#bib.bib70)\)\. For all benchmarksM=2M=2and the number of steps is set toT=30T=30for all problems\. We use the Qwen3 Embedding 8B\(Zhanget al\.[2025c](https://arxiv.org/html/2608.04113#bib.bib76)\)model to extract task features from text descriptions of the tasks\.[Figure˜4](https://arxiv.org/html/2608.04113#S4.F4)shows one of the textual descriptions used\. Refer to Appendix[B](https://arxiv.org/html/2608.04113#A2.SSx3)for information on the benchmarks and tasks\.

#### Hyperparameter Optimization

We evaluate the method on two hyperparameter optimization \(HPO\) benchmarks from HPOBench\(Eggenspergeret al\.[2021](https://arxiv.org/html/2608.04113#bib.bib84)\): \(i\) logistic regression \(LR\) and \(ii\) support vector machine \(SVM\)\. For each benchmark, the hyperparameter space is provided alongside the validation accuracy for different fidelity levels, withM=5M=5andM=3M=3for LR and SVM, respectively\. Following the complexity scaling of the benchmark fidelities, costs arec\(m\)=3​c\(m−1\)c^\{\(m\)\}=3c^\{\(m\-1\)\}withc\(1\)=1c^\{\(1\)\}=1and the budget is set toΛ=50\\Lambda=50\. In both benchmarks, tasks correspond to different training and validation datasets, for a total ofN=8N=8tasks\. We perform 10 trials per dataset in the role of the current task, randomly sampling only 3 of the remaining tasks to be present in the historical data for each trial\. Refer to Appendix[B](https://arxiv.org/html/2608.04113#A2.SSx3)for more information\.

The cumulative regret results for both families of benchmarks are presented in[Figure˜6](https://arxiv.org/html/2608.04113#S6.F6)\. Our proposed methodsMulti\-TaskandMulti\-Task Next Bestshow strong performance, obtaining the lowest cumulative regrets for three out of five benchmarks\. In the polarizability benchmark, the observed gains start from the strong initialization due to the knowledge transfer from historical tasks and are maintained thereafter—resulting in significantly lower cumulative regret throughout the iterations\. On the other hand, on both HPOBench tasks \(LR and SVM\) our methods show consistently lower instantaneous regret during the entire run, leading to better results\.

The benefits from historical data are clear when comparingMulti\-TaskandMulti\-Task Next Bestwith the ICM versions ofSingle TaskandNext Best\. Even though our proposed deep kernel has a structure very similar to the ICM kernel, namely a separable data kernel, the inclusion of historical information makes the results go from equivalent toRandom Searchto competitive in all evaluated benchmarks\.

The strong performance of the MISO\-based methods follows from[Remark˜3\.5](https://arxiv.org/html/2608.04113#S3.Thmtheorem5), andNext\-BestLMC on Xe/Kr selectivity is the same effect in another guise: when the maximizers of the target and next\-best fidelities are closely aligned, a method that refuses to extrapolate is difficult to improve upon\. Their subpar performance on the HPO and synthetic benchmarks shows that this condition is neither universal nor checkable in advance\. Gains from modelling the out\-of\-the\-loop fidelity directly are thus realized precisely in the regime characterized in[Section˜3](https://arxiv.org/html/2608.04113#S3)\.

## 7Conclusion

In this work, we explored the out\-of\-the\-loop multi\-fidelity Bayesian optimization setting\. We started from its formal definition and proceeded by characterizing the suboptimality of tackling problems of this family using standard MF\-BO methods with both LMC and ICM—even if the fidelity covariance matrix is known\. Next, we demonstrated how historical data with the highest fidelity information can be used to improve the performance in the OOL\-MF\-BO setting\. We showed the benefits of the proposed method across both synthetic and real\-world objective functions, demonstrating the effectiveness of our approach\.

#### Limitations and Future Work

Our analysis of the OOL suboptimality is developed for the LMC family of surrogates, which, although broad, does not cover all possible kernels; extending it to more general covariance structures remains open\. Furthermore, performance depends on the relevance of the available historical tasks, which in practice is hard to know in advance\. A natural next step is to apply OOL\-MF\-BO to challenging science and engineering problems using domain\-specific foundation models as low\-fidelity functions, and to develop more efficient methods for handling historical data that avoid the cubic complexity of GPs\.

## References

- Kernels for Vector\-Valued Functions: A Review\.Foundations and Trends in Machine Learning4\(3\)\.Cited by:[§2](https://arxiv.org/html/2608.04113#S2.SSx2.SSS0.Px1.p1.8)\.
- S\. Ament, A\. Witte, N\. Garg, and J\. Kusuma \(2023\)Sustainable Concrete via Bayesian Optimization\.arXiv2310\(18288\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- N\. Argaman and G\. Makov \(2000\)Density Functional Theory: An Introduction\.American Journal of Physics68\(1\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p3.1)\.
- M\. Balandat, B\. Karrer, D\. Jiang, S\. Daulton, B\. Letham, A\. G\. Wilson, and E\. Bakshy \(2020\)BoTorch: A Framework for Efficient Monte\-Carlo Bayesian Optimization\.InNeurIPS,Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx1.SSS0.Px4.p1.1),[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx4.p1.1),[§6](https://arxiv.org/html/2608.04113#S6.p2.1)\.
- R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill, E\. Brynjolfsson, S\. Buch, D\. Card, R\. Castellon, N\. S\. Chatterji, A\. S\. Chen, K\. A\. Creel, J\. Davis, D\. Demszky, C\. Donahue, M\. Doumbouya, E\. Durmus, S\. Ermon, J\. Etchemendy, K\. Ethayarajh, L\. Fei\-Fei, C\. Finn, T\. Gale, L\. E\. Gillespie, K\. Goel, N\. D\. Goodman, S\. Grossman, N\. Guha, T\. Hashimoto, P\. Henderson, J\. Hewitt, D\. E\. Ho, J\. Hong, K\. Hsu, J\. Huang, T\. F\. Icard, S\. Jain, D\. Jurafsky, P\. Kalluri, S\. Karamcheti, G\. Keeling, F\. Khani, O\. Khattab, P\. W\. Koh, M\. S\. Krass, R\. Krishna, R\. Kuditipudi, A\. Kumar, F\. Ladhak, M\. Lee, T\. Lee, J\. Leskovec, I\. Levent, X\. L\. Li, X\. Li, T\. Ma, A\. Malik, C\. D\. Manning, S\. P\. Mirchandani, E\. Mitchell, Z\. Munyikwa, S\. Nair, A\. Narayan, D\. Narayanan, B\. Newman, A\. Nie, J\. C\. Niebles, H\. Nilforoshan, J\. F\. Nyarko, G\. Ogut, L\. Orr, I\. Papadimitriou, J\. S\. Park, C\. Piech, E\. Portelance, C\. Potts, A\. Raghunathan, R\. Reich, H\. Ren, F\. Rong, Y\. H\. Roohani, C\. Ruiz, J\. Ryan, C\. R’e, D\. Sadigh, S\. Sagawa, K\. Santhanam, A\. Shih, K\. P\. Srinivasan, A\. Tamkin, R\. Taori, A\. W\. Thomas, F\. Tramèr, R\. E\. Wang, W\. Wang, B\. Wu, J\. Wu, Y\. Wu, S\. M\. Xie, M\. Yasunaga, J\. You, M\. A\. Zaharia, M\. Zhang, T\. Zhang, X\. Zhang, Y\. Zhang, L\. Zheng, K\. Zhou, and P\. Liang \(2021\)On the Opportunities and Risks of Foundation Models\.arXiv2108\(07258\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p4.1)\.
- K\. Eggensperger, P\. Müller, N\. Mallik, M\. Feurer, R\. Sass, A\. Klein, N\. Awad, M\. Lindauer, and F\. Hutter \(2021\)HPOBench: a Collection of Reproducible Multi\-Fidelity Benchmark Problems for HPO\.InThirty\-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track \(Round 2\),Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx3.p1.9),[§6](https://arxiv.org/html/2608.04113#S6.SSx1.p2.7),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px2.p1.6)\.
- Q\. Feng, B\. Letham, H\. Mao, and E\. Bakshy \(2020\)High\-Dimensional Contextual Policy Search with Unknown Context Rewards using Bayesian Optimization\.InNeurIPS,Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Feurer, K\. Eggensperger, S\. Falkner, M\. Lindauer, and F\. Hutter \(2022\)Auto\-sklearn 2\.0: hands\-free AutoML via Meta\-Learning\.JMLR23\(261\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- N\. Gantzler, A\. Deshwal, J\. R\. Doppa, and C\. M\. Simon \(2023\)Multi\-Fidelity Bayesian Optimization of Covalent Organic Frameworks For Xenon/Krypton Separations\.Digital Discovery2\(6\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx2.SSS0.Px2),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px1.p1.2)\.
- R\. Garnett \(2023\)Bayesian Optimization\.Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1),[§2](https://arxiv.org/html/2608.04113#S2.SSx1.p1.5)\.
- M\. M\. Ghahremanpour, P\. J\. Van Maaren, and D\. Van Der Spoel \(2018\)The Alexandria Library, A Quantum\-chemical Database of Molecular Properties for Force Field Development\.Scientific Data5\(1\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx2.SSS0.Px3),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px1.p1.2)\.
- P\. Goovaerts \(1997\)Geostatistics for Natural Resources Evaluation\.Oxford University Press\.Cited by:[§2](https://arxiv.org/html/2608.04113#S2.SSx2.SSS0.Px1.p2.6),[§2](https://arxiv.org/html/2608.04113#S2.SSx2.SSS0.Px1.p3.1)\.
- R\. L\. Greenaway, K\. E\. Jelfs, A\. C\. Spivey, and S\. N\. Yaliraki \(2023\)From Alchemist to AI Chemist\.Nature Reviews Chemistry7\(8\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- R\. Griffiths and J\. M\. Hernández\-Lobato \(2020\)Constrained Bayesian Optimization for Automatic Chemical Design Using Variational Autoencoders\.Chem\. Sci\.11\(2\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- Y\. Guan, D\. Chaffart, G\. Liu, Z\. Tan, D\. Zhang, Y\. Wang, J\. Li, and L\. Ricardez\-Sandoval \(2022\)Machine Learning in Solid Heterogeneous Catalysis: Recent Developments, Challenges and Perspectives\.Chemical Engineering Science248\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p2.1)\.
- D\. Huang, T\. T\. Allen, W\. I\. Notz, and R\. A\. Miller \(2006\)Sequential Kriging Optimization Using Multiple\-Fidelity Evaluations\.Structural and Multidisciplinary Optimization32\(5\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p2.1),[§4](https://arxiv.org/html/2608.04113#S4.SSx2.p1.1)\.
- C\. Hvarfner, E\. O\. Hellsten, and L\. Nardi \(2024\)Vanilla Bayesian Optimization Performs Great in High Dimensions\.InICML,Cited by:[§6](https://arxiv.org/html/2608.04113#S6.p2.1)\.
- A\. G\. Journel and C\. J\. Huijbregts \(1978\)Mining Geostatistics\.Academic Press\.Cited by:[§2](https://arxiv.org/html/2608.04113#S2.SSx2.SSS0.Px1.p2.6)\.
- M\. C\. Kennedy and A\. O’Hagan \(2000\)Predicting the Output From a Complex Computer Code When Fast Approximations Are Available\.Biometrika87\(1\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p2.1)\.
- M\. Kim, J\. Gu, Y\. Yuan, T\. Yun, Z\. Liu, Y\. Bengio, and C\. Chen \(2026\)Offline Model\-Based Optimization: Comprehensive Review\.TMLR\.External Links:ISSN 2835\-8856Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px2.p1.1)\.
- R\. Lam, M\. Poloczek, P\. Frazier, and K\. E\. Willcox \(2018\)Advances in Bayesian Optimization with Applications in Aerospace Engineering\.In2018 AIAA Non\-Deterministic Approaches Conference,Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- A\. Martens, M\. Neufang, A\. Butté, M\. von Stosch, A\. del Rio Chanona, and L\. M\. Helleckes \(2025\)Holistic Bioprocess Development Across Scales Using Multi\-Fidelity Batch Bayesian Optimization\.arXiv2508\(10970\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- P\. Mikkola, J\. Martinelli, L\. Filstroff, and S\. Kaski \(2023\)Multi\-Fidelity Bayesian Optimization with Unreliable Information Sources\.InAISTATS,Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx1.SSS0.Px1),[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx1.SSS0.Px2),[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px2.p1.1)\.
- D\. L\. Mobley and J\. P\. Guthrie \(2014\)FreeSolv: A Database of Experimental and Calculated Hydration Free Energies, with Input Files\.Journal of computer\-aided molecular design28\(7\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx2.SSS0.Px1),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px1.p1.2)\.
- J\. Močkus \(1975\)On Bayesian Methods for Seeking the Extremum\.InOptimization Techniques IFIP Technical Conference,Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1),[§2](https://arxiv.org/html/2608.04113#S2.SSx1.p1.5)\.
- M\. R\. Muthyala, F\. Sorourifar, T\. Tan, Y\. Peng, and J\. A\. Paulson \(2025\)Generative Multiobjective Bayesian Optimization with Scalable Batch Evaluations for Sample\-Efficient De Novo Molecular Design\.Industrial & Engineering Chemistry Research65\(1\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- M\. Poloczek, J\. Wang, and P\. Frazier \(2017\)Multi\-Information Source Optimization\.InNIPS,Cited by:[§2](https://arxiv.org/html/2608.04113#S2.SSx2.SSS0.Px1.p4.4)\.
- C\. E\. Rasmussen and C\. K\. I\. Williams \(2006\)Gaussian Processes for Machine Learning\.The MIT Press\.Cited by:[§2](https://arxiv.org/html/2608.04113#S2.SSx2.p1.8)\.
- P\. A\. Romero, A\. Krause, and F\. H\. Arnold \(2013\)Navigating the Protein Fitness Landscape with Gaussian Processes\.Proceedings of the National Academy of Sciences110\(3\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- S\. J\. Ruberg, F\. Beckers, R\. Hemmings, P\. Honig, T\. Irony, L\. LaVange, G\. Lieberman, J\. Mayne, and R\. Moscicki \(2023\)Application of Bayesian Approaches in Drug Development: Starting a Virtuous Cycle\.Nature Reviews Drug Discovery22\(3\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p1.1)\.
- V\. Sabanza\-Gil, R\. Barbano, D\. Pacheco Gutiérrez, J\. S\. Luterbacher, J\. M\. Hernández\-Lobato, P\. Schwaller, and L\. Roch \(2025\)Best Practices for Multi\-Fidelity Bayesian Optimization in Materials and Molecular Research\.Nature Computational Science5\(7\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx2.p1.1),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px1.p1.2)\.
- K\. Swersky, J\. Snoek, and R\. P\. Adams \(2013\)Multi\-Task Bayesian Optimization\.InNeurIPS,Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p1.1)\.
- S\. Takeno, H\. Fukuoka, Y\. Tsukada, T\. Koyama, M\. Shiga, I\. Takeuchi, and M\. Karasuyama \(2020\)Multi\-Fidelity Bayesian Optimization with Max\-value Entropy Search and its Parallelization\.InICML,Cited by:[§4](https://arxiv.org/html/2608.04113#S4.SSx2.p1.1),[§6](https://arxiv.org/html/2608.04113#S6.p2.1)\.
- J\. Vanschoren, J\. N\. van Rijn, B\. Bischl, and L\. Torgo \(2014\)OpenML: Networked Science in Machine Learning\.SIGKDD Explor\. Newsl\.15\(2\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx3.p2.1)\.
- A\. G\. Wilson, Z\. Hu, R\. Salakhutdinov, and E\. P\. Xing \(2016\)Deep Kernel Learning\.InAISTATS,Cited by:[§4](https://arxiv.org/html/2608.04113#S4.SSx1.p1.1),[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p1.1)\.
- B\. M\. Wood, M\. Dzamba, X\. Fu, M\. Gao, M\. Shuaibi, L\. Barroso\-Luque, K\. Abdelmaqsoud, V\. Gharakhanyan, J\. R\. Kitchin, D\. S\. Levine, K\. Michel, A\. Sriram, T\. Cohen, A\. Das, A\. Rizvi, S\. J\. Sahoo, Z\. W\. Ulissi, and C\. L\. Zitnick \(2025\)UMA: A Family of Universal Models for Atoms\.arXiv2506\(23971\)\.Cited by:[§1](https://arxiv.org/html/2608.04113#S1.p3.1)\.
- S\. Xiong, P\. Z\. G\. Qian, and C\. F\. J\. Wu \(2013\)Sequential Design and Analysis of High\-Accuracy and Low\-Accuracy Computer Codes\.Technometrics55\(1\)\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx1.SSS0.Px3)\.
- Y\. Zeng, N\. Maus, H\. T\. Jones, J\. Tao, F\. Wan, M\. Der Torossian Torres, C\. de la Fuente\-Nunez, R\. Marcus, O\. Bastani, and J\. R\. Gardner \(2025\)Large Scale Multi\-Task Bayesian Optimization with Large Language Models\.arXiv2503\(08131\)\.Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p1.1)\.
- F\. Zhang, T\. Desautels, and Y\. Chen \(2025a\)Robust Multi\-Fidelity Bayesian Optimization with Deep Kernel and Partition\.InAISTATS,Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Zhang, S\. Bouabid, C\. S\. Ong, S\. Flaxman, and D\. Sejdinovic \(2025b\)Indirect Query Bayesian Optimization with Integrated Feedback\.arXiv2412\(13559\)\.Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025c\)Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models\.arXiv\.Cited by:[Appendix B](https://arxiv.org/html/2608.04113#A2.SSx2.p1.1),[§6](https://arxiv.org/html/2608.04113#S6.SSx2.SSS0.Px1.p1.2)\.
- Y\. Zhang, S\. Park, and O\. Simeone \(2025d\)Multi\-Fidelity Bayesian Optimization with Across\-Task Transferable Max\-Value Entropy Search\.IEEE Transactions on Signal Processing73\(\)\.Cited by:[§5](https://arxiv.org/html/2608.04113#S5.SS0.SSS0.Px1.p2.1)\.

## Appendix AProofs

See[3\.3](https://arxiv.org/html/2608.04113#S3.Thmtheorem3)

###### Proof\.

Using the decomposition of the objective function and the linearity of expectation, the posterior mean can be written as

μt\(M\)=𝔼​\[f\(M\)∣𝒟t\]=𝔼​\[f∥\(M\)\+f⟂\(M\)∣𝒟t\]=𝔼​\[f∥\(M\)∣𝒟t\]\+𝔼​\[f⟂\(M\)∣𝒟t\]\.\\begin\{split\}\\mu^\{\(M\)\}\_\{t\}&=\\mathbb\{E\}\[f^\{\(M\)\}\\mid\\mathcal\{D\}\_\{t\}\]\\\\ &=\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\+\\smash\{f^\{\(M\)\}\_\{\\perp\}\}\\mid\\mathcal\{D\}\_\{t\}\]\\\\ &=\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\\mid\\mathcal\{D\}\_\{t\}\]\+\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\perp\}\}\\mid\\mathcal\{D\}\_\{t\}\]\.\\end\{split\}By[Definition˜3\.2](https://arxiv.org/html/2608.04113#S3.Thmtheorem2), the second term can be written as𝔼​\[f⟂\(M\)∣𝒟t\]=∑q∈𝒮obs𝖼aq\(M\)​𝔼​\[uq∣𝒟t\]\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\perp\}\}\\mid\\mathcal\{D\}\_\{t\}\]=\\sum\_\{q\\in\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}\}a^\{\(M\)\}\_\{q\}\\mathbb\{E\}\[u\_\{q\}\\mid\\mathcal\{D\}\_\{t\}\]; however,uq⟂𝒟tu\_\{q\}\\perp\\mathcal\{D\}\_\{t\}for allq∈𝒮obs𝖼q\\in\{\\mathcal\{S\}\}\_\{\\text\{obs\}\}^\{\\mathsf\{c\}\}\. Therefore,𝔼​\[uq∣𝒟t\]=𝔼​\[uq\]=0\\mathbb\{E\}\[u\_\{q\}\\mid\\mathcal\{D\}\_\{t\}\]=\\mathbb\{E\}\[u\_\{q\}\]=0, given that all latent processes have a zero\-mean prior\. ∎

See[3\.4](https://arxiv.org/html/2608.04113#S3.Thmtheorem4)

###### Proof\.

By[Prop\.˜3\.3](https://arxiv.org/html/2608.04113#S3.Thmtheorem3), the posterior mean of the target fidelity evaluates exactly to the posterior mean of the observable component, meaning𝔼​\[f\(M\)​\(𝒙\)∣𝒟t\]=𝔼​\[f∥\(M\)​\(𝒙\)∣𝒟t\]\\mathbb\{E\}\[f^\{\(M\)\}\(\{\\bm\{x\}\}\)\\mid\\mathcal\{D\}\_\{t\}\]=\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\(\{\\bm\{x\}\}\)\\mid\\mathcal\{D\}\_\{t\}\]\. Consequently, the candidate point selected at stepttvia \([3](https://arxiv.org/html/2608.04113#S3.E3)\) can be equivalently written as

𝒙^t=arg​max𝒙∈𝒳⁡𝔼​\[f∥\(M\)​\(𝒙\)∣𝒟t\]\.\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\(\{\\bm\{x\}\}\)\\mid\\mathcal\{D\}\_\{t\}\]\.
By the assumption of uniform convergence,𝔼​\[f∥\(M\)∣𝒟t\]→f∥\(M\)\\mathbb\{E\}\[\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\\mid\\mathcal\{D\}\_\{t\}\]\\to\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}ast→∞t\\to\\infty\. Since the lower\-fidelity observations become dense in𝒳\{\\mathcal\{X\}\}, the maximizer of the posterior mean converges to the maximizer of the limiting function\. Let𝒙∥∗=arg​max𝒙∈𝒳⁡f∥\(M\)​\(𝒙\)\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}\(\{\\bm\{x\}\}\)\. It follows that

limt→∞𝒙^t=𝒙∥∗\.\\lim\_\{t\\to\\infty\}\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}\.
Let𝒙∗=arg​max𝒙∈𝒳⁡f\(M\)​\(𝒙\)\{\\bm\{x\}\}\_\{\*\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}f^\{\(M\)\}\(\{\\bm\{x\}\}\)be the true global optimum\. We can now evaluate the limit of the simple regret ast→∞t\\to\\infty\. Assumingf\(M\)f^\{\(M\)\}is continuous, we have

limt→∞\(f\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙^t\)\)=f\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙∥∗\)\.\\lim\_\{t\\to\\infty\}\\left\(f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\\hat\{\{\\bm\{x\}\}\}\_\{t\}\)\\right\)=f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}\)\.
By the premise thatarg​max⁡f\(M\)≠arg​max⁡f∥\(M\)\\operatorname\*\{arg\\,max\}f^\{\(M\)\}\\neq\\operatorname\*\{arg\\,max\}\\smash\{f^\{\(M\)\}\_\{\\parallel\}\}, the point𝒙∥∗\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}is not a global maximizer of the true objective functionf\(M\)f^\{\(M\)\}\. Therefore, the value of the target fidelity at the true optimum is strictly greater than its value at the observable optimum:

f\(M\)​\(𝒙∗\)\>f\(M\)​\(𝒙∥∗\)\.f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\>f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}\)\.Subtractingf\(M\)​\(𝒙∥∗\)f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}\)from both sides yieldsf\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙∥∗\)\>0f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{\\parallel\}\)\>0, which concludes the proof that the simple regret converges to a strictly positive constant\. ∎

See[3\.6](https://arxiv.org/html/2608.04113#S3.Thmtheorem6)

###### Proof\.

We begin by writing the general formula for the posterior mean for allm≤Mm\\leq M:

μt\(m\)​\(𝒙\)=𝒌m​\(𝒙\)⊤​\(𝑲\+σn2​𝑰\)−1​𝒚,\\mu^\{\(m\)\}\_\{t\}\(\{\\bm\{x\}\}\)=\{\\bm\{k\}\}\_\{m\}\(\{\\bm\{x\}\}\)^\{\\top\}\(\{\\bm\{K\}\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\{\\bm\{y\}\},where𝒌m​\(𝒙\)=\[Cov​\(f\(m\)​\(𝒙\),f\(mi\)​\(𝒙i\)\)\]i=1N\{\\bm\{k\}\}\_\{m\}\(\{\\bm\{x\}\}\)=\[\\mathrm\{Cov\}\(f^\{\(m\)\}\(\{\\bm\{x\}\}\),f^\{\(m\_\{i\}\)\}\(\{\\bm\{x\}\}\_\{i\}\)\)\]\_\{i=1\}^\{N\}is the cross\-covariance vector,𝑲=\[Cov​\(f\(mi\)​\(𝒙i\),f\(mj\)​\(𝒙j\)\)\]i=1,j=1N,N\{\\bm\{K\}\}=\[\\mathrm\{Cov\}\(f^\{\(m\_\{i\}\)\}\(\{\\bm\{x\}\}\_\{i\}\),f^\{\(m\_\{j\}\)\}\(\{\\bm\{x\}\}\_\{j\}\)\)\]\_\{i=1,j=1\}^\{N,N\}is the covariance matrix of the observation dataset, and𝒚=\[yi\]i=1N\{\\bm\{y\}\}=\[y\_\{i\}\]\_\{i=1\}^\{N\}is a vector collecting all observed responses\. Recall that, as we are dealing with the ICM kernel,Cov​\(f\(m\)​\(𝒙\),f\(m′\)​\(𝒙′\)\)=𝑩m,m′​k​\(𝒙,𝒙′\)\\mathrm\{Cov\}\(f^\{\(m\)\}\(\{\\bm\{x\}\}\),f^\{\(m^\{\\prime\}\)\}\(\{\\bm\{x\}\}^\{\\prime\}\)\)=\{\\bm\{B\}\}\_\{m,m^\{\\prime\}\}k\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\.

Let𝒘=𝑩<M,<M−1​𝑩<M,M\{\\bm\{w\}\}=\{\\bm\{B\}\}\_\{<M,<M\}^\{\-1\}\{\\bm\{B\}\}\_\{<M,M\}, where𝑩<M,<M\{\\bm\{B\}\}\_\{<M,<M\}is the block of the coregionalization matrix corresponding to the observable fidelities and𝑩<M,M\{\\bm\{B\}\}\_\{<M,M\}is the block of cross\-terms between the observable and the non\-observable fidelity\. Since𝑩\{\\bm\{B\}\}is positive definite,𝑩<M,<M−1\{\\bm\{B\}\}\_\{<M,<M\}^\{\-1\}, and hence𝒘\{\\bm\{w\}\}, are well defined\.

Note that this definition of𝒘\{\\bm\{w\}\}implies∑m<Mwm​𝑩m,m′=𝑩M,m′\\sum\_\{m<M\}w\_\{m\}\{\\bm\{B\}\}\_\{m,m^\{\\prime\}\}=\{\\bm\{B\}\}\_\{M,m^\{\\prime\}\}for all observablem′m^\{\\prime\}\. Becausemi<Mm\_\{i\}<Mholds for all points in the dataset𝒟t\\mathcal\{D\}\_\{t\}, we have

𝒌M​\(𝒙\)=∑m<Mwm​𝒌m​\(𝒙\)\.\{\\bm\{k\}\}\_\{M\}\(\{\\bm\{x\}\}\)=\\sum\_\{m<M\}w\_\{m\}\{\\bm\{k\}\}\_\{m\}\(\{\\bm\{x\}\}\)\.\(4\)Finally, we write the posterior mean of the highest fidelity and substitute \([4](https://arxiv.org/html/2608.04113#A1.E4)\) for the cross\-covariance term:

μt\(M\)​\(𝒙\)=𝒌M​\(𝒙\)⊤​\(𝑲\+σn2​𝑰\)−1​𝒚=\[∑m<Mwm​𝒌m​\(𝒙\)\]⊤​\(𝑲\+σn2​𝑰\)−1​𝒚=∑m<Mwm​\[𝒌m​\(𝒙\)⊤​\(𝑲\+σn2​𝑰\)−1​𝒚\]=∑m<Mwm​μt\(m\)​\(𝒙\),\\begin\{split\}\\mu^\{\(M\)\}\_\{t\}\(\{\\bm\{x\}\}\)&=\{\\bm\{k\}\}\_\{M\}\(\{\\bm\{x\}\}\)^\{\\top\}\(\{\\bm\{K\}\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\{\\bm\{y\}\}\\\\ &=\\Big\[\\sum\_\{m<M\}w\_\{m\}\{\\bm\{k\}\}\_\{m\}\(\{\\bm\{x\}\}\)\\Big\]^\{\\top\}\(\{\\bm\{K\}\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\{\\bm\{y\}\}\\\\ &=\\sum\_\{m<M\}w\_\{m\}\\big\[\{\\bm\{k\}\}\_\{m\}\(\{\\bm\{x\}\}\)^\{\\top\}\(\{\\bm\{K\}\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\{\\bm\{y\}\}\\big\]\\\\ &=\\sum\_\{m<M\}w\_\{m\}\\mu^\{\(m\)\}\_\{t\}\(\{\\bm\{x\}\}\),\\end\{split\}which concludes the first part of the proof\.

It remains to establish the irreducible regret\. Assume that, ast→∞t\\to\\infty, each lower\-fidelity posterior mean converges uniformly to its function,μt\(m\)→f\(m\)\\mu^\{\(m\)\}\_\{t\}\\to f^\{\(m\)\}for allm<Mm<M, which holds as the lower\-fidelity observations become dense in𝒳\{\\mathcal\{X\}\}\. Combining this with the first part of the proof, the highest\-fidelity posterior mean converges to the fixed combination

μt\(M\)\(𝒙\)→∑m<Mwmf\(m\)\(𝒙\)=:g\(𝒙\)\.\\mu^\{\(M\)\}\_\{t\}\(\{\\bm\{x\}\}\)\\to\\sum\_\{m<M\}w\_\{m\}f^\{\(m\)\}\(\{\\bm\{x\}\}\)=:g\(\{\\bm\{x\}\}\)\.Consequently, the candidate point selected via \([3](https://arxiv.org/html/2608.04113#S3.E3)\) satisfies

limt→∞𝒙^t=arg​max𝒙∈𝒳g\(𝒙\)=:𝒙g∗\.\\lim\_\{t\\to\\infty\}\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}g\(\{\\bm\{x\}\}\)=:\{\\bm\{x\}\}^\{\*\}\_\{g\}\.Let𝒙∗=arg​max𝒙∈𝒳⁡f\(M\)​\(𝒙\)\{\\bm\{x\}\}\_\{\*\}=\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}f^\{\(M\)\}\(\{\\bm\{x\}\}\)be the true global optimum\. Assumingf\(M\)f^\{\(M\)\}is continuous, the limit of the simple regret is

limt→∞\(f\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙^t\)\)=f\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙g∗\)\.\\lim\_\{t\\to\\infty\}\\left\(f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\\hat\{\{\\bm\{x\}\}\}\_\{t\}\)\\right\)=f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{g\}\)\.By the premise𝒙∗≠arg​max𝒙∈𝒳​∑m<Mwm​f\(m\)​\(𝒙\)=𝒙g∗\{\\bm\{x\}\}\_\{\*\}\\neq\\operatorname\*\{arg\\,max\}\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\sum\_\{m<M\}w\_\{m\}f^\{\(m\)\}\(\{\\bm\{x\}\}\)=\{\\bm\{x\}\}^\{\*\}\_\{g\}, the point𝒙g∗\{\\bm\{x\}\}^\{\*\}\_\{g\}is not a global maximizer off\(M\)f^\{\(M\)\}, sof\(M\)​\(𝒙∗\)\>f\(M\)​\(𝒙g∗\)f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\>f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{g\}\)\. Thereforef\(M\)​\(𝒙∗\)−f\(M\)​\(𝒙g∗\)\>0f^\{\(M\)\}\(\{\\bm\{x\}\}\_\{\*\}\)\-f^\{\(M\)\}\(\{\\bm\{x\}\}^\{\*\}\_\{g\}\)\>0, concluding the proof\. ∎

## Appendix BExperimental Details

### Synthetic Functions

Here, we provide the formula for all the multi\-fidelity functions considered in our experiments\. Some multi\-fidelity extensions are taken from previous work \(indicated accordingly\), while others are introduced in this study\.

All functions have a continuous fidelity parameterm∈\[0,1\]m\\in\[0,1\], wherem=1m=1recovers the original function\. For each benchmark, the fidelity is discretized intoMMrandomly chosen levels\.

Throughout, we define the fidelity shift

δ​\(m\)=0\.1​\(1−m\)\.\\delta\(m\)=0\.1\(1\-m\)\.
#### Multi\-fidelity Branin\(Mikkolaet al\.[2023](https://arxiv.org/html/2608.04113#bib.bib53)\):

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=\(x2−\(b−δ​\(m\)\)​x12\+c​x1−r\)2\\displaystyle=\\Bigl\(x\_\{2\}\-\(b\-\\delta\(m\)\)x\_\{1\}^\{2\}\+cx\_\{1\}\-r\\Bigr\)^\{2\}\+10​\(1−t\)​cos⁡\(x1\)\+10,\\displaystyle\\quad\+0\(1\-t\)\\cos\(x\_\{1\}\)\+0,
where

𝒙∈\[−5,10\]×\[0,15\],b=5\.14​π2,c=5π,r=6,t=18​π\.\\begin\{gathered\}\{\\bm\{x\}\}\\in\[\-5,10\]\\times\[0,15\],\\\\ b=\\frac\{5\.1\}\{4\\pi^\{2\}\},\\qquad c=\\frac\{5\}\{\\pi\},\\\\ r=6,\\qquad t=\\frac\{1\}\{8\\pi\}\.\\end\{gathered\}

#### Multi\-fidelity Hartmann\(Mikkolaet al\.[2023](https://arxiv.org/html/2608.04113#bib.bib53)\):

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=−\(α1−δ​\(m\)\)​exp⁡\(−∑j=16A1​j​\(xj−P1​j\)2\)\\displaystyle=\-\(\\alpha\_\{1\}\-\\delta\(m\)\)\\exp\\\!\\Bigl\(\-\\sum\_\{j=1\}^\{6\}A\_\{1j\}\(x\_\{j\}\-P\_\{1j\}\)^\{2\}\\Bigr\)−∑i=24αi​exp⁡\(−∑j=16Ai​j​\(xj−Pi​j\)2\)\.\\displaystyle\\quad\-\\sum\_\{i=2\}^\{4\}\\alpha\_\{i\}\\exp\\\!\\Bigl\(\-\\sum\_\{j=1\}^\{6\}A\_\{ij\}\(x\_\{j\}\-P\_\{ij\}\)^\{2\}\\Bigr\)\.
where

𝒙∈\[0,1\]6,𝜶=\(1\.0,1\.2,3\.0,3\.2\),\{\\bm\{x\}\}\\in\[0,1\]^\{6\},\\qquad\\bm\{\\alpha\}=\(1\.0,1\.2,3\.0,3\.2\),andAAandPPare the standard Hartmann constants\.

#### Multi\-fidelity Park\(Xionget al\.[2013](https://arxiv.org/html/2608.04113#bib.bib77)\):

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=x12​\(1\+x4​\(x2\+x32\)x12−1\)\\displaystyle=\\frac\{x\_\{1\}\}\{2\}\\Biggl\(\\sqrt\{1\+\\frac\{x\_\{4\}\(x\_\{2\}\+x\_\{3\}^\{2\}\)\}\{x\_\{1\}^\{2\}\}\}\-1\\Biggr\)\+\(x1\+\(3−1\.5​δ​\(m\)\)​x4\)​exp⁡\(1\+sin⁡x3\)\.\\displaystyle\\quad\+\\bigl\(x\_\{1\}\+\(3\-5\\delta\(m\)\)x\_\{4\}\\bigr\)\\exp\(1\+\\sin x\_\{3\}\)\.
where𝒙∈\[0,1\]4\{\\bm\{x\}\}\\in\[0,1\]^\{4\}\.

#### Multi\-fidelity Rosenbrock:

\(adapted from BoTorch\(Balandatet al\.[2020](https://arxiv.org/html/2608.04113#bib.bib28)\)\)

f\(𝒙,m\)=∑i=1d−2\[100\(xi\+1−xi2\+δ\(m\)\)2\+\(xi−1\+δ\(m\)\)2\]\.f\(\{\\bm\{x\}\},m\)=\\sum\_\{i=1\}^\{d\-2\}\\Bigl\[100\(x\_\{i\+1\}\-x\_\{i\}^\{2\}\+\\delta\(m\)\)^\{2\}\\\\ \+\(x\_\{i\}\-1\+\\delta\(m\)\)^\{2\}\\Bigr\]\.
where𝒙∈\[−5,10\]d\{\\bm\{x\}\}\\in\[\-5,10\]^\{d\}\.

#### Multi\-fidelity Ackley:

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=−a​exp⁡\(−b​1d−1​∑i=1d−1\(xi−δ​\(m\)\)2\)\\displaystyle=\-a\\exp\\\!\\Biggl\(\-b\\sqrt\{\\frac\{1\}\{d\-1\}\\sum\_\{i=1\}^\{d\-1\}\(x\_\{i\}\-\\delta\(m\)\)^\{2\}\}\\Biggr\)−exp⁡\(1d−1​∑i=1d−1cos⁡\(c​\(xi−δ​\(m\)\)\)\)\\displaystyle\\quad\-\\exp\\\!\\Biggl\(\\frac\{1\}\{d\-1\}\\sum\_\{i=1\}^\{d\-1\}\\cos\\\!\\bigl\(c\(x\_\{i\}\-\\delta\(m\)\)\\bigr\)\\Biggr\)\+a\+e,\\displaystyle\\quad\+a\+e,
where

𝒙∈\[−32\.768,32\.768\]d,\(a,b,c\)=\(20,0\.2,2​π\)\.\{\\bm\{x\}\}\\in\[\-32\.768,32\.768\]^\{d\},\\qquad\(a,b,c\)=\(20,0\.2,2\\pi\)\.

#### Multi\-fidelity Levy:

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=sin2⁡\(π​w1\)\\displaystyle=\\sin^\{2\}\(\\pi w\_\{1\}\)\+∑i=1d−2\(wi−1\)2​\(1\+10​sin2⁡\(π​wi\+1\)\)\\displaystyle\\quad\+\\sum\_\{i=1\}^\{d\-2\}\(w\_\{i\}\-1\)^\{2\}\\bigl\(1\+0\\sin^\{2\}\(\\pi w\_\{i\}\+1\)\\bigr\)\+\(wd−1−1\)2​\(1\+sin2⁡\(2​π​wd−1\)\),\\displaystyle\\quad\+\(w\_\{d\-1\}\-1\)^\{2\}\\bigl\(1\+\\sin^\{2\}\(2\\pi w\_\{d\-1\}\)\\bigr\),
where

𝒙∈\[−10,10\]d,wi=1\+xi−1\+δ​\(m\)4\.\{\\bm\{x\}\}\\in\[\-10,10\]^\{d\},\\qquad w\_\{i\}=1\+\\frac\{x\_\{i\}\-1\+\\delta\(m\)\}\{4\}\.

#### Multi\-fidelity Michalewicz:

f​\(𝒙,m\)\\displaystyle f\(\{\\bm\{x\}\},m\)=−∑i=1d−1sin⁡\(xi−δ​\(m\)\)\\displaystyle=\-\\sum\_\{i=1\}^\{d\-1\}\\sin\(x\_\{i\}\-\\delta\(m\)\)×\[sin⁡\(i​\(xi−δ​\(m\)\)2π\)\]2​A,\\displaystyle\\qquad\\times\\left\[\\sin\\\!\\left\(\\frac\{i\(x\_\{i\}\-\\delta\(m\)\)^\{2\}\}\{\\pi\}\\right\)\\right\]^\{2A\},
where

𝒙∈\[0,π\]d,A=10\.\{\\bm\{x\}\}\\in\[0,\\pi\]^\{d\},\\qquad A=10\.

#### Multi\-fidelity Styblinski–Tang:

f\(𝒙,m\)=12∑i=1d−1\[\(xi−δ\(m\)\)4−16\(xi−δ\(m\)\)2\+5\(xi−δ\(m\)\)\]\.f\(\{\\bm\{x\}\},m\)=\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{d\-1\}\\Bigl\[\(x\_\{i\}\-\\delta\(m\)\)^\{4\}\\\\ \-16\(x\_\{i\}\-\\delta\(m\)\)^\{2\}\+5\(x\_\{i\}\-\\delta\(m\)\)\\Bigr\]\.
where𝒙∈\[−5,5\]d\{\\bm\{x\}\}\\in\[\-5,5\]^\{d\}\.

### Chemistry Applications

The three benchmarks are taken fromSabanza\-Gilet al\.\([2025](https://arxiv.org/html/2608.04113#bib.bib61)\), where the high\- and low\-fidelity values are provided directly in tabular form\. For each dataset, we partition it into separate tasks and generate a textual description of each task using Gemini 3 Pro222https://gemini\.google\.com/\. Such textual descriptions are passed to Qwen3 Embedding 8B\(Zhanget al\.[2025c](https://arxiv.org/html/2608.04113#bib.bib76)\), producing task context vectors\. For the three benchmarks, we use 30 examples per historical task, sampled uniformly on each trial\. We now explain the partitions and provide the task descriptions for each benchmark\.

#### Solvation energy\(Mobley and Guthrie[2014](https://arxiv.org/html/2608.04113#bib.bib69)\):

For this dataset, we split the tasks based on the functional groups present in the molecules\. The four resulting tasks are Nitrogenous, Oxygenated, Halogenated, and Hydrocarbon\. We establish a priority hierarchy \(Nitrogen \> Oxygen \> Halogens\) to assign each molecule to one of four mutually exclusive tasks\. The following task descriptions are used:

Hydrocarbon:“Context: Non\-polar, hydrophobic solutes\. Dominant interactions: weak Van der Waals dispersive forces\. Features: Zero dipole moment, lack of hydrogen bond donors or acceptors\. Chemical space: Alkanes, alkenes, and aromatics\. Solvation driven by cavity formation energy\.”Oxygenated:“Context: Polar, hydrophilic solutes\. Dominant interactions: Strong hydrogen bonding and permanent dipole\-dipole attraction\. Features: High electronegativity difference, presence of hydroxyl \(OH\) or carbonyl \(C=O\) groups\. Chemical space: Alcohols, ethers, ketones\.”Nitrogenous:“Context: Basic, polar solutes\. Dominant interactions: Proton acceptance and electrostatic contributions\. Features: Presence of lone pairs on Nitrogen, variable hybridization \(sp2/sp3\)\. Chemical space: Amines, amides, nitriles\. Solvation driven by specific H\-bond networks\.”Halogenated:“Context: Lipophilic, polarizable solutes\. Dominant interactions: Electrostatic sigma\-hole interactions and halogen bonding\. Features: Large atomic radii, high surface area, ’soft’ electron clouds\. Chemical space: Alkyl halides\. Solvation driven by entropy and polarizability\.”

#### Xe/Kr selectivity\(Gantzleret al\.[2023](https://arxiv.org/html/2608.04113#bib.bib68)\):

We partition this dataset based on the pore diameter of each instance\. This results in 3 tasks: Small Pore \(Å<15\\mathring\{A\}<15, Medium Pore \(15≤Å≤2515\\leq\\mathring\{A\}\\leq 25\), and Large Pore \(Å\>25\\mathring\{A\}\>25\)\. The following task descriptions are used:

Small Pore:“Context: High confinement regime \(< 15 Angstroms\)\. Mechanism: Steric sieving and overlap of potential energy surfaces\. Geometry: Dense packing, high framework density, restricted void fraction\. Separation driven by repulsive forces and size exclusion\.”Medium Pore:“Context: Optimal adsorption regime \(15\-25 Angstroms\)\. Mechanism: Strong guest\-host affinity without steric penalty\. Geometry: Mesoporous, balanced void fraction\. Separation driven by attractive Van der Waals wells and surface interactions\.”Large Pore:“Context: Bulk fluid regime \(\> 25 Angstroms\)\. Mechanism: Diffusion\-dominated transport, weak confinement\. Geometry: Open frameworks, extremely high void fraction, low crystal density\. Separation driven by pore wall surface area rather than pore size\.”

#### Polarizability\(Ghahremanpouret al\.[2018](https://arxiv.org/html/2608.04113#bib.bib70)\):

For this dataset, we split the tasks based on the elemental composition of the molecules\. The four resulting tasks are Hetero\-Halogens, CHN Compounds, CHO Compounds, and Hydrocarbons\. We established a priority hierarchy \(Hetero\-Halogens \[F, Cl, Br, I, S, P\] \> Nitrogen \> Oxygen\) to assign each molecule to one of four mutually exclusive tasks\. The following task descriptions are used:

Hydrocarbon:“Context: Saturated carbon frameworks\. Electronic environment: Hard electron clouds, high HOMO\-LUMO gap\. Polarizability source: Strictly volumetric scaling \(size\-dependent\)\. Absence of permanent dipoles or lone pairs\.”CHO Compound:“Context: Oxygenated carbon frameworks\. Electronic environment: High electronegativity contrast, distorted electron density\. Polarizability source: Localized electron density around Oxygen atoms, permanent dipole contributions\.”CHN Compound:“Context: Nitrogenous carbon frameworks\. Electronic environment: Lone pair donation, resonance effects\. Polarizability source: Mobile pi\-electrons and orbital hybridization changes \(sp/sp2/sp3\)\.”Hetero\-Halogens:“Context: Heavy atom substituted frameworks\. Electronic environment: Diffuse, soft orbitals \(d\-orbital participation for S/P\)\. Polarizability source: High deformability of large valence shells \(I, Br, S, P\)\. Significant Van der Waals radii overlap\.”

![Refer to caption](https://arxiv.org/html/2608.04113v1/x8.png)Figure 7:Synthetic benchmarks with theNext\-Bestmethods reporting the best observed point instead of the posterior\-mean maximizer\. All other methods are unchanged from[Figure˜5](https://arxiv.org/html/2608.04113#S5.F5)\.

### Hyperparameter Optimization Applications

The logistic regression \(LR\) and support vector machine \(SVM\) fromEggenspergeret al\.\([2021](https://arxiv.org/html/2608.04113#bib.bib84)\)are used\. The LR design space consists of the regularization strengthα\\alphaand the initial learning rateη0\\eta\_\{0\}, both within\[10−5,100\]\[10^\{\-5\},10^\{0\}\]; the SVM design space consists of the penalty parameterCCand the kernel coefficientγ\\gamma, both within\[2−10,210\]\[2^\{\-10\},2^\{10\}\]\. All four hyperparameters are optimized in log space\. Both benchmarks provide a single fidelity dimension, discretized by the original authors\. For LR it is the number of SGD iterations, taking the five values\{10,37,111,333,1000\}\\\{10,37,111,333,1000\\\}, with the subsample fidelity fixed to its default value of11; for SVM it is the fraction of the training data used, taking the three values\{1/9,1/3,1\}\\\{1/9,1/3,1\\\}\.

We use the following 8 tasks: 10101, 53, 146818, 146821, 9952, 146822, 31, and 3917\. Each task is represented by the meta\-features of its underlying OpenML\(Vanschorenet al\.[2014](https://arxiv.org/html/2608.04113#bib.bib85)\)dataset, retaining only those meta\-features that are available for all tasks and non\-constant across them\. The resulting 20\-dimensional feature vector is then standardized to zero mean and unit variance per dimension across the task set\. Each of the 8 tasks serves as the target task in turn, and for each run we uniformly sample 3 of the remaining 7 tasks as the historical tasks, so that no method is given access to the full task set at once\. Each historical task contributes 30 observations, uniformly distributed across its fidelity levels, and the target task is seeded with 5 observations at its lowest fidelity\. The historical subset and all initial observations are resampled independently for each of the 10 runs\.

### Computational Resources

Experiments are implemented in Python using the BoTorch\(Balandatet al\.[2020](https://arxiv.org/html/2608.04113#bib.bib28)\)library version 0\.16\.1\. The accompanying code contains all the software requirements and installation instructions\. All experiments are executed on a single NVIDIA L40S usingfloat64, as suggested by BoTorch’s official documentation\. 1

## Appendix CAdditional Results

Throughout the paper, all methods report the maximizer of the posterior mean of their target fidelity\. An alternative convention is to report the best point observed so far,𝒙^t=arg⁡max𝒙i∈𝒟t⁡yi\\hat\{\{\\bm\{x\}\}\}\_\{t\}=\\arg\\max\_\{\{\\bm\{x\}\}\_\{i\}\\in\\mathcal\{D\}\_\{t\}\}y\_\{i\}\. This rule is only available to methods that observe the fidelity they report on, i\.e\., theNext\-Bestvariants; the remaining methods target the out\-of\-the\-loop fidelity, which is never observed in the loop\.

[Figures˜7](https://arxiv.org/html/2608.04113#A2.F7)and[8](https://arxiv.org/html/2608.04113#A3.F8)repeat the experiments of[Section˜6](https://arxiv.org/html/2608.04113#S6)with theNext\-Bestmethods using the observed incumbent\. All other methods and all experimental settings are unchanged\.

![Refer to caption](https://arxiv.org/html/2608.04113v1/x9.png)Figure 8:Real\-world benchmarks under the observed\-incumbent rule\. All other methods are unchanged from[Figure˜6](https://arxiv.org/html/2608.04113#S6.F6)\.
## Appendix DAblation Studies

![Refer to caption](https://arxiv.org/html/2608.04113v1/x10.png)Figure 9:Results of the ablation study of our proposed methods, Multi\-task \(MT\) and Multi\-task Next\-Best \(MT\-NB\) on Branin, Park and Hartmann synthetic functions\. We vary the historical dataset size \(top\), the number of tasks \(middle\), and historical tasks’ relevance \(bottom\)\. Results are averaged over 10 independent trials\.In this section, we evaluate the impact of different components of our method on the Branin, Park, and Hartmann synthetic functions described in Section[6](https://arxiv.org/html/2608.04113#S6)\. Unless stated otherwise, we use the same setting as before and perform 10 trials per configuration\.

#### Historical dataset size

We first analyze the effect of the historical dataset size \(first row of[Figure˜9](https://arxiv.org/html/2608.04113#A4.F9)\)\. We vary\|Hn\|∈\{20,30,40\}\|H\_\{n\}\|\\in\\\{20,30,40\\\}for all previous tasks\. For lower\-dimensional problems \(Branin and Park\), increasing the amount of historical data consistently reduces cumulative regret\. For Hartmann, however, the results are mixed and do not show a clear trend\.

#### Number of tasks

Next, we vary the number of available historical tasks,N∈\{2,3,4\}N\\in\\\{2,3,4\\\}\. The task\-defining parameters are sampledznz\_\{n\}uniformly\. For Branin and Park, having more historical tasks clearly improves performance\. For Hartmann, the effect is less conclusive, likely because task diversity increases with dimensionality\.

#### Auxiliary Task Relevance

Finally, we examine the effect of task similarity\. We sample the incumbent task parameter aszN∼Uniform​\(0,1\)z\_\{N\}\\sim\\text\{Uniform\}\(0,1\)and draw the remaining parametersz1,…,zN−1z\_\{1\},\\dots,z\_\{N\-1\}under low, medium, or high relevance\. Low relevance is defined byplow​\(z∣zN\)∝\|z−zN\|2p\_\{\\text\{low\}\}\(z\\mid z\_\{N\}\)\\propto\|z\-z\_\{N\}\|^\{2\}, medium bypmid​\(z∣zN\)=Uniform​\(0,1\)p\_\{\\text\{mid\}\}\(z\\mid z\_\{N\}\)=\\text\{Uniform\}\(0,1\), and high byphigh​\(z∣zN\)=𝒩​\(zN,0\.12\)p\_\{\\text\{high\}\}\(z\\mid z\_\{N\}\)=\\mathcal\{N\}\(z\_\{N\},0\.1^\{2\}\)\. As shown in the last row of[Figure˜9](https://arxiv.org/html/2608.04113#A4.F9), highly relevant historical tasks substantially improve optimization performance\. Moreover, as dimensionality increases, the differences between low, medium, and high relevance become more pronounced\.