Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization

# Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization
Source: [https://arxiv.org/html/2610.08969](https://arxiv.org/html/2610.08969)
Alejandro Comas\-Leon11footnotemark:1Affiliation:Vector Institute, Toronto, ON, CanadaDavid HolzmüllerAffiliation:SODA Team, INRIA Saclay, Palaiseau, FranceHao WangAffiliation:Department of Chemical Engineering, University of Waterloo, Waterloo, ON, CanadaLuis Ricardez\-SandovalAffiliation:Department of Chemical Engineering, University of Waterloo, Waterloo, ON, CanadaAffiliation:Waterloo Institute for Nanotechnology, University of Waterloo, Waterloo, ON, CanadaPascal PoupartAffiliation:Vector Institute, Toronto, ON, CanadaAgustinus KristiadiAffiliation:Vector Institute, Toronto, ON, CanadaAffiliation:Department of Computer Science, Western University, London, ON, Canada\[4pt\] Cheriton School of Computer ScienceUniversity of WaterlooWaterlooONCanada

###### Abstract

Black\-box optimization problems are ubiquitous across science and engineering, often dealing with expensive objective functions\. This objective latency has two consequences during optimization: \(i\) the objective evaluation dominates execution time, and \(ii\) sample\-efficient algorithms are crucial to accelerate development and avoid wasting resources\. Bayesian optimization \(BO\) methods are thede factochoice of planners for suggesting the next point to try\. Standard BO fits the surrogate model’s hyperparameters with a point estimate\. Alternatively, a fully Bayesian approach uses model averaging to account for uncertainty over the hyperparameters, leading to better uncertainty estimates—useful in the low\-data regime that is pervasive in BO\. However, it is often prohibitively expensive and thus rarely used\. In this work, we propose ELF\-BO, an algorithm that uses the objective evaluation latency to headstart the computation of the next suggestion, allowing for fully Bayesian optimization without incurring substantial decision\-time costs\. This is done by sampling from the hyperparameter posterior*while*the objective is being evaluated, only requiring reweighting of the samples once the objective value is observed\. Across synthetic functions and real\-world applications, we show that ELF\-BO matches the performance of fully Bayesian methods while only incurring decision latency on par with or better than standard BO\. Thus, ELF\-BO makes fully Bayesian optimization practical in real\-world use cases\.

## 1Introduction

Bayesian optimization\([Močkus, 1975](https://arxiv.org/html/2610.08969#bib.bib6), BO;\)often acts as a planner \(suggesting the next query point to evaluate\) for optimizing black\-box functions commonly encountered in science\([Greenaway et al\., 2023](https://arxiv.org/html/2610.08969#bib.bib4);[Muthyala et al\., 2026](https://arxiv.org/html/2610.08969#bib.bib16)\)and engineering\([Lam et al\., 2018](https://arxiv.org/html/2610.08969#bib.bib14);[Ament et al\., 2023b](https://arxiv.org/html/2610.08969#bib.bib13)\)\. Across domains, most applications of BO deal with expensive objective functions, a property that motivates the development of sample\-efficient methods\. The inherent high evaluation latency makes it the main bottleneck in the optimization run\. Meanwhile, after suggesting a query location, the planner waits idly for the evaluation results before continuing\. Crucially, expensive function evaluations also limit applications to cases with restricted data availability, both in initial dataset size and in the number of evaluations allowed in the optimization loop\([Jones et al\., 1998](https://arxiv.org/html/2610.08969#bib.bib10)\)\. The former increases the difficulty of achieving the latter, as a misspecified model can lead to suboptimal queries that waste precious evaluation budget\.

Standard BO methods utilize Gaussian processes\([Rasmussen and Williams, 2006](https://arxiv.org/html/2610.08969#bib.bib5), GP;\)as surrogate models and use a point estimate approach—either maximum likelihood estimation \(MLE\) or maximum a posteriori \(MAP\)—to find the GP’s hyperparameters \(e\.g\., lengthscale, output scale, deep\-kernel parameters, and noise variance\)\. However, relying on a single estimate of the hyperparameters can drastically underestimate the uncertainty over the true hyperparameters, and therefore the uncertainty over the objective values\([De Ath et al\., 2021b](https://arxiv.org/html/2610.08969#bib.bib26)\)\. Importantly, this is a bigger problem in the low\-data regimes where well\-calibrated models are required for sample\-efficient decision\-making\. The same trade\-off arises beyond GPs: the high computational overhead of Bayesian neural networks\([Li et al\., 2024](https://arxiv.org/html/2610.08969#bib.bib43);[Kristiadi et al\., 2024](https://arxiv.org/html/2610.08969#bib.bib3), BNNs;\)makes them similarly expensive at decision time\.

⋯\\cdots⋯\\cdotsWorker⋯\\cdots⋯\\cdotsPlannerTimeStandard MCMC\-Based BO⋯\\cdots⋯\\cdotsWorker⋯\\cdots⋯\\cdotsPlannerTimeELF\-BO \(Ours\)Function evaluationMCMC samplingAcquisition optimizationSample reweighting

Figure 1:Where the time goes in a sequential BO campaign\.Each timeline shows the*worker*\(the experiment or simulator evaluatingff\) and the*planner*\(the BO algorithm\); thick segments denote busy time\.Left:In standard fully Bayesian BO, the two alternate: the planner waits whileffis evaluated, and the worker waits while the planner runs MCMC and optimizes the acquisition function\.Right:ELF\-BO runs MCMC*while*ffis being evaluated, so onceyty\_\{t\}arrives, only a cheap importance reweighting and the acquisition optimization remain, greatly reducing decision latency\.An alternative to point estimates is the use of a fully Bayesian optimization\([Osborne et al\., 2009](https://arxiv.org/html/2610.08969#bib.bib18);[Benassi et al\., 2011](https://arxiv.org/html/2610.08969#bib.bib19)\), in which the hyperparameters are averaged out according to their posterior distribution\. In practice, this is done by using Markov chain Monte Carlo\([Neal, 2011](https://arxiv.org/html/2610.08969#bib.bib25), MCMC;\)methods to obtain samples from the posterior distribution and computing the average of the acquisition function over the samples\. As usual in Bayesian inference, the price is paid with more computation and time, mostly due to the expensive MCMC sampling, rendering fully Bayesian methods prohibitively expensive for many practitioners\. This expensive decision\-making step between objective function evaluations is undesirable in various applications; e\.g\., in automated materials discovery, machine time is valuable, so minimal downtime is desired\([MacLeod et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib38)\)\.

In this work, we show that most of the expensive steps needed in a fully Bayesian approach can be performed while the expensive objective function is being evaluated\. We proposeEvaluationLatencyFullyBayesianOptimization \(ELF\-BO\), which uses the BO planner’s otherwise idle time to reduce the footprint of fully Bayesian optimization after the evaluation result becomes available—see[Fig\.1](https://arxiv.org/html/2610.08969#S1.F1)\. This is accomplished by showing that an expectation over the next hyperparameter posterior can be computed using a weighted expectation with the posterior of the hyperparameters at the current step\. These weights are proportional to the posterior predictive density of the pending evaluation results—thus only the weights are computed once the evaluation is done\. Since this identity relies only on Bayes’ rule, ELF\-BO is not tied to GPs and can be applied to BNNs\. In this work, however, we focus on GPs due to their prevalence and simplicity in BO\.

We evaluate ELF\-BO on 15 standard benchmarks ranging from 2 to 10 dimensions and on three real\-world experimental datasets\. Across these problems, ELF\-BO matches the optimization performance of naïve MCMC\-based fully Bayesian BO after every observation, while its decision latency—the time the experiment sits idle waiting for the next query—stays close to that of point\-estimate BO\. The latter amounts to a decision latency roughly 10×\\timeslower than standard fully Bayesian BO with a Matérn kernel, 70–110×\\timeslower with a deep kernel, and more than two orders of magnitude lower as the dataset grows\. We further use the effective sample size of the importance weights to diagnose when reweighting is reliable, and characterize the regimes in which ELF\-BO yields the largest gains\. Altogether, ELF\-BO makes MCMC\-based fully Bayesian BO practical in real\-world scenarios—the more expensive the function evaluation, the more calibrated the planner will be, without high compute overhead\.

## 2Preliminaries

We consider the maximization problem𝒙∗∈arg​max𝐱∈𝒳⁡f​\(𝐱\)\{\\bm\{x\}\}\_\{\*\}\\in\\argmax\_\{\{\\bm\{x\}\}\\in\\mathcal\{X\}\}f\(\{\\bm\{x\}\}\)wheref:𝒳→ℝf:\\mathcal\{X\}\\to\\mathbb\{R\}is an expensive black\-box objective function that can only be accessed via noisy observationsyi=f⁡\(𝒙i\)\+ϵiy\_\{i\}=f\(\{\\bm\{x\}\}\_\{i\}\)\+\\epsilon\_\{i\}, withϵi∼𝒩⁡\(0,σn2\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{n\}^\{2\}\)\. Bayesian optimization \(BO\) provides a principled solution to this problem grounded in Bayesian decision theory\. The key components of BO are: a surrogate modelp⁡\(f∣𝒟t\)p\(f\\mid\\mathcal\{D\}\_\{t\}\)and an acquisition functionα⁡\(𝒙,𝒟t\)\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\), where𝒟t=\{\(𝒙i,yi\)\}i=1t\\mathcal\{D\}\_\{t\}=\\\{\(\{\\bm\{x\}\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{t\}is the set of observations of the objective function up to the current time stept∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\. The surrogate models the current belief over the objective function given the data observed so far, which the acquisition function then uses to score the value of querying each point next\. The next query is chosen as𝒙t\+1∈arg​max𝐱∈𝒳⁡α​\(𝐱,𝒟t\)\{\\bm\{x\}\}\_\{t\+1\}\\in\\argmax\_\{\{\\bm\{x\}\}\\in\\mathcal\{X\}\}\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\), the objective is evaluated at𝒙t\+1\{\\bm\{x\}\}\_\{t\+1\}, and the procedure repeats until the evaluation budget is exhausted\.

### 2\.1Gaussian Processes

The standard surrogate choice for BO is the Gaussian process\. A Gaussian process𝒢​𝒫​\(μ\(𝜽\),k\(𝜽\)\)\\mathcal\{GP\}\(\\mu^\{\(\{\\bm\{\\theta\}\}\)\},k^\{\(\{\\bm\{\\theta\}\}\)\}\)is a distribution over functions fully characterized by its prior mean functionμ\(𝜽\):𝒳→ℝ\\mu^\{\(\{\\bm\{\\theta\}\}\)\}:\{\\mathcal\{X\}\}\\to\\mathbb\{R\}and its covariance functionk\(𝜽\):𝒳×𝒳→ℝk^\{\(\{\\bm\{\\theta\}\}\)\}:\{\\mathcal\{X\}\}\\times\{\\mathcal\{X\}\}\\to\\mathbb\{R\}, where𝜽\{\\bm\{\\theta\}\}collects all hyperparameters of the mean and covariance functions, together with the observation noise varianceσn2\\sigma\_\{n\}^\{2\}\.

Conditioned on an observation dataset𝒟t=\{\(𝒙i,yi\)\}i=1t\\mathcal\{D\}\_\{t\}=\\\{\(\{\\bm\{x\}\}\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{t\}and on the hyperparameters𝜽\{\\bm\{\\theta\}\}, the posterior distribution over function values is also a GP, with posterior mean and covariance given by

μt\(𝜽\)​\(𝒙\)\\displaystyle\\mu^\{\(\{\\bm\{\\theta\}\}\)\}\_\{t\}\(\{\\bm\{x\}\}\)=μ\(𝜽\)​\(𝒙\)\+κ\(𝜽\)​\(𝒙\)⊤​\(𝑲\(𝜽\)\+σn2​𝑰\)−1​\(𝒚t−μ\(𝜽\)​\(𝑿t\)\)\\displaystyle=\\mu^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\)\+\\mathbf\{\\kappa\}^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\)^\{\\top\}\(\{\\bm\{K\}\}^\{\(\{\\bm\{\\theta\}\}\)\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\(\{\\bm\{y\}\}\_\{t\}\-\\mu^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{X\}\}\_\{t\}\)\)\(1\)kt\(𝜽\)​\(𝒙,𝒙′\)\\displaystyle k^\{\(\{\\bm\{\\theta\}\}\)\}\_\{t\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)=k\(𝜽\)​\(𝒙,𝒙′\)−κ\(𝜽\)​\(𝒙\)⊤​\(𝑲\(𝜽\)\+σn2​𝑰\)−1​κ\(𝜽\)​\(𝒙′\),\\displaystyle=k^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}^\{\\prime\}\)\-\\mathbf\{\\kappa\}^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\)^\{\\top\}\(\{\\bm\{K\}\}^\{\(\{\\bm\{\\theta\}\}\)\}\+\\sigma\_\{n\}^\{2\}\{\\bm\{I\}\}\)^\{\-1\}\\mathbf\{\\kappa\}^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}^\{\\prime\}\),\(2\)where𝑲\(𝜽\)=\[k\(𝜽\)​\(𝒙i,𝒙j\)\]i=1,j=1t,t∈ℝt×t\{\\bm\{K\}\}^\{\(\{\\bm\{\\theta\}\}\)\}=\[k^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\_\{i\},\{\\bm\{x\}\}\_\{j\}\)\]\_\{i=1,j=1\}^\{t,t\}\\in\\mathbb\{R\}^\{t\\times t\}is the observation covariance matrix,κ\(𝜽\)​\(𝒙\)=\[k\(𝜽\)​\(𝒙,𝒙i\)\]i=1t∈ℝt\\mathbf\{\\kappa\}^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\)=\[k^\{\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}\_\{i\}\)\]\_\{i=1\}^\{t\}\\in\\mathbb\{R\}^\{t\}is the vector of cross\-covariances,𝒚t=\[y1,…,yt\]∈ℝt\{\\bm\{y\}\}\_\{t\}=\[y\_\{1\},\\dots,y\_\{t\}\]\\in\\mathbb\{R\}^\{t\}collects the observed objective values, and𝑿t\{\\bm\{X\}\}\_\{t\}collects all observed points\. Definingσt2​\(𝜽\)​\(𝒙\):=kt\(𝜽\)​\(𝒙,𝒙\)\\sigma^\{2\(\{\\bm\{\\theta\}\}\)\}\_\{t\}\(\{\\bm\{x\}\}\):=k^\{\(\{\\bm\{\\theta\}\}\)\}\_\{t\}\(\{\\bm\{x\}\},\{\\bm\{x\}\}\)\\, the marginal posterior at any point𝒙\{\\bm\{x\}\}isp⁡\(f⁡\(𝒙\)∣𝒟t,𝜽\)=𝒩⁡\(f⁡\(𝒙\)∣μt\(𝜽\)​\(𝒙\),σt2​\(𝜽\)​\(𝒙\)\)p\(f\(\{\\bm\{x\}\}\)\\mid\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}\)=\{\\mathcal\{N\}\}\(f\(\{\\bm\{x\}\}\)\\mid\\mu^\{\(\{\\bm\{\\theta\}\}\)\}\_\{t\}\(\{\\bm\{x\}\}\),\\sigma\_\{t\}^\{2\(\{\\bm\{\\theta\}\}\)\}\(\{\\bm\{x\}\}\)\)\. We emphasize that these expressions are conditional on𝜽\{\\bm\{\\theta\}\}; recovering the surrogatep⁡\(f∣𝒟t\)p\(f\\mid\\mathcal\{D\}\_\{t\}\)introduced above requires marginalizing the hyperparameters out, which we discuss in Section[2\.2](https://arxiv.org/html/2610.08969#S2.SS2)\.

### 2\.2Model Averaging

The posterior in[Section2\.1](https://arxiv.org/html/2610.08969#S2.SS1)is conditional on a particular setting of hyperparameters𝜽\{\\bm\{\\theta\}\}; thus, following the prescription of Bayesian inference, we should marginalize out our uncertainty over it, calculatingp⁡\(f∣𝒟t\)=∫p⁡\(f∣𝒟t,𝜽\)​p​\(𝜽∣𝒟t\)​𝑑𝜽p\(f\\mid\\mathcal\{D\}\_\{t\}\)=\\int p\(f\\mid\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)d\{\\bm\{\\theta\}\}\. The problem, however, is that such an integral is intractable\. The hyperparameter posteriorp⁡\(𝜽∣𝒟t\)∝p⁡\(𝒟t∣𝜽\)​p​\(𝜽\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\\propto p\(\\mathcal\{D\}\_\{t\}\\mid\{\\bm\{\\theta\}\}\)\\,p\(\{\\bm\{\\theta\}\}\), given a priorp⁡\(𝜽\)p\(\{\\bm\{\\theta\}\}\)and the GP marginal likelihood, is not Gaussian, since𝜽\{\\bm\{\\theta\}\}enters the likelihood nonlinearly through the kernel matrix, its inverse, and its determinant; moreover, its normalizing constant has no closed form\.

As a consequence, practitioners rely on a point estimate𝜽^\\smash\{\\hat\{\{\\bm\{\\theta\}\}\}\}obtained via maximum likelihood or maximum a posteriori\. Such practice is the standard in BO, and is equivalent to approximatingp⁡\(𝜽∣𝒟t\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)withδ⁡\(𝜽−𝜽^\)\\delta\(\{\\bm\{\\theta\}\}\-\\smash\{\\hat\{\{\\bm\{\\theta\}\}\}\}\)\. Unfortunately, this discards all uncertainty about𝜽\{\\bm\{\\theta\}\}, which is especially bad in low\-data regimes prevalent in BO\.

Monte Carlo \(MC\) methods offer a solution to this problem, allowing us to approximate the intractable integral asp\(f∣𝒟t\)≈1/M∑m=1M𝒢𝒫\(f∣μt\(𝜽m\),kt\(𝜽m\)\)p\(f\\mid\\mathcal\{D\}\_\{t\}\)\\approx 1/M\\sum\_\{m=1\}^\{M\}\\mathcal\{GP\}\(f\\mid\\mu\_\{t\}^\{\(\{\\bm\{\\theta\}\}\_\{m\}\)\},k\_\{t\}^\{\(\{\\bm\{\\theta\}\}\_\{m\}\)\}\), where\{𝜽\(m\)\}m=1M∼p⁡\(𝜽∣𝒟t\)\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\. Different Markov chain Monte Carlo \(MCMC\) algorithms can be used to generate the samples, such as slice sampling\([Neal, 2003](https://arxiv.org/html/2610.08969#bib.bib21);[Murray and Adams, 2010](https://arxiv.org/html/2610.08969#bib.bib20)\)or Hamiltonian Monte Carlo\([Duane et al\., 1987](https://arxiv.org/html/2610.08969#bib.bib24);[Neal, 2011](https://arxiv.org/html/2610.08969#bib.bib25)\)and its adaptive variant NUTS\([Hoffman and Gelman, 2014](https://arxiv.org/html/2610.08969#bib.bib23)\)\.

Under this mixture surrogate, the marginal acquisition function becomes the corresponding mixture of per\-sample acquisitions,

α⁡\(𝒙,𝒟t\)=𝔼p⁡\(𝜽∣𝒟t\)​\[α⁡\(𝒙,𝒟t,𝜽\)\]≈1M​∑m=1Mα⁡\(𝒙,𝒟t,𝜽\(m\)\),\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\)=\\mathbb\{E\}\_\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\[\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}\)\]\\approx\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m\)\}\),\(3\)providedα\\alphais linear in the predictive distribution, so that the marginalization over𝜽\\bm\{\\theta\}and the expectation over function values commute\. This holds for expected improvement\([Jones et al\., 1998](https://arxiv.org/html/2610.08969#bib.bib10), EI;\); for its numerically stable variant Log\-EI\([Ament et al\., 2023a](https://arxiv.org/html/2610.08969#bib.bib2)\), which we use in this work, we average in EI space and take the logarithm of the result\. Other acquisition functions can also be applied but may not reduce to a mixture of per\-sample acquisitions; for instance, upper confidence bound\([Auer, 2002](https://arxiv.org/html/2610.08969#bib.bib40);[Srinivas et al\., 2010](https://arxiv.org/html/2610.08969#bib.bib41)\)is instead computed from the mean and variance of the mixture\.

Figure 2:Overview ofELF\-BO\.Fully Bayesian BO marginalizes the acquisition function over the GP hyperparameters𝜽\{\\bm\{\\theta\}\}, which requires MCMC after every new observation\. We instead run it while the objective is still being evaluated\.*Left:*\(I\)with the candidate𝒙t\{\\bm\{x\}\}\_\{t\}dispatched and its valueyty\_\{t\}still pending, we sample\{𝜽\(m\)\}\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}from the posterior given the data collected so far,p⁡\(𝜽∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\), which is already available\. Each sample gives a different GP fit, and hence a different posterior at𝒙t\{\\bm\{x\}\}\_\{t\}\.*Right:*\(II\)whenyty\_\{t\}arrives, each sample is reweighted by how well it predicted it,w\(m\)∝p⁡\(yt∣𝒟t−1,𝒙t,𝜽\(m\)\)w^\{\(m\)\}\\propto p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m\)\}\), and\(III\)the per\-sample acquisition functionsα⁡\(𝒙,𝒟t,𝜽\(m\)\)\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m\)\}\)are combined into the marginal acquisitionα⁡\(𝒙,𝒟t\)\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\)used to pick the next candidate\.

## 3Evaluation Latency Fully Bayesian Optimization

Consider a dataset𝒟t−1\\mathcal\{D\}\_\{t\-1\}has been observed, a point𝒙t\{\\bm\{x\}\}\_\{t\}was just dispatched for evaluation, andyty\_\{t\}is pending\. Our goal is to start the computations to generate the suggestion as follows:

𝒙t\+1=arg​max𝐱∈𝒳⁡𝔼𝜽∼p⁡\(𝜽∣𝒟t\)​\[α⁡\(𝐱,𝒟t,𝜽\)\],\{\\bm\{x\}\}\_\{t\+1\}=\\argmax\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\[\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}\)\],\(4\)where𝒟t=𝒟t−1∪\{\(𝒙t,yt\)\}\\mathcal\{D\}\_\{t\}=\\mathcal\{D\}\_\{t\-1\}\\cup\\\{\(\{\\bm\{x\}\}\_\{t\},y\_\{t\}\)\\\}and𝜽\{\\bm\{\\theta\}\}are the GP hyperparameters\. The obvious challenge is thatyty\_\{t\}, and consequently𝒟t\\mathcal\{D\}\_\{t\}, are not known yet, so we would have to wait for it to run MCMC to obtain samples\{𝜽\(m\)\}m=1M\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}needed to compute the acquisition\.

The central idea of ELF\-BO is that we can use the objective evaluation time to obtain samples\{𝜽\(m\)\}m=1M\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}from the posterior of the previous stepp⁡\(𝜽∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)and use a reweighted average to compute \([4](https://arxiv.org/html/2610.08969#S3.E4)\), as illustrated in[Fig\.2](https://arxiv.org/html/2610.08969#S2.F2)\. The following proposition justifies this \(proof in Appendix[A](https://arxiv.org/html/2610.08969#A1)\):

###### Proposition 1\.

Letφ\\varphibe a function of𝛉\{\\bm\{\\theta\}\}andp⁡\(𝛉∣𝒟t\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)be the posterior of hyperparameters at stept∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\. Suppose that, for eacht≥1t\\geq 1,p⁡\(𝛉∣𝒟t\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)is absolutely continuous w\.r\.t\.p⁡\(𝛉∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\. Then, for allt=1,…,Tt=1,\\dots,T,

𝔼𝜽∼p⁡\(𝜽∣𝒟t\)​\[φ⁡\(𝜽\)\]=𝔼𝜽∼p⁡\(𝜽∣𝒟t−1\)​\[w⁡\(𝜽\)​φ​\(𝜽\)\],wherew⁡\(𝜽\)=p⁡\(yt∣𝒟t−1,𝒙t,𝜽\)p⁡\(yt∣𝒟t−1,𝒙t\)\.\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\[\\varphi\(\{\\bm\{\\theta\}\}\)\]=\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\[w\(\{\\bm\{\\theta\}\}\)\\,\\varphi\(\{\\bm\{\\theta\}\}\)\],\\qquad\\text\{where\}\\quad w\(\{\\bm\{\\theta\}\}\)=\\frac\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\}\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\.

###### Proof sketch\.

We use the importance sampling argument with the previous posterior as the proposal, so the importance weight is the ratio of the new posterior to the old one\. By Bayes’ rule, this ratio is proportional to the likelihood of the new observation under the predictive distribution\. ∎

The preceding result implies that we can start computing the expectation w\.r\.t\. the posteriorp⁡\(𝜽∣𝒟t\)p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\}\)at the time steptt*before*we receive the actual result foryty\_\{t\}\. In particular, we can already start sampling𝜽\{\\bm\{\\theta\}\}fromp⁡\(𝜽∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\-1\}\)of the previous time stept−1t\-1\. This is the expensive part of this expectation computation due to expensive MCMC\. Indeed, onceyty\_\{t\}is obtained, only simple computation ofw⁡\(𝜽\)w\(\{\\bm\{\\theta\}\}\),φ⁡\(𝜽\)\\varphi\(\{\\bm\{\\theta\}\}\), and averaging are required\.

Notice that[Proposition1](https://arxiv.org/html/2610.08969#Thmtheorem1)casts the expectation as importance sampling\. We can thus use self\-normalized importance sampling\([Owen, 2013](https://arxiv.org/html/2610.08969#bib.bib39)\)to approximate𝔼𝜽∼p⁡\(𝜽∣𝒟t\)​\[α⁡\(𝒙,𝒟t,𝜽\)\]\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\[\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}\)\]using MCMC samples\{𝜽\(m\)\}m=1M\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}fromp⁡\(𝜽∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)obtained*before*the evaluation is ready and reweighted*after*yty\_\{t\}is observed:

α⁡\(𝒙,𝒟t\)≈∑m=1Mw\(m\)​α​\(𝒙,𝒟t,𝜽\(m\)\),w\(m\)=p⁡\(yt\|𝒟t−1,𝒙t,𝜽\(m\)\)∑m′p⁡\(yt\|𝒟t−1,𝒙t,𝜽\(m′\)\)\.\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\}\)\\approx\\sum\_\{m=1\}^\{M\}w^\{\(m\)\}\\,\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m\)\}\),\\qquad w^\{\(m\)\}=\\frac\{p\(y\_\{t\}\|\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m\)\}\)\}\{\\sum\_\{m^\{\\prime\}\}p\(y\_\{t\}\|\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}^\{\(m^\{\\prime\}\)\}\)\}\.\(5\)This moves all MCMC computations to objective evaluation time\. Once the objective value is observed, the acquisition function can be computed by reweighting the sampled hyperparameters based on how well each explains the observed value\.[Algorithm1](https://arxiv.org/html/2610.08969#alg1)summarizes the complete procedure\. The reweighting strategy in ELF\-BO may resemble sequential Monte Carlo\([Chopin, 2002](https://arxiv.org/html/2610.08969#bib.bib28), SMC;\), but in ELF\-BO, a fresh set of samples is generated at each function evaluation, whereas SMC propagates a single population across all steps\.

Importantly, as in all importance sampling algorithms, the approximation is only reliable when the proposalp⁡\(𝜽∣𝒟t−1\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)adequately covers the targetp⁡\(𝜽∣𝒟t\)p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\. This can be monitored through the effective sample size of the self\-normalizing weights:Meff=1/∑m=1M\(w\(m\)\)2M\_\{\\text\{eff\}\}=1/\{\\sum\_\{m=1\}^\{M\}\(w^\{\(m\)\}\)^\{2\}\}, which is available immediately afteryty\_\{t\}is observed\. Values ofMeffM\_\{\\text\{eff\}\}nearMMindicate that the value ofyty\_\{t\}had high probability under a variety of sampled hyperparameters, so the reweighted samples are representative, while low values indicate thatyty\_\{t\}was poorly explained by most of the samples, meaning a few samples dominate and the estimate will collapse to a single point\. In[Section6](https://arxiv.org/html/2610.08969#S6)we analyze the effective sample size of the method across optimization steps\.

Algorithm 1ELF\-BOInitial dataset

𝒟0\\mathcal\{D\}\_\{0\}, acquisition

α\\alpha, number of samples

MM, evaluation budget

TT
Final dataset

𝒟T\\mathcal\{D\}\_\{T\}
\{𝜽\(m\)\}m=1M←MCMC​\(p⁡\(𝜽∣𝒟0\)\)\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\\leftarrow\\texttt\{MCMC\}\(p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{0\}\)\)

w\(m\)←1/Mw^\{\(m\)\}\\leftarrow 1/M

for

t=1,…,Tt=1,\\dots,Tdo

𝒙t=arg​max𝐱∈𝒳∑mw\(m\)α\(𝐱;𝒟t−1,𝜽\(m\)\)\{\\bm\{x\}\}\_\{t\}=\\argmax\_\{\{\\bm\{x\}\}\\in\{\\mathcal\{X\}\}\}\\sum\_\{m\}w^\{\(m\)\}\\,\\alpha\(\{\\bm\{x\}\};\\mathcal\{D\}\_\{t\-1\},\{\\bm\{\\theta\}\}^\{\(m\)\}\)

h←𝐚𝐬𝐲𝐧𝐜​𝚌𝚘𝚖𝚙𝚞𝚝𝚎​\(f,𝒙t\)h\\leftarrow\\mathtt\{\\mathbf\{async\}\\kern 5\.24994ptcompute\}\(f,\{\\bm\{x\}\}\_\{t\}\)⊳\\trianglerightevaluation off⁡\(𝒙t\)f\(\{\\bm\{x\}\}\_\{t\}\)starts

\{𝜽\(m\)\}m=1M←MCMC​\(p⁡\(𝜽∣𝒟t−1\)\)\\\{\{\\bm\{\\theta\}\}^\{\(m\)\}\\\}\_\{m=1\}^\{M\}\\leftarrow\\texttt\{MCMC\}\(p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\)⊳\\trianglerightComputed in parallel

yt←𝐚𝐰𝐚𝐢𝐭​𝚐𝚎𝚝​\_​𝚛𝚎𝚜𝚞𝚕𝚝​\(h\)y\_\{t\}\\leftarrow\\mathtt\{\\mathbf\{await\}\\kern 5\.24994ptget\\\_result\}\(h\)⊳\\trianglerightevaluation off⁡\(𝒙t\)f\(\{\\bm\{x\}\}\_\{t\}\)ends; obtainyty\_\{t\}

𝒟t=𝒟t−1∪\{\(𝒙t,yt\)\}\\mathcal\{D\}\_\{t\}=\\mathcal\{D\}\_\{t\-1\}\\cup\\\{\(\{\\bm\{x\}\}\_\{t\},y\_\{t\}\)\\\}

Compute

\{w\(m\)\}\\\{w^\{\(m\)\}\\\}as in \([5](https://arxiv.org/html/2610.08969#S3.E5)\)

endfor

### 3\.1Decision Latency Reduction

We now analyze the*decision latency*, the wall\-clock time elapsed between an observation arriving and the next query being proposed\. This is relevant whenever idle equipment is costly, samples degrade while waiting, or an operator must dispatch the next experiment immediately, prevalent in automated scientific discovery\([MacLeod et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib38)\)\.

LetTfT\_\{f\}denote the evaluation latency of the objective,TMCT\_\{\\text\{MC\}\}the time to drawMMposterior samples,TαT\_\{\\alpha\}the time to optimize the expected acquisition function, andTwT\_\{w\}the time to compute the weights in \([5](https://arxiv.org/html/2610.08969#S3.E5)\)\. The decision latencies of standard fully Bayesian optimization and ELF\-BO are given by

LFBO=TMC\+TαandLELF\-BO=max⁡\(0,TMC−Tf\)\+Tw\+Tα,L\_\{\\text\{FBO\}\}=T\_\{\\text\{MC\}\}\+T\_\{\\alpha\}\\qquad\\text\{and\}\\qquad L\_\{\\text\{ELF\-BO\}\}=\\max\(0,\\,T\_\{\\text\{MC\}\}\-T\_\{f\}\)\+T\_\{w\}\+T\_\{\\alpha\},respectively\. Therefore, ELF\-BO reduces decision latency by

Δ​L=LFBO−LELF\-BO=min⁡\(TMC,Tf\)−Tw\.\\Delta L=L\_\{\\text\{FBO\}\}\-L\_\{\\text\{ELF\-BO\}\}=\\min\(T\_\{\\text\{MC\}\},T\_\{f\}\)\-T\_\{w\}\.
Note that in many real\-world applications,Tf≥TMCT\_\{f\}\\geq T\_\{\\text\{MC\}\}, so the entire MCMC sampling procedure runs behind the objective evaluation\. In such cases, the reweighting overhead is negligible in practice\. We validate this analysis in[Section6](https://arxiv.org/html/2610.08969#S6)\.

## 4Related Work

#### Fully Bayesian Optimization

Early work on GPs for global optimization considered a fully Bayesian approach to model hyperparameters\([Osborne et al\., 2009](https://arxiv.org/html/2610.08969#bib.bib18);[Benassi et al\., 2011](https://arxiv.org/html/2610.08969#bib.bib19)\)\. For instance, the seminal work of[Snoek et al\. \(2012\)](https://arxiv.org/html/2610.08969#bib.bib7)demonstrated that an MCMC approach using slice sampling\([Neal, 2003](https://arxiv.org/html/2610.08969#bib.bib21);[Murray and Adams, 2010](https://arxiv.org/html/2610.08969#bib.bib20)\)outperforms point estimate\-based methods\. More recently,[De Ath et al\. \(2021b\)](https://arxiv.org/html/2610.08969#bib.bib26)analyzed the benefits of fully Bayesian methods in BO, demonstrating that the MCMC approach is superior to MAP and variational inference methods when using EI and automatic relevance determination kernels\. In the high\-dimensional setting, SAASBO\([Eriksson and Jankowiak, 2021](https://arxiv.org/html/2610.08969#bib.bib22)\)demonstrated the effectiveness of a fully Bayesian method with NUTS\([Hoffman and Gelman, 2014](https://arxiv.org/html/2610.08969#bib.bib23)\)and priors that capture sparseness\. Marginalizing out hyperparameters also provides a textbook solution to overfitting and is beneficial in the overparameterized regime of deep kernels\([Ober et al\., 2021](https://arxiv.org/html/2610.08969#bib.bib27)\)\. Beyond GP surrogates, fully Bayesian BO has also been performed with Bayesian neural networks\([Li et al\., 2024](https://arxiv.org/html/2610.08969#bib.bib43);[Kristiadi et al\., 2023](https://arxiv.org/html/2610.08969#bib.bib1);[Kristiadi et al\., 2024](https://arxiv.org/html/2610.08969#bib.bib3)\), which could equally benefit from ELF\-BO\.

#### Asynchronous Batch Bayesian Optimization

In this setting, multiple function evaluations run in parallel, and, as soon as one of the function evaluations finishes, a new query is launched without waiting for the pending results\.[Ginsbourger et al\. \(2011\)](https://arxiv.org/html/2610.08969#bib.bib29)was the first to work on this problem, using samples from the predictive posterior at the pending location to calculate the expected EI under different objective realizations\.[Alvi et al\. \(2019\)](https://arxiv.org/html/2610.08969#bib.bib32)proposed the use of local penalization around the pending point to reduce redundant selections, while[De Ath et al\. \(2021a\)](https://arxiv.org/html/2610.08969#bib.bib33)used anε\\varepsilon\-greedy algorithm to achieve the same goal\. Alternatively,[Kandasamy et al\. \(2018\)](https://arxiv.org/html/2610.08969#bib.bib30)proposed the use of Thompson Sampling\([Thompson, 1933](https://arxiv.org/html/2610.08969#bib.bib8)\)in which a new sample of the posterior is taken after including only the newly revealed observation\. More recently,[Riegler et al\. \(2026\)](https://arxiv.org/html/2610.08969#bib.bib31)also demonstrated good results across acquisition functions without explicitly modelling the pending evaluations\. On the related problem of stochastically delayed feedback,[Verma et al\. \(2022\)](https://arxiv.org/html/2610.08969#bib.bib34)considered substituting the minimum function value for observations not yet revealed, a pessimistic version of the Kriging Believer method\([Ginsbourger et al\., 2010](https://arxiv.org/html/2610.08969#bib.bib11)\)\. ELF\-BO complements this line of work by operating in settings with a single function evaluation at a time, a scenario often found in real\-world applications\.

Figure 3:Standard benchmarks\.Optimization performance of 15 synthetic functions\.ELF\-BOconsistently followsStandard FBOon most functions, in many of which the fully Bayesian approaches achieve earlier convergence thanMAP\.Figure 4:Real\-world tasks\.Results on three experimental datasets from the Olympus suite\.*Top:*best GAP found versus iteration, with standard error shaded\.ELF\-BOconverges as quickly and reliably asStandard FBO, whileMAPlags and varies more across seeds\.*Bottom:*mean decision latency versus area under GAP curve, with error bars showing the standard error over all steps\. By sampling while the objective is evaluated,ELF\-BOhas area under GAP close toStandard FBO’s at a latency similar toMAP’s, more than 10×\\timeslower thanStandard FBO\.

## 5Experiments

We now evaluate the empirical performance of ELF\-BO\. Our experiments aim to answer two questions: \(i\) Does reweighting hyperparameter samples drawn while the oracle is busy match the performance of running MCMC after each observation? \(ii\) What is the wall\-clock time effect of moving sampling under the objective evaluation?

#### Baselines\.

We compare ELF\-BO against random search and two baselines sharing the surrogate and acquisition function, so that differences between methods reflect only how the GP hyperparameters are handled and when the compute is spent\.Randomsearch samples uniformly from the domain\.MAPfits the hyperparameters by maximizing the marginal likelihood plus log prior at every step, which is the default practice\. Standard fully Bayesian optimization \(Standard FBO\) runs NUTS on the full dataset after each observation arrives; it is the fully Bayesian gold standard and pays the entire sampling cost between evaluations\.ELF\-BOruns the same NUTS sampler on the data available when a query is dispatched, while the oracle is evaluating, and when the observation arrives, it corrects the draws with a single importance reweighting\.

#### Setup\.

All methods use a GP with a constant mean, homoskedastic Gaussian likelihood, and ARD Matérn\-5/2 kernel with output scale\. The lengthscale prior is scaled by dimension as proposed by[Hvarfner et al\. \(2024\)](https://arxiv.org/html/2610.08969#bib.bib15)\. For the sampling methods, we shift the distribution so that the MAP prior’s mode becomes the sampled prior’s median, and we reduce the scale from3\\sqrt\{3\}to 1\. Unless stated otherwise, Standard FBO and ELF\-BO use NUTS\([Hoffman and Gelman, 2014](https://arxiv.org/html/2610.08969#bib.bib23)\)with 4 vectorized chains, 512 warmup steps per chain, keeping a total ofMMsamples after thinning by 16 \(total of16​M16Msamples generated after warmup\)\. We use a dense mass matrix and a max tree depth of 6\. Log\-EI\([Ament et al\., 2023a](https://arxiv.org/html/2610.08969#bib.bib2)\)is used for all methods\. For all experiments, we report the results across 10 trials, with standard errors indicated\. All methods are implemented in BoTorch\([Balandat et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib9)\)\. Further details on the experimental setup are presented in Appendix[B](https://arxiv.org/html/2610.08969#A2)\.

#### Metrics\.

As different objective functions have different scales, GAP\([Barton, 1984](https://arxiv.org/html/2610.08969#bib.bib35)\)is used to evaluate performance:GAP=\(yimax−y0max\)/\(y∗−y0max\)\\text\{GAP\}=\(y\_\{i\}^\{\\text\{max\}\}\-y\_\{0\}^\{\\text\{max\}\}\)/\(y\_\{\*\}\-y\_\{0\}^\{\\text\{max\}\}\), whereyimaxy\_\{i\}^\{\\text\{max\}\}is the best observed value at the i\-th iteration,y0maxy\_\{0\}^\{\\text\{max\}\}is the highest initial value, andy∗y\_\{\*\}is the optimal value\. To evaluate the benefits of leveraging objective evaluation time, we also measure the decision latency of each algorithm\.

### 5\.1Standard Benchmark Functions

Following[De Ath et al\. \(2021a\)](https://arxiv.org/html/2610.08969#bib.bib33), we use the 15 benchmark functions111[https://www\.sfu\.ca/~ssurjano/optimization\.html](https://www.sfu.ca/~ssurjano/optimization.html)ranging fromd=2d=2tod=10d=10: Branin, Eggholder, Goldstein–Price and Six\-Hump Camel \(d=2d=2\); Hartmann \(d=3,6d=3,6\); Ackley and Michalewicz \(d=5,10d=5,10\); Styblinski–Tang \(d=5,7,10d=5,7,10\); and Rosenbrock \(d=7,10d=7,10\)\. Each run starts from2​d2dSobol points and uses a total budget of 100 optimization steps\. The methods that use MCMC takeM=64M=64hyperparameter samples per BO step\. We consider noiseless observations\.

The results presented in[Fig\.3](https://arxiv.org/html/2610.08969#S4.F3)demonstrate thatELF\-BOtracksStandard FBOclosely on most of the 15 functions, with curves largely overlapping within standard error\. This agreement holds across input dimensions and a range of landscape characteristics, including smooth functions such as Branin and more rugged, highly multimodal functions such as Eggholder\. There are, however, functions for whichELF\-BOandStandard FBOdiverge, such as Michalewicz \(d=5d=5\)\. In[Section6](https://arxiv.org/html/2610.08969#S6)we analyze this failure through the lens of effective sample size\. Finally, we observe thatELF\-BOfollowsStandard FBOboth when it outperformsMAPand when they are equivalent, showing no clear preference for cases where point estimates are “good enough”\. This matches the goal of the work, which is to provide a small\-latency alternative to fully Bayesian optimization\.

Figure 5:Deep kernel on real\-world tasks\.Results of the deep kernel on Olympus datasets\. As in experiments with Matérn kernel,ELF\-BOachieves Pareto optimality overStandard FBOandMAP, with an even more pronounced reduction in decision latency\. See Appendix[C](https://arxiv.org/html/2610.08969#A3)for GAP curves\.
### 5\.2Real\-World Applications

In addition, we evaluate ELF\-BO on three experimental datasets curated in the Olympus benchmark suite\([Häse et al\., 2021](https://arxiv.org/html/2610.08969#bib.bib12)\)\. Rather than using learned emulators, we directly treat the datasets as a virtual library—thus, querying a point returns its recorded experimental measure\. HPLC \(d=6d=6and\|𝒳\|=1,386\|\{\\mathcal\{X\}\}\|=1\{,\}386\) tunes the sampling sequence of a robotic direct\-inject HPLC system to maximize peak area\([Roch et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib37)\)\. Photo\-PCE10 and Photo\-WF3 \(d=4d=4and\|𝒳\|=1,040\|\{\\mathcal\{X\}\}\|=1\{,\}040each\) optimize the composition of polymer blends to minimize222We negate the objective and consider it a maximization problem\.photodegradation\([Langner et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib36)\)\. We choose the three Olympus datasets with more than 1000 data points to provide a large enough library for BO\. GAP is computed withy∗y\_\{\*\}set to the best measurement in each dataset\.

All three datasets were collected through physical experiments\. In a live campaign where Standard FBO chooses each experiment before it is run, every evaluation would take far longer than sampling the hyperparameter posterior, leaving the planner idle for most of the campaign—precisely the setting ELF\-BO is designed for\. Although the photodegradation data was collected in a high\-throughput setting, we optimize all three tasks sequentially\. We use an initial dataset of2​d2dSobol points and a total budget of 50 steps for all tasks\. Standard FBO and ELF\-BO useM=128M=128hyperparameter samples per BO step\. More details on the tasks and datasets are provided in Appendix[B](https://arxiv.org/html/2610.08969#A2)\.

The optimization curves presented in[Fig\.4](https://arxiv.org/html/2610.08969#S4.F4)\(top\) demonstrate that, across all tasks,ELF\-BOclosely tracksStandard FBOin performance\. The results also highlight that in real\-world applications with limited evaluation budget, a fully Bayesian approach can improve sample efficiency, achieving better results in earlier optimization steps\. We analyze the sample efficiency using the area under the GAP curve, which is presented in[Fig\.4](https://arxiv.org/html/2610.08969#S4.F4)\(bottom\) alongside average decision latency\. This demonstrates the Pareto dominance ofELF\-BOoverStandard FBOandMAP, displaying optimization performance on par withStandard FBOwhile providing suggestions more than10×10\\timesfaster\.

#### Deep kernel\.

A natural use case for a fully Bayesian approach is with deep kernels\([Wilson et al\., 2016](https://arxiv.org/html/2610.08969#bib.bib17)\), as marginalizing out the network weights can mitigate the overfitting of point estimates in the overparameterized regime\([Ober et al\., 2021](https://arxiv.org/html/2610.08969#bib.bib27)\)\. In addition, they offer a challenging setting forELF\-BO, as the sampled parameters include neural\-network weights, a step toward BNN surrogates\. We repeat the real\-world benchmarks, replacing the Matérn kernel with a deep kernel \(two\-layer neural network; see Appendix[B](https://arxiv.org/html/2610.08969#A2)\)\. As shown in[Fig\.5](https://arxiv.org/html/2610.08969#S5.F5), the decision latency ofStandard FBOgrows 5–9×\\timesrelative to Matérn, whereasELF\-BOremains under 1 s, a 70–110×\\timesreduction, while remaining competitive in sample efficiency\.

## 6Analysis

We now analyze ELF\-BO under different settings, improving our understanding of the method\. To this end, we study the scalability with respect to dataset size and the impact of effective sample size\.

#### Scaling with the number of observations\.

The cost of fully Bayesian optimization grows with the number of observationsnn\. However, ELF\-BO is unaffected by this growth as long as sampling fits within the evaluation time\.[Figure7](https://arxiv.org/html/2610.08969#S6.F7)shows the decision latency as a function of the number of observations on Photo\-WF3 \(d=4d=4\) for the deep kernel model in[Section5\.2](https://arxiv.org/html/2610.08969#S5.SS2)\. WhileStandard FBOlatency grows up to 24 minutes,ELF\-BOremains nearly constant at under 3 seconds\. This highlights the large gains that can be obtained by ELF\-BO when optimization campaigns are long and use more complex models\. The same figure indicates the evaluation time required to fully hide sampling at each step, providing a practical guide for when ELF\-BO is applicable\.

#### Impact of effective sample size\.

As noted in[Section3](https://arxiv.org/html/2610.08969#S3), ELF\-BO depends on how representative the samples collected during the objective evaluation are\. We analyzeMeffM\_\{\\text\{eff\}\}of Michalewicz \(d=5d=5\), a function for which ELF\-BO has difficulties tracking Standard FBO performance, and Hartmann \(d=6d=6\), a function close in dimensionality for which ELF\-BO performs well\.[Figure7](https://arxiv.org/html/2610.08969#S6.F7)presents the step\-wise fraction of effective samples, defined asMeff/MM\_\{\\text\{eff\}\}/M\. The effective sample size for Michalewicz \(d=5d=5\) is lower for most of the optimization steps, although it remains far from collapse for both functions\. The lower values ofMeffM\_\{\\text\{eff\}\}for Michalewicz \(d=5d=5\) help explain why ELF\-BO performs worse on that function\. SinceMeffM\_\{\\text\{eff\}\}is available as soon asyty\_\{t\}is observed, it provides a cheap diagnostic for flagging steps where the reweighted estimate may be less reliable\.

Figure 6:Deep kernel on larger datasets\.Decision latency on Photo\-WF3 \(d=4d=4\) as observation set grows\.Standard FBOincreases with dataset size whileELF\-BOremains constant\.Figure 7:Normalized effective sample size per iteration\.Michalewicz \(d=5d=5\) exhibits a lower effective sample size than Hartmann \(d=6d=6\) for most of the optimization steps\.

## 7Conclusion

We introduced ELF\-BO, a fully Bayesian optimization algorithm that leverages the objective’s execution latency to alleviate the time taken to make a decision on the next point to query\. By drawing samples from the posterior given the data available at dispatch time and reweighting them once the observation is revealed, ELF\-BO removes MCMC from the time between function evaluations\. Across 15 synthetic functions and three real\-world experimental datasets, ELF\-BO matches the optimization performance of MCMC\-based fully Bayesian optimization on most tasks while keeping decision latency close to point estimate BO times\.

#### Limitations and Future Work

ELF\-BO’s latency gains are observed in applications for which objective function evaluations take more time than MCMC, but are absent in applications where evaluations are near\-instant\. Moreover, we only consider GP surrogates in the sequential setting\. Natural extensions include BNN surrogates and asynchronous batch BO with multiple pending evaluations\. More broadly, ELF\-BO shows that the planner’s idle time, which is inherent to expensive black\-box optimization, can be spent on better decisions\.

## References

- Alviet al\.\(2019\)A\. S\. Alvi, B\. Ru, J\. Calliess, S\. J\. Roberts, and M\. A\. OsborneAsynchronous Batch Bayesian Optimisation with Improved Local Penalisation\.InICML,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Amentet al\.\(2023a\)S\. Ament, S\. Daulton, D\. Eriksson, M\. Balandat, and E\. BakshyUnexpected improvements to expected improvement for Bayesian optimization\.InNeurIPS,Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p4.2),[§5](https://arxiv.org/html/2610.08969#S5.SS0.SSS0.Px2.p1.1)\.
- Amentet al\.\(2023b\)S\. Ament, A\. Witte, N\. Garg, and J\. KusumaSustainable Concrete via Bayesian Optimization\.arXiv2310\(18288\)\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1)\.
- Auer \(2002\)P\. AuerUsing Confidence Bounds for Exploitation\-Exploration Trade\-offs\.JMLR3\(Nov\)\.Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p4.2)\.
- Balandatet al\.\(2020\)M\. Balandat, B\. Karrer, D\. Jiang, S\. Daulton, B\. Letham, A\. G\. Wilson, and E\. BakshyBoTorch: A Framework for Efficient Monte\-Carlo Bayesian Optimization\.InNeurIPS,Cited by:[§B\.3](https://arxiv.org/html/2610.08969#A2.SS3.p1.1),[§5](https://arxiv.org/html/2610.08969#S5.SS0.SSS0.Px2.p1.1)\.
- Barton \(1984\)R\. R\. BartonMinimization Algorithms for Functions with Random Noise\.American Journal of Mathematical and Management Sciences4\(1\-2\)\.Cited by:[§5](https://arxiv.org/html/2610.08969#S5.SS0.SSS0.Px3.p1.1)\.
- Benassiet al\.\(2011\)R\. Benassi, J\. Bect, and E\. VazquezRobust Gaussian Process\-Based Global Optimization Using a Fully Bayesian Expected Improvement Criterion\.InInternational Conference on Learning and Intelligent Optimization,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p3.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Chopin \(2002\)N\. ChopinA Sequential Particle Filter Method for Static Models\.Biometrika89\(3\)\.Cited by:[§3](https://arxiv.org/html/2610.08969#S3.p5.2)\.
- De Athet al\.\(2021a\)G\. De Ath, R\. M\. Everson, and J\. E\. FieldsendAsynchronousε\\varepsilon\-Greedy Bayesian Optimisation\.InUAI,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2610.08969#S5.SS1.p1.1)\.
- De Athet al\.\(2021b\)G\. De Ath, R\. M\. Everson, and J\. E\. FieldsendHow Bayesian Should Bayesian Optimisation Be?\.InProceedings of the Genetic and Evolutionary Computation Conference Companion,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p2.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Duaneet al\.\(1987\)S\. Duane, A\.D\. Kennedy, B\. J\. Pendleton, and D\. RowethHybrid Monte Carlo\.Physics Letters B195\(2\)\.Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p3.1)\.
- Eriksson and Jankowiak \(2021\)D\. Eriksson and M\. JankowiakHigh\-Dimensional Bayesian Optimization with Sparse Axis\-Aligned Subspaces\.InUAI,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Ginsbourgeret al\.\(2011\)D\. Ginsbourger, J\. Janusevskis, and R\. Le RicheDealing with Asynchronicity in Parallel Gaussian Process Based Global Optimization\.Ph\.D\. Thesis,Mines Saint\-Etienne\.Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Ginsbourgeret al\.\(2010\)D\. Ginsbourger, R\. Le Riche, and L\. CarraroKriging is well\-suited to parallelize optimization\.InComputational Intelligence in Expensive Optimization Problems,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Greenawayet al\.\(2023\)R\. L\. Greenaway, K\. E\. Jelfs, A\. C\. Spivey, and S\. N\. YalirakiFrom Alchemist to AI Chemist\.Nature Reviews Chemistry7\(8\)\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1)\.
- Häseet al\.\(2021\)F\. Häse, M\. Aldeghi, R\. J\. Hickman, L\. M\. Roch, M\. Christensen, E\. Liles, J\. E\. Hein, and A\. Aspuru\-GuzikOlympus: a benchmarking framework for noisy optimization and experiment planning\.Machine Learning: Science and Technology2\(3\)\.Cited by:[§5\.2](https://arxiv.org/html/2610.08969#S5.SS2.p1.1)\.
- Hoffman and Gelman \(2014\)M\. D\. Hoffman and A\. GelmanThe No\-U\-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo\.\.JMLR15\(1\)\.Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p3.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.08969#S5.SS0.SSS0.Px2.p1.1)\.
- Hvarfneret al\.\(2024\)C\. Hvarfner, E\. O\. Hellsten, and L\. NardiVanilla Bayesian Optimization Performs Great in High Dimensions\.InICML,Cited by:[§5](https://arxiv.org/html/2610.08969#S5.SS0.SSS0.Px2.p1.1)\.
- Joneset al\.\(1998\)D\. R\. Jones, M\. Schonlau, and W\. J\. WelchEfficient global optimization of expensive black\-box functions\.Journal of Global Optimization13\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p4.2)\.
- Kandasamyet al\.\(2018\)K\. Kandasamy, A\. Krishnamurthy, J\. Schneider, and B\. PóczosParallelised Bayesian Optimisation via Thompson Sampling\.InAISTATS,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: A Method for Stochastic Optimization\.InICLR,Cited by:[§B\.2](https://arxiv.org/html/2610.08969#A2.SS2.p2.1)\.
- Kristiadiet al\.\(2023\)A\. Kristiadi, A\. Immer, R\. Eschenhagen, and V\. FortuinPromises and pitfalls of the linearized Laplace in Bayesian optimization\.InFifth Symposium on Advances in Approximate Bayesian Inference,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Kristiadiet al\.\(2024\)A\. Kristiadi, F\. Strieth\-Kalthoff, M\. Skreta, P\. Poupart, A\. Aspuru\-Guzik, and G\. PleissA sober look at LLMs for material discovery: Are they actually good for Bayesian optimization over molecules?\.InICML,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p2.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Lamet al\.\(2018\)R\. Lam, M\. Poloczek, P\. Frazier, and K\. E\. WillcoxAdvances in Bayesian Optimization with Applications in Aerospace Engineering\.In2018 AIAA Non\-Deterministic Approaches Conference,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1)\.
- Langneret al\.\(2020\)S\. Langner, F\. Häse, J\. D\. Perea, T\. Stubhan, J\. Hauch, L\. M\. Roch, T\. Heumueller, A\. Aspuru\-Guzik, and C\. J\. BrabecBeyond Ternary OPV: High\-Throughput Experimentation and Self\-Driving Laboratories Optimize Multicomponent Systems\.Advanced Materials32\(14\)\.Cited by:[§B\.1](https://arxiv.org/html/2610.08969#A2.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2610.08969#S5.SS2.p1.1)\.
- Liet al\.\(2024\)Y\. Li, T\. G\. Rudner, and A\. G\. WilsonA Study of Bayesian Neural Network Surrogates for Bayesian Optimization\.InICLR,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p2.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- MacLeodet al\.\(2020\)B\. P\. MacLeod, F\. G\. L\. Parlane, T\. D\. Morrissey, F\. Häse, L\. M\. Roch, K\. E\. Dettelbach, R\. Moreira, L\. P\. E\. Yunker, M\. B\. Rooney, J\. R\. Deeth, V\. Lai, G\. J\. Ng, H\. Situ, R\. H\. Zhang, M\. S\. Elliott, T\. H\. Haley, D\. J\. Dvorak, A\. Aspuru\-Guzik, J\. E\. Hein, and C\. P\. BerlinguetteSelf\-Driving Laboratory for Accelerated Discovery of Thin\-Film Materials\.Science Advances6\(20\)\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.08969#S3.SS1.p1.1)\.
- Močkus \(1975\)J\. MočkusOn Bayesian Methods for Seeking the Extremum\.InOptimization Techniques IFIP Technical Conference,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1)\.
- Murray and Adams \(2010\)I\. Murray and R\. P\. AdamsSlice Sampling Covariance Hyperparameters of Latent Gaussian Models\.InNIPS,Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p3.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Muthyalaet al\.\(2026\)M\. R\. Muthyala, F\. Sorourifar, T\. Tan, Y\. Peng, and J\. A\. PaulsonGenerative Multiobjective Bayesian Optimization with Scalable Batch Evaluations for Sample\-Efficient De Novo Molecular Design\.Industrial & Engineering Chemistry Research65\(1\)\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p1.1)\.
- Neal \(2003\)R\. M\. NealSlice sampling\.The Annals of Statistics31\(3\)\.Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p3.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Neal \(2011\)R\. NealHandbook of Markov Chain Monte Carlo\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p3.1)\.
- Oberet al\.\(2021\)S\. W\. Ober, C\. E\. Rasmussen, and M\. van der WilkThe Promises and Pitfalls of Deep Kernel Learning\.InUAI,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.08969#S5.SS2.SSS0.Px1.p1.1)\.
- Osborneet al\.\(2009\)M\. A\. Osborne, R\. Garnett, and S\. J\. RobertsGaussian Processes for Global Optimization\.InInternational Conference on Learning and Intelligent Optimization,Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p3.1),[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Owen \(2013\)A\. B\. OwenMonte Carlo Theory, Methods and Examples\.[https://artowen\.su\.domains/mc/](https://artowen.su.domains/mc/)\.Cited by:[§3](https://arxiv.org/html/2610.08969#S3.p5.1)\.
- Rasmussen and Williams \(2006\)C\. E\. Rasmussen and C\. K\. I\. WilliamsGaussian Processes for Machine Learning\.The MIT Press\.Cited by:[§1](https://arxiv.org/html/2610.08969#S1.p2.1)\.
- Riegleret al\.\(2026\)B\. Riegler, J\. Odgers, and V\. FortuinStandard Acquisition Is Sufficient for Asynchronous Bayesian Optimization\.InAISTATS,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Rochet al\.\(2020\)L\. M\. Roch, F\. Häse, C\. Kreisbeck, T\. Tamayo\-Mendoza, L\. P\. E\. Yunker, J\. E\. Hein, and A\. Aspuru\-GuzikChemOS: An Orchestration Software to Democratize Autonomous Discovery\.PLOS ONE15\(4\)\.Cited by:[§B\.1](https://arxiv.org/html/2610.08969#A2.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2610.08969#S5.SS2.p1.1)\.
- Snoeket al\.\(2012\)J\. Snoek, H\. Larochelle, and R\. P\. AdamsPractical Bayesian optimization of machine learning algorithms\.InNeurIPS,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px1.p1.1)\.
- Srinivaset al\.\(2010\)N\. Srinivas, A\. Krause, S\. M\. Kakade, and M\. SeegerGaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design\.InICML,Cited by:[§2\.2](https://arxiv.org/html/2610.08969#S2.SS2.p4.2)\.
- Thompson \(1933\)W\. R\. ThompsonOn the likelihood that one unknown probability exceeds another in view of the evidence of two samples\.Biometrika25\(3\-4\)\.Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Vermaet al\.\(2022\)A\. Verma, Z\. Dai, and B\. K\. H\. LowBayesian Optimization Under Stochastic Delayed Feedback\.InICML,Cited by:[§4](https://arxiv.org/html/2610.08969#S4.SS0.SSS0.Px2.p1.1)\.
- Wilsonet al\.\(2016\)A\. G\. Wilson, Z\. Hu, R\. Salakhutdinov, and E\. P\. XingDeep Kernel Learning\.InAISTATS,Cited by:[§5\.2](https://arxiv.org/html/2610.08969#S5.SS2.SSS0.Px1.p1.1)\.

## Appendix AProofs

See[1](https://arxiv.org/html/2610.08969#Thmtheorem1)

###### Proof\.

Lett∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}be arbitrary\. By the importance sampling argument:

𝔼𝜽∼p⁡\(𝜽∣𝒟t\)​\[φ​\(𝜽\)\]=∫φ⁡\(𝜽\)​p​\(𝜽∣𝒟t\)​𝑑𝜽=∫φ\(𝜽\)p⁡\(𝜽∣𝒟t\)p⁡\(𝜽∣𝒟t−1\)⏟=:w⁡\(𝜽\)p\(𝜽∣𝒟t−1\)d𝜽=𝔼𝜽∼p⁡\(𝜽∣𝒟t−1\)​\[w⁡\(𝜽\)​φ​\(𝜽\)\]\.\\begin\{split\}\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\[\\varphi\(\{\\bm\{\\theta\}\}\)\]&=\\int\\varphi\(\{\\bm\{\\theta\}\}\)\\,p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\\,d\{\\bm\{\\theta\}\}\\\\ &=\\int\\varphi\(\{\\bm\{\\theta\}\}\)\\underbrace\{\\frac\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\}\_\{=:w\(\{\\bm\{\\theta\}\}\)\}p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\\,d\{\\bm\{\\theta\}\}\\\\ &=\\mathbb\{E\}\_\{\{\\bm\{\\theta\}\}\\sim p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\[w\(\{\\bm\{\\theta\}\}\)\\,\\varphi\(\{\\bm\{\\theta\}\}\)\]\\,\.\\end\{split\}In particular, the importance weight isw⁡\(𝜽\)=p⁡\(𝜽∣𝒟t\)p⁡\(𝜽∣𝒟t−1\)w\(\{\\bm\{\\theta\}\}\)=\\frac\{p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\}\)\}\{p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\-1\}\)\}\.

Recall that𝒟t=𝒟t−1∪\{\(𝒙t,yt\)\}\{\\mathcal\{D\}\}\_\{t\}=\{\\mathcal\{D\}\}\_\{t\-1\}\\cup\\\{\(\{\\bm\{x\}\}\_\{t\},y\_\{t\}\)\\\}\. Thusp⁡\(𝜽∣𝒟t\)=p⁡\(𝜽∣𝒟t−1,𝒙t,yt\)p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\}\)=p\(\{\\bm\{\\theta\}\}\\mid\{\\mathcal\{D\}\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},y\_\{t\}\)\. Therefore, by Bayes’ rule we have that:

p⁡\(𝜽∣𝒟t\)=p⁡\(𝜽∣𝒟t−1,𝒙t,yt\)=p⁡\(yt∣𝒟t−1,𝒙t,𝜽\)​p​\(𝜽∣𝒟t−1,𝒙t\)p⁡\(yt∣𝒟t−1,𝒙t\)=p⁡\(yt∣𝒟t−1,𝒙t,𝜽\)​p​\(𝜽∣𝒟t−1\)p⁡\(yt∣𝒟t−1,𝒙t\)\.\\begin\{split\}p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)&=p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},y\_\{t\}\)\\\\ &=\\frac\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\\,p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\\\\ &=\\frac\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\\,p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\\,\.\\end\{split\}Therefore, plugging this intow⁡\(𝜽\)w\(\{\\bm\{\\theta\}\}\)we get:

w⁡\(𝜽\)=p⁡\(𝜽∣𝒟t\)p⁡\(𝜽∣𝒟t−1\)=p⁡\(yt∣𝒟t−1,𝒙t,𝜽\)​p​\(𝜽∣𝒟t−1\)p⁡\(yt∣𝒟t−1,𝒙t\)​1p⁡\(𝜽∣𝒟t−1\)=p⁡\(yt∣𝒟t−1,𝒙t,𝜽\)p⁡\(yt∣𝒟t−1,𝒙t\)∝p⁡\(yt∣𝒟t−1,𝒙t,𝜽\),\\begin\{split\}w\(\{\\bm\{\\theta\}\}\)&=\\frac\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\}\)\}\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\\\\ &=\\frac\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\\,p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\\frac\{1\}\{p\(\{\\bm\{\\theta\}\}\\mid\\mathcal\{D\}\_\{t\-1\}\)\}\\\\ &=\\frac\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\}\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\}\)\}\\\\ &\\propto\{p\(y\_\{t\}\\mid\\mathcal\{D\}\_\{t\-1\},\{\\bm\{x\}\}\_\{t\},\{\\bm\{\\theta\}\}\)\},\\end\{split\}where the last proportionality is because the denominator is constant w\.r\.t\.𝜽\{\\bm\{\\theta\}\}\. ∎

## Appendix BExperimental Details

### B\.1Real\-World Applications

#### HPLC\.

The dataset originates from the autonomous calibration of a robotic sampling sequence for direct\-inject HPLC analysis\([Roch et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib37)\)\. The authors report sessions of 100 autonomous experiments over approximately seven hours, corresponding to roughly four minutes per evaluation\.

#### Photo\-PCE10 and Photo\-WF3\.

These datasets report the photostability of photovoltaic blends\([Langner et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib36)\)\. The experimental time reported by the authors is 18 hours for a batch of samples\. As mentioned in the main text, we consider the problem in the sequential \(non\-batch\) optimization setting\.

### B\.2Deep Kernel Details

The deep kernel is implemented asσf⋅k⁡\(ℰ⁡\(𝒙\),ℰ⁡\(𝒙′\)\)\\sigma\_\{f\}\\cdot k\(\{\\mathcal\{E\}\}\(\{\\bm\{x\}\}\),\{\\mathcal\{E\}\}\(\{\\bm\{x\}\}^\{\\prime\}\)\), wherekkis a Matérn kernel with lengthscales fixed at 1,σf\\sigma\_\{f\}is a kernel outputscale, andℰ:ℝd→ℝ8\{\\mathcal\{E\}\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{8\}is a neural network with a single hidden layer of dimension 16 andtanh\\tanhactivations\. Taking into consideration the observation noise and constant mean, the deep kernel has a total of16​d\+15516d\+155hyperparameters\. To give NUTS a proper posterior to sample and prevent overfitting in MAP, we impose a zero\-mean normal prior with variance 0\.25 for all the neural network’s biases and4/Li​n4/L\_\{in\}for its weights, whereLi​nL\_\{in\}is the dimension of the layer’s input\.

For the experiments in[Section5\.2](https://arxiv.org/html/2610.08969#S5.SS2), the MAP deep kernel is optimized using 500 iterations of Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.08969#bib.bib42)\)per restart, with 4 restarts and a learning rate of 0\.01\. For Standard FBO and ELF\-BO, samples are obtained using NUTS with a diagonal mass matrix, performing 512 warmup steps and 4096 samples, resulting inM=256M=256after thinning by 16, with a max tree depth of 10\. For the scaling experiments in[Section6](https://arxiv.org/html/2610.08969#S6)we useM=128M=128and the Log\-EI is evaluated on batches of 32 candidates at a time\.

### B\.3Computational Resources

Experiments are implemented in Python using the BoTorch\([Balandat et al\., 2020](https://arxiv.org/html/2610.08969#bib.bib9)\)library version 0\.18\.1\. All synthetic benchmarks are executed on a single NVIDIA L40S, while real\-world experiments make use of an NVIDIA H100\.

### B\.4Further Experimental Details

A throwaway NUTS run happens before the first BO step, so that device start\-up is not considered in latency results\.

#### Acquisition function optimization

For the continuous synthetic functions, we use 10 restarts, with 512 raw samples\. For the discrete real\-world problems, every unused row is scored, 512 rows at a time\.

## Appendix CAdditional Results

Figure 8:Deep kernel on real\-world tasks\.Optimization curves of the deep kernel methods on Olympus datasets\.#### Deep Kernel

[Figure8](https://arxiv.org/html/2610.08969#A3.F8)presents the complete GAP curves for the deep kernel experiments described in[Section5\.2](https://arxiv.org/html/2610.08969#S5.SS2)\.