Efficient Conditioning Why Pseudo Observation Batch Bayesian Optimization Works When It Does not

arXiv cs.LG Papers

Summary

This paper provides a unified theoretical framework for pseudo observation batch Bayesian optimization, proving that Gaussian processes produce distinct batch points and that common methods like Constant Liar and Kriging Believer are instances of a single conditioning mechanism. It introduces the Structural Diversity Diagnostic (SDD) for testing surrogate compatibility and validates predictions across multiple benchmark functions and hyperparameter tuning.

arXiv:2605.18819v1 Announce Type: new Abstract: Constant Liar (CL), Kriging Believer (KB), and fantasy models are widely used for batch selection in parallel Bayesian Optimization, yet a unified theory explaining their effectiveness and conditions under which they fail has been lacking. We identify efficient conditioning as the key surrogate property the ability to update predictions in closed form when data is augmented. We prove that Gaussian Processes satisfy this requirement, producing provably distinct batch points with separation of order l, and that this holds for any acquisition function monotonically non decreasing in posterior uncertainty (EI, UCB, PI), with qualitatively similar behavior for Thompson Sampling. We unify CL, KB, and fantasy models as instances of a single conditioning mechanism differing only in the lie value distribution, and draw quantitative connections to Local Penalization (LP) and qualitative connections to Determinantal Point Processes (DPPs). To disentangle model structure from optimizer randomness, we introduce the Structural Diversity Diagnostic (SDD), a reusable methodology for testing surrogate compatibility. Experiments on Hartmann6D, Ackley 8D, Levy10D, and SVM hyperparameter tuning validate all theoretical predictions: CL or KBs implicit penalty matches or outperforms explicit LP greedy conditioning achieves convergence on par with joint qEI efficient conditioning extends to Multiquadric RBF networks; and parametric surrogates produce degenerate batches even when fully retrained (random forests), while neural networks regain diversity only at 15x the wall clock cost of GP conditioning. Robustness is confirmed across multiple initial datasets and under observation noise.
Original Article

Similar Articles

Pitfalls and Remedies for Multi-Task Bayesian Optimization

arXiv cs.LG

This paper identifies two structural mechanisms causing multi-task Gaussian processes to misestimate cross-task correlation in Bayesian optimization transfer learning, even for affinely related tasks. The authors propose three conservative remedies to mitigate these issues.

Adaptive KappaSharp: Condition-Number Shaping for Preferential Bayesian Optimization

arXiv cs.LG

This paper introduces KappaSharp, a method for condition-number shaping in Preferential Bayesian Optimization that corrects the ill-conditioned Hessian from isolated pairwise comparisons, showing up to +10.9% improvement over the standard PairedGP/EUBO baseline on 11 benchmarks including plasma medicine controller tuning.

Out-Of-The-Loop Multi-Fidelity Bayesian Optimization

arXiv cs.LG

The paper tackles multi-fidelity Bayesian optimization where the highest-fidelity function is too expensive to be part of the optimization loop, and proposes incorporating historical high-fidelity data with task descriptors. The method is demonstrated on synthetic functions, chemistry, and hyperparameter optimization tasks.