Coverage Aware Active Evaluation for Failure Discovery with Paired Systems

# Coverage Aware Active Evaluation for Failure Discovery with Paired Systems
Source: [https://arxiv.org/html/2608.13719](https://arxiv.org/html/2608.13719)
Anjali ParasharThanks:Corresponding authorAffiliation:Laboratory of Information & Decision Systems \(LIDS\), MIT, USARachel LuoAffiliation:NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit\.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia\.comApoorva SharmaAffiliation:NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit\.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia\.comSushant VeerAffiliation:NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit\.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia\.comEdward SchmerlingAffiliation:NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit\.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia\.comMingxin YuAffiliation:Laboratory of Information & Decision Systems \(LIDS\), MIT, USAChuchu FanAffiliation:Laboratory of Information & Decision Systems \(LIDS\), MIT, USAMarco PavoneAffiliation:NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit\.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia\.com

###### Abstract

Autonomous systems can fail in rare and heterogeneous ways, making real\-world failure discovery difficult under limited testing budgets\. Although cheaper proxies such as simulators, lower\-fidelity systems, or related policies can be sampled extensively to find failures, proxy failures often do not transfer to the real world due to sim\-to\-real and system\-to\-system gaps\. The key challenge is therefore to effectively leverage proxy system information for accurate prediction of severe target system failures\. We propose an adaptive failure discovery method that combines proxy evaluations with limited target system results to guide scenario selection for target system testing\. Our method learns a local predictor of target risk by correcting proxy failure signals using control\-variate\-inspired residual modeling\. To find failures that are both likely and diverse, we combine this predictor with a support\-aware mutual\-information objective that favors realistic, well\-supported regions while expanding coverage across failure modes\. Across autonomous driving, manipulation, and quadruped velocity\-tracking tasks, our method discovers up to 2x as many failures as random sampling and active\-learning baselines, including severe and diverse failures missed by competing methods\.

> Keywords:Testing & evaluation, Adaptive Experimental Design

## 1Introduction

Autonomous systems must be evaluated not only for average\-case performance, but also for rare, severe, and heterogeneous failures that arise under specific scenario conditions\. Such failures are especially important in safety\-critical domains, where a small number of unanticipated edge cases can lead to unsafe behavior after deployment\[[11](https://arxiv.org/html/2608.13719#bib.bib10),[12](https://arxiv.org/html/2608.13719#bib.bib6),[8](https://arxiv.org/html/2608.13719#bib.bib11),[7](https://arxiv.org/html/2608.13719#bib.bib12),[25](https://arxiv.org/html/2608.13719#bib.bib9)\]\. However, discovering these failures through direct evaluation of the target system is difficult, since real\-world or high\-fidelity closed\-loop tests can be expensive, time\-consuming, and resource\-constrained, and modern autonomous systems are often too complex to model analytically\[[18](https://arxiv.org/html/2608.13719#bib.bib15),[24](https://arxiv.org/html/2608.13719#bib.bib27)\]\. Cheap proxy systems, such as simulators, lower\-fidelity platforms, related policies, or alternative evaluation processes, provide a natural way to scale failure search, but also introduce a central challenge over accuracy of failure estimation\. Failures discovered in a proxy may reflect proxy\-specific artifacts rather than true weaknesses of the target system\. A scenario may appear critical because of simplified dynamics, fidelity mismatch, or distributional gaps, yet be benign on the target system; conversely, scenarios that appear safe in the proxy may still induce severe target\-system failures\. Thus, the goal is not merely to find failures cheaply, but to leverage proxy system carefully to accurately support the prediction of predictive of target\-system failures under a limited target\-evaluation budget\.

Existing approaches address only parts of this problem\. Simulation\-based testing can efficiently search low\-cost models but may not transfer to the target system, while target\-system active testing directly searches for critical scenarios but often does not exploit abundant proxy evaluations\[[10](https://arxiv.org/html/2608.13719#bib.bib5),[9](https://arxiv.org/html/2608.13719#bib.bib7),[20](https://arxiv.org/html/2608.13719#bib.bib13),[23](https://arxiv.org/html/2608.13719#bib.bib8),[2](https://arxiv.org/html/2608.13719#bib.bib25),[24](https://arxiv.org/html/2608.13719#bib.bib27),[18](https://arxiv.org/html/2608.13719#bib.bib15)\]\. Methods that combine proxy and target evaluations can improve estimation of aggregate target\-system metrics\[[17](https://arxiv.org/html/2608.13719#bib.bib26),[2](https://arxiv.org/html/2608.13719#bib.bib25)\], but these methods are often designed for Monte Carlo sampled data that can be ineffective for failure search\. Failure discovery requires a different objective: actively selecting scenarios that reveal many distinct target\-system failures\. This also requires estimation of scenario specific metrics as opposed to aggregate metrics proposed by existing works\.

![Refer to caption](https://arxiv.org/html/2608.13719v1/overview_new.png)Figure 1:Overview of approach\.Our method uses support\-aware Mutual Information \([Equation3](https://arxiv.org/html/2608.13719#S4.E3)\) to subsampleKM​IK\_\{MI\}scenariosxx, which are evaluated on the cheap proxy system to get metricsysy\_\{s\}, and a surrogate modelqθq\_\{\\theta\}trained using data evaluated on target system\. We subsequently optimizeμC​V​\(x\)\\mu\_\{CV\}\(x\)\([Equation2](https://arxiv.org/html/2608.13719#S4.E2)\) over these sub sampled candidates to select the next batch of evaluation points as likely failure scenarios\. The process is repeated until evaluation budgetBBis exhausted\.In this work, we study adaptive failure discovery using cheap proxy evaluations and limited target\-system tests\. Our goal is to discover a large and diverse set of target\-system failures under a fixed target\-evaluation budget, where diversity corresponds to covering distinct regions of the scenario space, operating conditions, or failure modes\. We propose an adaptive acquisition strategy that treats proxy evaluations as biased but informative signals\. Rather than treating proxy failures as ground truth, our method learns a local failure predictor that estimates target\-system risk by correcting proxy failure signals using limited target evaluations\. This predictor is inspired by control variate estimation\[[17](https://arxiv.org/html/2608.13719#bib.bib26)\], where proxy observations provide a low\-cost signal correlated with target behavior, while target evaluations correct for local proxy\-target mismatch\. The resulting estimate prioritizes scenarios that are likely to fail on the target system, rather than merely fail in the proxy\. Our contributions are as follows:

1. 1\.We formulate proxy\-guided failure discovery as an adaptive scenario selection problem, where the goal is to use limited target\-system evaluations to discover many distinct target failure modes\.
2. 2\.We propose a local, proxy\-corrected failure predictor that uses limited paired proxy\-target evaluations to identify scenarios likely to fail on the target system, rather than merely in the proxy evaluator\.
3. 3\.We introduce a support\-aware mutual\-information acquisition strategy that balances target\-failure likelihood with coverage of realistic and underexplored regions of the scenario space, encouraging acquisition of novel failure scenarios\.
4. 4\.We evaluate our method across autonomous driving, multi\-fidelity manipulation, and quadruped velocity\-tracking tasks, showing that it discovers more target\-system failures and reveals a broader range of critical failure modes than random sampling and active\-learning baselines\.

Across these domains, our method discovers upto twice as many target\-system failures than the strongest baseline under the same target\-evaluation budget, discovering several low and high severity failures, where the baselines completely miss discovery of high severity failures\. Our method also achieves highest diversity across all tasks and baselines, under a pairwise\-distance\-based coverage metric\. These results show that cheap proxy evaluations can substantially improve failure discovery when used to guide target\-system testing in an adaptive manner, rather than as direct substitutes for target\-system evaluation\.

## 2Related Work

#### Simulation\-based failure discovery\.

Simulation enables scalable, low\-cost testing of autonomous systems via adaptive sampling, stress testing, and failure discovery\[[10](https://arxiv.org/html/2608.13719#bib.bib5),[9](https://arxiv.org/html/2608.13719#bib.bib7)\]\. However, these methods are sensitive to sim\-to\-real gaps and model misspecification, so scenarios appearing safe in simulation may still fail on the real system\[[20](https://arxiv.org/html/2608.13719#bib.bib13),[23](https://arxiv.org/html/2608.13719#bib.bib8),[18](https://arxiv.org/html/2608.13719#bib.bib15)\]\. We address this by treating proxy evaluations as biased but informative signals, and using limited target evaluations to adaptively correct the proxy\-to\-target relationship during failure discovery\.

#### Sample\-efficient target\-system testing\.

A complementary line of work focuses on failure discovery directly on the target system using expert\-designed scenario sets, or surrogate based Bayesian optimization, or active learning under limited budgets\[[18](https://arxiv.org/html/2608.13719#bib.bib15),[21](https://arxiv.org/html/2608.13719#bib.bib24),[1](https://arxiv.org/html/2608.13719#bib.bib4)\]\. Surrogate based approaches can struggle to scale when scenarios are high\-dimensional or failure landscapes are complex\[[24](https://arxiv.org/html/2608.13719#bib.bib27),[21](https://arxiv.org/html/2608.13719#bib.bib24)\]\. Our method differs by not depending on restrictive model choices while modeling target\-system risk\.

#### Combining proxy and target evaluations\.

Recent works use Control variates and related variance\-reduction methods to use correlated proxy signals for improved estimation of aggregate real\-world metrics under limited target evaluations\[[17](https://arxiv.org/html/2608.13719#bib.bib26),[2](https://arxiv.org/html/2608.13719#bib.bib25)\]\. These methods are typically designed for Monte Carlo estimation of global quantities, which is inefficient for discovering rare failures\. Our goal instead is to actively select scenarios that reveal many target\-system failures, motivating an adaptive acquisition strategy that uses proxy\-target correction for both failure estimation and scenario selection\.

#### Bayesian experimental design and information\-based acquisition\.

Bayesian experimental design approaches select informative evaluations under limited budgets\[[22](https://arxiv.org/html/2608.13719#bib.bib14),[5](https://arxiv.org/html/2608.13719#bib.bib23)\], often via Mutual Information \(MI\)\[[4](https://arxiv.org/html/2608.13719#bib.bib22),[19](https://arxiv.org/html/2608.13719#bib.bib18)\]\. However, for failure discovery, uncertainty alone may target non\-critical scenarios, while exploitation alone may produce duplicate failures\. We therefore combine a support\-aware mutual\-information objective for exploring under\-covered regions with a proxy\-corrected risk estimate to prioritize severe failures\.

#### Positioning of this work\.

Our work connects these lines by combining cheap proxy evaluations, limited target\-system tests, and information\-based acquisition\. Unlike simulation\-only testing, we do not assume proxy failures transfer directly to the target\. Unlike target\-only active testing, we leverage proxy evaluations to guide search\. Unlike variance\-reduced estimation, our goal is not to estimate an aggregate metric but to select evaluations that reveal diverse failure modes\. Together, these components enable sample\-efficient discovery of distinct target\-system failures\.

## 3Problem Statement

We consider a black\-box dynamical system that maps scenario parametersx∈𝒳x\\in\\mathcal\{X\}to trajectory rollouts\. Our goal is to discover failures of thistargetsystem, which can be expensive to evaluate\. Failure is measured by a safety metricyr∈𝒴y\_\{r\}\\in\\mathcal\{Y\}, where, without loss of generality, larger values indicate less safe behavior\. We assume access to a cheaperproxysystem that provides low\-cost observationsysy\_\{s\}for the same scenarios\. Rollout stochasticity or environmental variations induce distributionsp⁡\(ys∣x\)p\(y\_\{s\}\\mid x\)andp⁡\(yr∣x\)p\(y\_\{r\}\\mid x\)for these systems\. Because proxy evaluations are inexpensive,p⁡\(ys∣x\)p\(y\_\{s\}\\mid x\)can be estimated by sampling, while estimatingp⁡\(yr∣x\)p\(y\_\{r\}\\mid x\)remains difficult due to high cost of evaluations \([Figure1](https://arxiv.org/html/2608.13719#S1.F1)\)\.

Our objective is to use paired proxy and target information to discover a large number of target system failures, spanning diverse scenarios or operating conditions, under a target evaluation budgetBB\. For scenarioxx, letμr​\(x\)≔𝔼p⁡\(yr∣x\)​\[yr\]\\mu\_\{r\}\(x\)\\coloneqq\\mathbb\{E\}\_\{p\(y\_\{r\}\\mid x\)\}\[y\_\{r\}\]\. We define failure asμr​\(x\)≥γ\\mu\_\{r\}\(x\)\\geq\\gamma, whereγ\\gammais a user\-specified severity threshold\. The goal is to identify a diverse set of failuresXB=\{xi∈𝒳∣μr​\(xi\)≥γ\}i=1K\.X\_\{B\}=\\\{x^\{i\}\\in\\mathcal\{X\}\\mid\\mu\_\{r\}\(x^\{i\}\)\\geq\\gamma\\\}\_\{i=1\}^\{K\}\.We promote diversity using mutual information \(MI\), which is commonly used to quantify epistemic uncertainty from limited evaluations\[[19](https://arxiv.org/html/2608.13719#bib.bib18),[4](https://arxiv.org/html/2608.13719#bib.bib22),[22](https://arxiv.org/html/2608.13719#bib.bib14),[5](https://arxiv.org/html/2608.13719#bib.bib23)\]\.

To maintain the evaluation budget, we formulate failure discovery as adaptive scenario sampling using Bayesian Experimental Design \(BED\)\[[22](https://arxiv.org/html/2608.13719#bib.bib14)\]\. At iterationk≤Bk\\leq B, given observed target evaluations𝒟k=\(xi,yri\)i=1k\\mathcal\{D\}\_\{k\}=\{\(x^\{i\},y\_\{r\}^\{i\}\)\}\_\{i=1\}^\{k\}, we select the next scenario by maximizing an acquisition functionα:𝒳→ℝ\\alpha:\\mathcal\{X\}\\to\\mathbb\{R\}asxk\+1=arg⁡maxx∈𝒳⁡α⁡\(x\)\.x\_\{k\+1\}=\\arg\\max\_\{x\\in\\mathcal\{X\}\}\\alpha\(x\)\.Our acquisition must balance two goals: selecting scenarios likely to satisfyμr​\(x\)≥γ\\mu\_\{r\}\(x\)\\geq\\gamma, and covering diverse operating conditions through high MI\. We next describe the design of this acquisition strategy\.

## 4Data acquisition strategy for meeting dual evaluation objectives

Because the target system is black\-box, directly estimatingμr\\mu\_\{r\}is difficult\. We can instead learn a surrogate,qθ​\(x\)q\_\{\\theta\}\(x\)from observed target evaluations𝒟k=\{\(xi,yri\)\}i=1k\\mathcal\{D\}\_\{k\}=\\\{\(x^\{i\},y\_\{r\}^\{i\}\)\\\}\_\{i=1\}^\{k\}to approximatep⁡\(yr\|x\)p\(y\_\{r\}\|x\), and use it to guide future scenario selection\. However, target\-only surrogates ignore the readily available proxy data\. To combine predictions fromqθ​\(x\)q\_\{\\theta\}\(x\)learnt using limited target evaluations with biased, noisy proxy signals for accurate failure prediction, we introduce a local control\-variate predictorμCV\\mu\_\{\\mathrm\{CV\}\}in[Section4\.1](https://arxiv.org/html/2608.13719#S4.SS1)\. To promote scenario diversity, we introduce a support\-aware MI termI\(Zx;Rx∣x,𝒟t\)I\(Z\_\{x\};R\_\{x\}\\mid x,\\mathcal\{D\}\_\{t\}\)in[Section4\.2](https://arxiv.org/html/2608.13719#S4.SS2), whereZxZ\_\{x\}captures the evaluated data support andRxR\_\{x\}denotes the potential support improvement from candidatexx\. Combining this diversity objective with the control\-variate failure predictor gives the high level objective for data acquisition:

α⁡\(x∣𝒟t\)=μCV​\(x\)⏟failure estimation\+I\(Zx;Rx∣x,𝒟t\)⏟scenario diversity\.\\alpha\(x\\mid\\mathcal\{D\}\_\{t\}\)=\\underbrace\{\\mu\_\{\\mathrm\{CV\}\}\(x\)\}\_\{\\text\{failure estimation\}\}\+\\underbrace\{I\(Z\_\{x\};R\_\{x\}\\mid x,\\mathcal\{D\}\_\{t\}\)\}\_\{\\text\{scenario diversity\}\}\.\(1\)The two objectives cumulatively represent candidates that are both likely target failures and informative for expanding support\. We defineμCV\\mu\_\{\\mathrm\{CV\}\}and the support\-aware MI term next\.

### 4\.1Combining target and proxy system information for failure estimation

We first describe the construction ofμC​V\\mu\_\{CV\}for failure estimation\. Consider a candidate scenariox∈𝒳x\\in\\mathcal\{X\}for target\-system testing, where the goal is to identify scenarios withμr​\(x\)≥γ\\mu\_\{r\}\(x\)\\geq\\gamma\. As discussed above, relying only on the surrogateqθq\_\{\\theta\}can be inefficient because it is learned from limited target data and ignores proxy evaluationsp⁡\(ys\)p\(y\_\{s\}\)\. We therefore propose a local, scenario\-specific mean estimatorμCV​\(x\)\\mu\_\{\\text\{CV\}\}\(x\)based on the Control Variate \(CV\) method\[[17](https://arxiv.org/html/2608.13719#bib.bib26)\]\. By incorporatingμC​V\\mu\_\{CV\}inα\\alpha, we enable a more accurate failure estimation at a givenxx, than can be achieved using justYrY\_\{r\}with limited samples, or, using a large number of proxy samplesYsY\_\{s\}\. This estimator uses proxy evaluationsp⁡\(ys\)p\(y\_\{s\}\)as additional information to improve the target mean estimate obtained fromqθq\_\{\\theta\}\. For a candidate scenarioxx, we construct a neighborhoodℬx\\mathcal\{B\}\_\{x\}and sampleNNscenariosxi∼ℬxx^\{i\}\\sim\\mathcal\{B\}\_\{x\}for surrogate prediction\. Here,ℬx\\mathcal\{B\}\_\{x\}can denote a categorical scenario type or a Euclidean ball in continuous scenario space, as detailed in[AppendixC](https://arxiv.org/html/2608.13719#A3)\. Within this region, we uniformly sample at mostN=n\+kN=n\+kdatapoints: paired evaluations\(xi,y^ri,ysi\)i=1n\(x^\{i\},\\hat\{y\}\_\{r\}^\{i\},y\_\{s\}^\{i\}\)\_\{i=1\}^\{n\}, wherey^r∼qθ​\(x\)\\hat\{y\}\_\{r\}\\sim q\_\{\\theta\}\(x\), and additional proxy evaluations\(xj,ysj\)j=1k\(x^\{j\},y\_\{s\}^\{j\}\)\_\{j=1\}^\{k\}\. Using this data, we approximateμr​\(x\)\\mu\_\{r\}\(x\)usingμC​V​\(x\)\\mu\_\{CV\}\(x\), a local mean estimate of the metric atxx, as:

μC​V​\(x\)=1n​∑i=1n\(y^ri−𝜷​ysi\)\+1k​∑j=1k𝜷​ysi,where​𝜷=\(kk\+n\)​Var​\(Ys\)−1​Cov​\(Yr,Ys\)\.\\mu\_\{CV\}\(x\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\\hat\{y\}\_\{r\}^\{i\}\-\\boldsymbol\{\\beta\}y\_\{s\}^\{i\}\)\+\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}\\boldsymbol\{\\beta\}y\_\{s\}^\{i\},~~\\text\{where\}~~\\boldsymbol\{\\beta\}=\\bigg\(\\frac\{k\}\{k\+n\}\\bigg\)\\text\{Var\}\(Y\_\{s\}\)^\{\-1\}\\text\{Cov\}\(Y\_\{r\},Y\_\{s\}\)\.\(2\)HereYs=\(ysj\)j=1n\+k,Yr=\(y^rj\)j=1nY\_\{s\}=\(y\_\{s\}^\{j\}\)\_\{j=1\}^\{n\+k\},Y\_\{r\}=\(\\hat\{y\}\_\{r\}^\{j\}\)\_\{j=1\}^\{n\}, andβ\\betaadjusts the mean estimated using proxy metrics based on the correlation between target and proxy distributionsqθ​\(x\)q\_\{\\theta\}\(x\)andp⁡\(ys\|x\)p\(y\_\{s\}\|x\)forx∈ℬxx\\in\\mathcal\{B\}\_\{x\}\. This results in estimatorμC​V\\mu\_\{CV\}that corresponds to minimum variance estimator of the failure statistics in the local region \(see[AppendixA](https://arxiv.org/html/2608.13719#A1)for details\)\.[Figure7](https://arxiv.org/html/2608.13719#A3.F7)shows the application of this approach for failure prediction on a 1D toy task\.

### 4\.2Formalizing scenario diversity for exploration

To encourage scenario diversity in adaptive data acquisition, we aim to prioritize scenarios away from observed scenarios\. To this end, we construct a support aware MI, such that maximizing MI identifies novel scenarios\. As a first step, we explicitly identify scenario space covered using𝒟t\\mathcal\{D\}\_\{t\}\. This can be done by clustering observed data inKKclusters in the scenario space\. For a an unseen potential candidatexx, letZx∈\[1,…,K,new\]Z\_\{x\}\\in\[1,\\dots,K,\\text\{new\}\]denote cluster assignment atxx, whereZx=newZ\_\{x\}=\\text\{new\}implies formation of new cluster by evaluating atxx, implying that the point is not within the covered scenario space\. Tied toxx, we also define a support discovery variableRx∈\{0,1\}R\_\{x\}\\in\\\{0,1\\\}, whereRx=1R\_\{x\}=1denotes thatxxreveals new support\. Thus, our goal of encouraging diversity in scenario can be defined as discovery of novel candidates far from support and those within support with uncertainty on cluster assignment\. Mathematically, this can be naturally captured by MI asI\(Zx;Rx\|𝒟t,x\)I\(Z\_\{x\};R\_\{x\}\|\\mathcal\{D\}\_\{t\},x\)in a principled manner, defined using entropyHHas:

I\(Zx;Rx∣Dt,x\)=H\(Zx\|𝒟t,x\)−H\(Zx\|Rx,𝒟t,x\)\.I\(Z\_\{x\};R\_\{x\}\\mid D\_\{t\},x\)=H\(Z\_\{x\}\|\\mathcal\{D\}\_\{t\},x\)\-H\(Z\_\{x\}\|R\_\{x\},\\mathcal\{D\}\_\{t\},x\)\.\(3\)
To enable a direct and tractable connection betweenRxR\_\{x\}andZxZ\_\{x\}, we model the distribution overRxR\_\{x\}asp⁡\(Rx\|x,𝒟t\)=∑k=1Kp⁡\(Rx\|Zx=k\)​p​\(Zx=k\|x,𝒟t\)p\(R\_\{x\}\|x,\\mathcal\{D\}\_\{t\}\)=\\sum\_\{k=1\}^\{K\}p\(R\_\{x\}\|Z\_\{x\}=k\)p\(Z\_\{x\}=k\|x,\\mathcal\{D\}\_\{t\}\)\. Here, priorp⁡\(Zx=k\|𝒟t,x\)p\(Z\_\{x\}=k\|\\mathcal\{D\}\_\{t\},x\)controls probability of assignment of a pointxxto clusterkk, and we define the likelihoodp⁡\(Rx=1\|Zx=k\)p\(R\_\{x\}=1\|Z\_\{x\}=k\)to be very low for scenarios well within one of thekkclusters, and set it to be high forZx=newZ\_\{x\}=\\text\{new\}\(see[AppendixD](https://arxiv.org/html/2608.13719#A4)for details\)\. Under these modeling choices, the MI term in[Equation3](https://arxiv.org/html/2608.13719#S4.E3)prioritizes scenarios for which cluster assignment is ambiguous, due to lying between cluster boundaries, and scenarios far from existing support\. Thus, evaluating on these points enables meaningful support expansion, meeting our goal of scenario diversity\.

Unlike GP\-based approaches such as\[[24](https://arxiv.org/html/2608.13719#bib.bib27)\], our method scales to high\-dimensional scenarios because it is agnostic to the surrogate architectureqθq\_\{\\theta\}and encodes exploration directly through support\. This enables scaling beyond evaluation budgets of only tens of samples\. In[Figure2](https://arxiv.org/html/2608.13719#S5.F2), we illustrate this behavior on a large\-scale, high\-dimensional dataset, where the method prioritizes unexplored regions under different supports induced by evaluated samples𝒟t\\mathcal\{D\}\_\{t\}\.

### 4\.3Batched data acquisition for diverse failure discovery

Using the two main components in[Equation1](https://arxiv.org/html/2608.13719#S4.E1), we can optimizeα\\alphafor scenario discovery in a continuous domain\. However, in our experimental validation, the scenario space𝒳\\mathcal\{X\}is a collection of discrete samples, and our objective is to getμC​V≥γ\\mu\_\{CV\}\\geq\\gamma\. Hence, we transform the optimization procedure into a severity adjusted data sampling rule with batched data acquisition ofbbscenarios at each iteration, for a budget ofB=T​bB=Tbsamples\. The batch sizebbis assumed to be user defined, and selected to be large enough to provide sufficient dataset for updatingqθq\_\{\\theta\}based on scenario dimension, and low enough that we do not surpass the budgetBB\. At each steptt, we select top\-KM​IK\_\{MI\}candidates that maximizeI\(Zx;Rx\|x,𝒟t\)I\(Z\_\{x\};R\_\{x\}\|x,\\mathcal\{D\}\_\{t\}\), followed by construction of the setΩt=\{x\|μC​V​\(x\)≥γ\}\\Omega\_\{t\}=\\\{x\|\\mu\_\{CV\}\(x\)\\geq\\gamma\\\}\. We selectbbcandidates fromΩt\\Omega\_\{t\}by clustering the samples intobbclusters and selecting candidate closest to cluster center for each, to further reinforce diversity within the batch\. We note that the final algorithm is not a complete equivalent of[Equation1](https://arxiv.org/html/2608.13719#S4.E1), but represents a constrained batched optimization equivalent instead, with the said constraints and batching procedures enabled for efficiency\.[Algorithm1](https://arxiv.org/html/2608.13719#alg1)summarizes the data acquisition procedure\.

Algorithm 1Adaptive scenario discovery using \([1](https://arxiv.org/html/2608.13719#S4.E1)\)\.1:Input:Scenario space

𝒳\\mathcal\{X\}, initial paired

𝒟0=\{xi,yir\}i=1N0\\mathcal\{D\}\_\{0\}=\\\{x^\{i\},y^\{r\}\_\{i\}\\\}\_\{i=1\}^\{N\_\{0\}\}
2:Initialize surrogate models

q⁡\(x\)q\(x\)trained using

𝒟0\\mathcal\{D\}\_\{0\}\.

3:for

t=0t=0to

T−1T\-1do

4:Select scenario

𝒳u=\{xi\|argmaxxi∈𝒳I\(Z;Rx\|xi,𝒟t\)\}i=1KM​I\\mathcal\{X\}\_\{u\}=\\\{x^\{i\}\|\\arg\\max\_\{x^\{i\}\\in\\mathcal\{X\}\}I\(Z;R\_\{x\}\|x^\{i\},\\mathcal\{D\}\_\{t\}\)\\\}\_\{i=1\}^\{K\_\{MI\}\}for

IIin[Equation3](https://arxiv.org/html/2608.13719#S4.E3),

5:Construct

Ωt=\{x∣μC​V​\(x\)≥γ\}\\Omega\_\{t\}=\\\{x\\mid\\mu\_\{CV\}\(x\)\\geq\\gamma\\\}for

μC​V\\mu\_\{CV\}in[Equation2](https://arxiv.org/html/2608.13719#S4.E2)\.

6:Select

\(xi\)i=1b\(x^\{i\}\)\_\{i=1\}^\{b\}for

xi∈Ωtx^\{i\}\\in\\Omega\_\{t\}, closest to cluster centers for

bbcluster on

Ωt\\Omega\_\{t\}\.

7:Collect

\(yri\)i=1b\(y\_\{r\}^\{i\}\)\_\{i=1\}^\{b\}for

\(xi\)i=1b\(x^\{i\}\)\_\{i=1\}^\{b\}using target system evaluation

8:Update dataset:

𝒟t\+1=𝒟t∪\{\(xi,yri\)\}i=1b\\mathcal\{D\}\_\{t\+1\}=\\mathcal\{D\}\_\{t\}\\cup\\\{\(x^\{i\},y^\{i\}\_\{r\}\)\\\}\_\{i=1\}^\{b\}
9:Update surrogate models

qθq\_\{\\theta\}, using

𝒟t\+1\\mathcal\{D\}\_\{t\+1\}\.

10:endfor

## 5Experimental validation

The central hypotheses of this paper are: \(1\) our approach can identify a concise set of scenarios that reveal failures across a range of severity levels and diverse operating conditions, and \(2\) it scales to larger evaluation budgets for high\-dimensional and complex scenarios\. We validate these hypotheses on four autonomous system tasks:nuPlan,SIMPLER,Quadruped, andKITTI, spanning autonomous driving, manipulation, and quadruped velocity tracking\. These tasks represent a broad range of scenario specifications, proxy\-target correlation structures, and evaluation costs\.

#### Tasks\.

InnuPlan, we evaluate motion\-planning behavior using driving scenarios from thenuPlandataset\[[14](https://arxiv.org/html/2608.13719#bib.bib21)\], with open\-loop playback as the proxy and closed\-loop rollouts as the target\. Scenarios are encoded in a 384\-dimensional space, and we use Time\-to\-Collision \(TTC\) as the failure metric\. InSIMPLER, we evaluate the language\-conditioned manipulation policyRT\-1x\[[6](https://arxiv.org/html/2608.13719#bib.bib19)\]on a grasp and lift task\[[15](https://arxiv.org/html/2608.13719#bib.bib20)\], where proxy and target correspond to two photorealistic simulator environments differing in visual specifications; scenarios are parameterized by a 19\-dimensional space encoding object positions, distractors, and camera pose\. InQuadruped, we evaluate aUnitree Go2quadruped\[[26](https://arxiv.org/html/2608.13719#bib.bib16)\]on a 3 dimensional command velocity tracking task, using aMuJoCo\-based simulation with an RL policy\[[27](https://arxiv.org/html/2608.13719#bib.bib17)\]as the proxy, demonstrating real\-world applicability of our approach\. InKITTI, we analyze perception failures in autonomous driving, using a continuous failure score from GT vehicle coverage on a real world driving dataset KITTI\[[13](https://arxiv.org/html/2608.13719#bib.bib2)\], paired with its synthetic clone, Virtual KITTI\[[3](https://arxiv.org/html/2608.13719#bib.bib3)\]\. These results demonstrate that our method utilizes proxy information effectively, and improves failure diversity\. We also provide a supplementary video showing our approach for failure discovery onQuadrupedtask\.

We use BNN for modelingqθq\_\{\\theta\}fornuPlan, SIMPLER, KITTIand MLP forQuadrupedto show that unlike baselines such as\[[24](https://arxiv.org/html/2608.13719#bib.bib27),[21](https://arxiv.org/html/2608.13719#bib.bib24)\], our method is not sensitive to the choice of surrogate model\.[AppendixI](https://arxiv.org/html/2608.13719#A9)provides additional implementation details for all tasks, and[AppendixH](https://arxiv.org/html/2608.13719#A8)shows visualization of exemplar failure scenarios discovered by our approach, in addition to dominant failure modes observed\.

#### Proxy system budget\.

We control proxy budget using a fixed proxy\-compute budgetBp=Np​cpB\_\{p\}=N\_\{p\}c\_\{p\}across tasks, adjustingNpN\_\{p\}inversely with per\-scenario proxy runtimecpc\_\{p\}\. Total measurable computational cost isC=Nt​ct\+Np​cp\+Ctraint\+Ctrainp,C=N\_\{t\}c\_\{t\}\+N\_\{p\}c\_\{p\}\+C\_\{\\mathrm\{train\}\}^\{t\}\+C\_\{\\mathrm\{train\}\}^\{p\},whereCtrainpC\_\{\\mathrm\{train\}\}^\{p\}\(surrogate training for proxy\) is optionally used when proxy data is scarce, andCtrainp,CtraintC\_\{\\mathrm\{train\}\}^\{p\},C\_\{\\mathrm\{train\}\}^\{t\}are much smaller than evaluation costs, thereforeC≈Nt​ct\+Np​cpC\\approx N\_\{t\}c\_\{t\}\+N\_\{p\}c\_\{p\}\. We consider tasks with a variety of proxy runtimes, for example, SIMPLER has highcpc\_\{p\},≈\\approx83s/eval vs\. 2s/eval for Quadruped\. This limits proxy data in SIMPLER to a total 110 samples \(50 initial and 60 sequential\) whereas in Quadruped, we are able to use 2,000 samples\. In practice, target testing also incurs resource, and potential\-damage costs that cannot be reduced to a scalar cost\. These task\-specific constraints motivate conservative target testing budgetNt=N0\+BN\_\{t\}=N\_\{0\}\+B, reflecting our focus on fixed target\-evaluation budgets\.

#### Baselines\.

We compare our approach against the following baselines:Random\(naïve MC sampling\);BAMS\[[24](https://arxiv.org/html/2608.13719#bib.bib27)\], which uses a GP surrogate with multi\-fidelity variance for acquisition;BNN\-C/GP\-C\[[18](https://arxiv.org/html/2608.13719#bib.bib15)\], which acquires worst\-case scenarios via clustering with a BNN or GP surrogate; andBNN\-CV, an ablation of our method without MI\-driven exploration\. We use GP forSIMPLER, and BNN fornuPlanforBNN\-C/GP\-C\.

#### Metrics\.

To validate our first hypothesis, we evaluate all methods using three metrics\.Cumulative average metricmeasures the average severity of discovered scenarios up to budgetkk, computed as1k​∑k=1Byrk\\frac\{1\}\{k\}\\sum\_\{k=1\}^\{B\}y\_\{r\}^\{k\}\.Cumulative coveragemeasures diversity among high\-failure scenariosℱtrue=\{x∈XB:yr≥γ\}\\mathcal\{F\}\_\{\\mathrm\{true\}\}=\\\{x\\in X\_\{B\}:y\_\{r\}\\geq\\gamma\\\}via their average pairwise distance,Ck​\(ℱtrue\)=1k​∑i,j∈\[1,…,k\]‖xi−xj‖22C\_\{k\}\(\\mathcal\{F\}\_\{\\mathrm\{true\}\}\)=\\frac\{1\}\{k\}\\sum\_\{i,j\\in\[1,\\dots,k\]\}\\left\\\|x^\{i\}\-x^\{j\}\\right\\\|^\{2\}\_\{2\}for1≤k≤B,1\\leq k\\leq B,where larger values indicate failures spread across the scenario space rather than collapsing onto a single region\.Positive samplesP=\|ℱtrue\|P=\\lvert\\mathcal\{F\}\_\{\\mathrm\{true\}\}\\rvertcounts discovered failures at varying severity thresholdsγ\\gamma, reported fornuPlanin[Table1](https://arxiv.org/html/2608.13719#S5.T1)\.

#### Ablations

We perform following ablations on our approach using thenuPlantask, \(1\) varying size of initial dataset \([AppendixE](https://arxiv.org/html/2608.13719#A5)\) and form of sampling to demonstrate the utility of support aware MI \([Figure2](https://arxiv.org/html/2608.13719#S5.F2)\), \(2\) ablations on formulation ofμC​V\\mu\_\{CV\}in[Equation2](https://arxiv.org/html/2608.13719#S4.E2)\([AppendixE](https://arxiv.org/html/2608.13719#A5)\)\. OnKITTI, we additionally demonstrate component wise ablation of our approach, isolating surrogate, proxy contribution, MI and clustering \(Table[3](https://arxiv.org/html/2608.13719#S5.T3)\)\. MI\-only \(MI\) yields high coverage but low failure discovery; no clustering \(Greedy\) reduces coverage due to lesser intra\-batch diversity and suffers from mode collapse; no MI with clustering \(−\-MI\-C\) under\-performs on high\-severity failure discovery and coverage; and no MI and no clustering \(−\-MI\-G\) further reduces coverage\. Thus, ablations show that MI promotes support coverage, CV improves failure targeting, and clustering preserves within\-batch diversity\.

Table 1:Positive Samples for different severity levels fornuPlan\(mean±\\pmstd across four seeds\)\.Taskγ\\gammaRandomBNN\-CBAMSBNN\-CVOurnuPlan0\.30\.23±\\pm0\.010\.25±\\pm0\.010\.28±\\pm0\.010\.27±\\pm0\.010\.33±\\pm0\.010\.20\.22±\\pm0\.000\.24±\\pm0\.020\.27±\\pm0\.010\.26±\\pm0\.010\.32±\\pm0\.010\.050\.21±\\pm0\.010\.21±\\pm0\.020\.25±\\pm0\.010\.22±\\pm0\.020\.29±\\pm0\.01

### 5\.1nuPlan

![Refer to caption](https://arxiv.org/html/2608.13719v1/mi_ablations_updated.png)Figure 2:MI driven exploration for different forms of data acquisition onnuPlantask\. 2D T\-SNE embeddings of scenario dataset showing high and low MI regions for 200 datapoints sampled from \(1\) small concentrated region of low failures, \(2\) MC sampling, \(3\) data recovered from a specific type of scenarios\. The plots show that our support aware MI strategy successfully identifies unobserved regions agnostic to the choice of sampling strategy\.[Figure3](https://arxiv.org/html/2608.13719#S5.F3)shows that our approach achieves lower cumulative average metric than all baselines exceptBNN\-CV, which repeatedly samples the same failures, hurting both diversity and failure count\. Our approach outperforms all baselines on both cumulative coverage and number of failures discovered, whereℱ\\mathcal\{F\}is defined by closed\-loop TTC<0\.3<0\.3\. In practice, we often lack control over how the initial scenario set is collected\.[Figure2](https://arxiv.org/html/2608.13719#S5.F2)visualizes MI values across the fullnuPlanscenario embedding space, showing that our support\-aware MI consistently prioritizes unexplored regions regardless of how prior samples were collected, providing a principled measure of scenario diversity\.

![Refer to caption](https://arxiv.org/html/2608.13719v1/nuplan_r1.png)Figure 3:Baseline comparisonfornuPlantask, \(avg±\\pmstd across four seeds\) forB=170B=170samples, with2020initial random samples for training surrogate, and 150 samples acquired in batches ofb=5b=5\.
### 5\.2Quadruped

[Figure4](https://arxiv.org/html/2608.13719#S5.F4)compares our approach againstBAMSandRandomon theQuadrupedtask, where failures are defined as scenariosx∈ℝ3x\\in\\mathbb\{R\}^\{3\}with average velocity tracking error exceeding0\.70\.7\(e\.g\., early stopping, large trajectory deviation; see supplementary video\)\. Our method consistently discovers more diverse failures than both baselines \([Figure8](https://arxiv.org/html/2608.13719#A6.F8)\), and identifies more failures across severity levels forγ=0\.7\\gamma=0\.7\([Table2](https://arxiv.org/html/2608.13719#S5.T2)\)\. In contrast,BAMSconcentrates near the threshold and often misses high\-severity failures\. We observe that diverse severity coverage also improves surrogate prediction\. Our method achieves mean accuracy of0\.800\.80over validation failure sets atγ∈\{0\.7,0\.8,0\.9,1\.0\}\\gamma\\in\\\{0\.7,0\.8,0\.9,1\.0\\\}, compared to0\.700\.70forBAMS, with0\.680\.68accuracy on high\-severity scenarios \(yr≥1\.0y\_\{r\}\\geq 1\.0\)\.[Figure9](https://arxiv.org/html/2608.13719#A6.F9)shows that our surrogate captures two symmetric failure modes over\(vy,wz\)\(v\_\{y\},w\_\{z\}\), consistent with the task structure, confirming that our coverage\-aware acquisition actively identifies distinct failures\.

![Refer to caption](https://arxiv.org/html/2608.13719v1/quadruped_r1.png)Figure 4:Baseline comparisonforQuadrupedtask, \(avg±\\pmstd, two seeds\) forB=30B=30samples, with1010initial random samples for training surrogate, and 20 samples acquired in batches ofb=5b=5\.Table 2:Positive samples for differentγ\\gammavalues inQuadrupedtask\.γ\\gamma0\.7 \(Our\)0\.8 \(Our\)1\.0 \(Our\)0\.7 \(BAMS\)0\.8 \(BAMS\)1\.0 \(BAMS\)P\.S\.0\.6±\\pm0\.10\.3±\\pm0\.10\.2±\\pm0\.10\.5±\\pm0\.10\.3±\\pm0\.20\.0±\\pm0\.0Table 3:KITTI component ablationFullGreedyβ=0​\(n\+k\)\\beta=0\(n\+k\)β=0\\beta\{=\}0MI−\-MI\-C−\-MI\-GUtility0\.380\.380\.360\.360\.320\.40\.41P\.S\.\-0\.50\.240\.220\.140\.170\.110\.300\.32P\.S\.\-0\.70\.030\.010\.020\.010\.020\.020\.01Cov\.0\.780\.580\.790\.720\.910\.60\.48
### 5\.3SIMPLER

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/true_true.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/fail_fail.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/true_fail.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/true_fail_2.png)

Figure 5:Examples of grasping failures forSIMPLERtask\.Top\- scenarios for which both proxy and target systems succeed \(left\) and fail \(right\) to grasp the can\. Bottom\- two examples for which the proxy system succeeds and target system fails to grasp the can, discovered by our approach\.[Figure10](https://arxiv.org/html/2608.13719#A7.F10)shows baseline comparison, withBNN\-CVperforming comparable toRandomsampling\. This is due to the fact that the failure metric of this task is sparsely defined as success \(yr=0\.0y\_\{r\}=0\.0\) or failure \(ys=1\.0y\_\{s\}=1\.0\), so the lack of exploration leads to heavy dependence on the failures present in initial dataset for accurately learning the surrogate model\. From the collected failure scenarios, we observe that perceptually challenging conditions with visual clutter are the dominant driver of failure all seeds\. We discover two dominant modes for camera positioning to the right and left of reference positions, each corresponding to object placement to the left and far from the robot\. The average proxy success rate is20%20\\%showing reasonable sim to real gap\.[Figure5](https://arxiv.org/html/2608.13719#S5.F5)\(bottom\) shows failure scenarios corresponding to this gap with success in proxy system, and failure in target system\.

### 5\.4KITTI

We use this task to highlight the merit of our approach with combined sampling against target only sampling, and quantify the contribution of proxy assisted evaluation\. Specifically, we test whether proxy information improves acquisition over surrogate\-only sampling, usingβ=0,\(n\+k\)\\beta=0,\(n\+k\)\(Table[3](https://arxiv.org/html/2608.13719#S5.T3)\), where the surrogate receives alln\+kn\+ksamples, and proxy andqθq\_\{\\theta\}sampling have the same cost; therefore any gains reflect proxy information, not reduced MC error\. Importantly, the proxy provides additional system information, not a cheaper substitute for MC samples fromqθq\_\{\\theta\}\. At equal sample count, proxy\-assisted acquisition improves failure discovery and utility, preserving coverage, achieving nearly70%70\\%higher coverage of meaningful failure modes, and increases surrogate failure recall rate from0\.180\.18to0\.490\.49\.

We also compare withβ=0\\beta=0\(nnsurrogate samples\)\. On nuPlan, weak proxy–surrogate correlation in failure regions \(avg\.ρ≈0\.15\\rho\\approx 0\.15\) drivesβ→0\\beta\\to 0, yielding small gains over no proxy ablation\. On KITTI,ρ≈0\.5\\rho\\approx 0\.5, therefore gains in P\.S\. and utility are clearer in Table[3](https://arxiv.org/html/2608.13719#S5.T3)\. It can be shown that proxy information can reduce target prediction risk by capturing target variation hidden by𝒳\\mathcal\{X\}, whereas additionalqθq\_\{\\theta\}samples only reduce variance\. This benefit increases with proxy\-target correlation\. Thus,β\\beta\-weighted proxy guidance exposes failure regions missed by surrogate\-only sampling and improves the learned target surrogate\.

## 6Discussion & Conclusion

Our approach scales gracefully with evaluation budget and scenario complexity, validating our central hypothesis\. Baseline comparisons underscore the importance of each component\.BAMSdegrades at170170evaluations innuPlanas GP surrogates struggle to scale; a known limitation of GPs noted in\[[18](https://arxiv.org/html/2608.13719#bib.bib15),[24](https://arxiv.org/html/2608.13719#bib.bib27),[21](https://arxiv.org/html/2608.13719#bib.bib24)\]\.BNN\-CV’s myopic acquisition collapses onto a narrow failure region inSIMPLER, where the sparse failure metric demands exploration, andGP\-C’s assumption that all proxy failures transfer to the target causes it to miss unpaired failures, despite focusing on diversity of scenarios\. Our work shows that proxy corrected risk estimation and scenario diversity are intertwined\. Lack of diversity leads to redundant failures, and without accurate risk estimates, evaluations are wasted\. Coupling both components is essential for sample efficient discovery of distinct, severe target\-system failures\.

## 7Limitations

Our approach assumes a scenario representation in which locality, support, and diversity are meaningful\. While such representations are available in our experiments, performance may depend on how well they capture behaviorally relevant differences between scenarios\. Like other black\-box failure discovery methods, our approach also requires target\-system evaluations; larger or more complex scenario spaces may require larger budgets to learn accurate target\-risk surrogates\. Our method is therefore best viewed as a way to use limited target evaluations more effectively by combining proxy guidance, local correction, and diversity\-aware acquisition, with future work exploring learned representations and stronger transfer across related systems\.

## References

- \[1\]A\. Anwar, R\. Gupta, Z\. Merchant, S\. Ghosh, W\. Neiswanger, and J\. Thomason\(2025\)Efficient evaluation of multi\-task robot policies with active experiment selection\.arXiv preprint arXiv:2502\.09829\.Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]A\. Badithela, D\. Snyder, L\. Zha, J\. Mikhail, M\. O’Kelly, A\. Dixit, and A\. Majumdar\(2025\)Reliable and scalable robot policy evaluation with imperfect simulators\.arXiv preprint arXiv:2510\.04354\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px3.p1.1)\.
- \[3\]Y\. Cabon, N\. Murray, and M\. Humenberger\(2020\)Virtual KITTI 2\.arXiv preprint arXiv:2001\.10773\.Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[4\]X\. Cao and I\. W\. Tsang\(2021\)Bayesian active learning by disagreements: a geometric perspective\.arXiv preprint arXiv:2105\.02543\.Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.13719#S3.p2.1)\.
- \[5\]K\. Chaloner and I\. Verdinelli\(1995\)Bayesian experimental design: A review\.Statistical science,pp\. 273–304\.Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.13719#S3.p2.1)\.
- \[6\]O\. X\. Collaboration, A\. O’Neill, A\. Rehman, A\. Gupta, A\. Maddukuri, A\. Gupta, A\. Padalkar, A\. Lee, A\. Pooley, A\. Gupta, A\. Mandlekar, A\. Jain, A\. Tung, A\. Bewley, A\. Herzog, A\. Irpan, A\. Khazatsky, A\. Rai, A\. Gupta, A\. Wang, A\. Kolobov, A\. Singh, A\. Garg, A\. Kembhavi, A\. Xie, A\. Brohan, A\. Raffin, A\. Sharma, A\. Yavary, A\. Jain, A\. Balakrishna, A\. Wahid, B\. Burgess\-Limerick, B\. Kim, B\. Schölkopf, B\. Wulfe, B\. Ichter, C\. Lu, C\. Xu, C\. Le, C\. Finn, C\. Wang, C\. Xu, C\. Chi, C\. Huang, C\. Chan, C\. Agia, C\. Pan, C\. Fu, C\. Devin, D\. Xu, D\. Morton, D\. Driess, D\. Chen, D\. Pathak, D\. Shah, D\. Büchler, D\. Jayaraman, D\. Kalashnikov, D\. Sadigh, E\. Johns, E\. Foster, F\. Liu, F\. Ceola, F\. Xia, F\. Zhao, F\. V\. Frujeri, F\. Stulp, G\. Zhou, G\. S\. Sukhatme, G\. Salhotra, G\. Yan, G\. Feng, G\. Schiavi, G\. Berseth, G\. Kahn, G\. Yang, G\. Wang, H\. Su, H\. Fang, H\. Shi, H\. Bao, H\. B\. Amor, H\. I\. Christensen, H\. Furuta, H\. Bharadhwaj, H\. Walke, H\. Fang, H\. Ha, I\. Mordatch, I\. Radosavovic, I\. Leal, J\. Liang, J\. Abou\-Chakra, J\. Kim, J\. Drake, J\. Peters, J\. Schneider, J\. Hsu, J\. Vakil, J\. Bohg, J\. Bingham, J\. Wu, J\. Gao, J\. Hu, J\. Wu, J\. Wu, J\. Sun, J\. Luo, J\. Gu, J\. Tan, J\. Oh, J\. Wu, J\. Lu, J\. Yang, J\. Malik, J\. Silvério, J\. Hejna, J\. Booher, J\. Tompson, J\. Yang, J\. Salvador, J\. J\. Lim, J\. Han, K\. Wang, K\. Rao, K\. Pertsch, K\. Hausman, K\. Go, K\. Gopalakrishnan, K\. Goldberg, K\. Byrne, K\. Oslund, K\. Kawaharazuka, K\. Black, K\. Lin, K\. Zhang, K\. Ehsani, K\. Lekkala, K\. Ellis, K\. Rana, K\. Srinivasan, K\. Fang, K\. P\. Singh, K\. Zeng, K\. Hatch, K\. Hsu, L\. Itti, L\. Y\. Chen, L\. Pinto, L\. Fei\-Fei, L\. Tan, L\. "\. Fan, L\. Ott, L\. Lee, L\. Weihs, M\. Chen, M\. Lepert, M\. Memmel, M\. Tomizuka, M\. Itkina, M\. G\. Castro, M\. Spero, M\. Du, M\. Ahn, M\. C\. Yip, M\. Zhang, M\. Ding, M\. Heo, M\. K\. Srirama, M\. Sharma, M\. J\. Kim, M\. Z\. Irshad, N\. Kanazawa, N\. Hansen, N\. Heess, N\. J\. Joshi, N\. Suenderhauf, N\. Liu, N\. D\. Palo, N\. M\. M\. Shafiullah, O\. Mees, O\. Kroemer, O\. Bastani, P\. R\. Sanketi, P\. "\. Miller, P\. Yin, P\. Wohlhart, P\. Xu, P\. D\. Fagan, P\. Mitrano, P\. Sermanet, P\. Abbeel, P\. Sundaresan, Q\. Chen, Q\. Vuong, R\. Rafailov, R\. Tian, R\. Doshi, R\. Mart’in\-Mart’in, R\. Baijal, R\. Scalise, R\. Hendrix, R\. Lin, R\. Qian, R\. Zhang, R\. Mendonca, R\. Shah, R\. Hoque, R\. Julian, S\. Bustamante, S\. Kirmani, S\. Levine, S\. Lin, S\. Moore, S\. Bahl, S\. Dass, S\. Sonawani, S\. Tulsiani, S\. Song, S\. Xu, S\. Haldar, S\. Karamcheti, S\. Adebola, S\. Guist, S\. Nasiriany, S\. Schaal, S\. Welker, S\. Tian, S\. Ramamoorthy, S\. Dasari, S\. Belkhale, S\. Park, S\. Nair, S\. Mirchandani, T\. Osa, T\. Gupta, T\. Harada, T\. Matsushima, T\. Xiao, T\. Kollar, T\. Yu, T\. Ding, T\. Davchev, T\. Z\. Zhao, T\. Armstrong, T\. Darrell, T\. Chung, V\. Jain, V\. Kumar, V\. Vanhoucke, V\. Guizilini, W\. Zhan, W\. Zhou, W\. Burgard, X\. Chen, X\. Chen, X\. Wang, X\. Zhu, X\. Geng, X\. Liu, X\. Liangwei, X\. Li, Y\. Pang, Y\. Lu, Y\. J\. Ma, Y\. Kim, Y\. Chebotar, Y\. Zhou, Y\. Zhu, Y\. Wu, Y\. Xu, Y\. Wang, Y\. Bisk, Y\. Dou, Y\. Cho, Y\. Lee, Y\. Cui, Y\. Cao, Y\. Wu, Y\. Tang, Y\. Zhu, Y\. Zhang, Y\. Jiang, Y\. Li, Y\. Li, Y\. Iwasawa, Y\. Matsuo, Z\. Ma, Z\. Xu, Z\. J\. Cui, Z\. Zhang, Z\. Fu, and Z\. Lin\(2023\)Open X\-Embodiment: robotic learning datasets and RT\-X models\.Note:[https://arxiv\.org/abs/2310\.08864](https://arxiv.org/abs/2310.08864)Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[7\]A\. Corso, P\. Du, K\. Driggs\-Campbell, and M\. J\. Kochenderfer\(2019\)Adaptive stress testing with reward augmentation for autonomous vehicle validatio\.In2019 IEEE Intelligent Transportation Systems Conference \(ITSC\),pp\. 163–168\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1)\.
- \[8\]A\. Corso, R\. Lee, and M\. J\. Kochenderfer\(2020\)Scalable autonomous vehicle safety validation through dynamic programming and scene decomposition\.In2020 IEEE 23rd International Conference on Intelligent Transportation Systems \(ITSC\),pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1)\.
- \[9\]C\. Dawson and C\. Fan\(2022\)Robust counterexample\-guided optimization for planning from differentiable temporal logic\.InIEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 7205–7212\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]C\. Dawson and C\. Fan\(2023\)A Bayesian approach to breaking things: efficiently predicting and repairing failure modes via sampling\.In7th Annual Conference on Robot Learning,External Links:[Link](https://openreview.net/forum?id=fNLBmtyBiC)Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]T\. Dreossi, T\. Dang, A\. Donzé, J\. Kapinski, X\. Jin, and J\. V\. Deshmukh\(2015\)Efficient guiding strategies for testing of temporal properties of hybrid systems\.InNASA Formal Methods: 7th International Symposium, NFM 2015, Pasadena, CA, USA, April 27\-29, 2015, Proceedings 7,pp\. 127–142\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1)\.
- \[12\]J\. M\. Esposito, J\. Kim, and V\. Kumar\(2005\)Adaptive RRTs for validating hybrid robotic control systems\.InAlgorithmic foundations of robotics vi,pp\. 107–121\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1)\.
- \[13\]A\. Geiger, P\. Lenz, and R\. Urtasun\(2012\)Are we ready for autonomous driving? The KITTI vision benchmark suite\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 3354–3361\.Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[14\]N\. Karnchanachari, D\. Geromichalos, K\. S\. Tan, N\. Li, C\. Eriksen, S\. Yaghoubi, N\. Mehdipour, G\. Bernasconi, W\. K\. Fong, Y\. Guo,et al\.\(2024\)Towards learning\-based planning: the nuplan benchmark for real\-world autonomous driving\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 629–636\.Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[15\]X\. Li, K\. Hsu, J\. Gu, K\. Pertsch, O\. Mees, H\. R\. Walke, C\. Fu, I\. Lunawat, I\. Sieh, S\. Kirmani, S\. Levine, J\. Wu, C\. Finn, H\. Su, Q\. Vuong, and T\. Xiao\(2024\)Evaluating real\-world robot manipulation policies in simulation\.arXiv preprint arXiv:2405\.05941\.Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[16\]Y\. Li, A\. Parashar, E\. Zhou, and C\. Fan\(2026\)Pragmatic curiosity: a hybrid learning\-optimization paradigm via active inference\.arXiv preprint arXiv:2602\.06104\.Cited by:[Appendix B](https://arxiv.org/html/2608.13719#A2.p1.1)\.
- \[17\]R\. Luo, H\. Yang, M\. Watson, A\. Sharma, S\. Veer, E\. Schmerling, and M\. PavoneLeveraging correlation across test platforms for variance\-reduced metric estimation\.In9th Annual Conference on Robot Learning,Cited by:[Appendix A](https://arxiv.org/html/2608.13719#A1.p1.1),[Appendix C](https://arxiv.org/html/2608.13719#A3.p2.1),[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§1](https://arxiv.org/html/2608.13719#S1.p3.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.13719#S4.SS1.p1.1)\.
- \[18\]A\. Parashar, K\. Garg, J\. Zhang, and C\. FanFailure prediction from few expert demonstrations\.InNeurIPS 2024 Workshop on Bayesian Decision\-making and Uncertainty,Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1),[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.13719#S6.p1.1)\.
- \[19\]A\. Parashar, Y\. Li, E\. Y\. Yu, F\. Chen, J\. Neidhoefer, D\. Upadhyay, and C\. Fan\(2026\)SEED\-set: scalable evolving experimental design for system\-level ethical testing\.arXiv preprint arXiv:2603\.01630\.Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.13719#S3.p2.1)\.
- \[20\]A\. Parashar, J\. Yin, C\. Dawson, P\. Tsiotras, and C\. Fan\(2024\)Learning\-based bayesian inference for testing of autonomous systems\.IEEE Robotics and Automation Letters\(\),pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2024.3455782)Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]A\. Parashar, J\. Zhang, Y\. Li, and C\. Fan\(2025\)Cost\-aware discovery of contextual failures using bayesian active learning\.In9th Annual Conference on Robot Learning,Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p2.1),[§6](https://arxiv.org/html/2608.13719#S6.p1.1)\.
- \[22\]T\. Rainforth, A\. Foster, D\. R\. Ivanova, and F\. Bickford Smith\(2024\)Modern bayesian experimental design\.Statistical Science39\(1\),pp\. 100–114\.Cited by:[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.13719#S3.p2.1),[§3](https://arxiv.org/html/2608.13719#S3.p3.1)\.
- \[23\]A\. Z\. Ren, H\. Dai, B\. Burchfiel, and A\. Majumdar\(2023\)AdaptSim: task\-driven simulation adaptation for sim\-to\-real transfer\.InConference on Robot Learning,pp\. 3434–3452\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]A\. Sinha, P\. Nikdel, S\. Paul, and S\. Whiteson\(2025\)Rate\-informed discovery via bayesian adaptive multifidelity sampling\.InConference on Robot Learning,pp\. 2579–2598\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1),[§1](https://arxiv.org/html/2608.13719#S1.p2.1),[§2](https://arxiv.org/html/2608.13719#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.13719#S4.SS2.p3.1),[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.13719#S6.p1.1)\.
- \[25\]A\. Sinha, M\. O’Kelly, R\. Tedrake, and J\. C\. Duchi\(2020\)Neural bridge sampling for evaluating safety\-critical autonomous systems\.Advances in Neural Information Processing Systems33,pp\. 6402–6416\.Cited by:[§1](https://arxiv.org/html/2608.13719#S1.p1.1)\.
- \[26\]Unitree Robotics\(2026\)Unitree Go2 Pro Quadruped Robot\.Note:[https://www\.unitree\.com/go2](https://www.unitree.com/go2)Quadruped robot platform\. Accessed: 2026\-05\-28Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.
- \[27\]Unitree Robotics\(2026\)Unitree RL Mjlab\.Note:[https://github\.com/unitreerobotics/unitree\_rl\_mjlab](https://github.com/unitreerobotics/unitree_rl_mjlab)Reinforcement learning implementation for Unitree robots based on MuJoCo\. Accessed: 2026\-05\-28Cited by:[§5](https://arxiv.org/html/2608.13719#S5.SS0.SSS0.Px1.p1.1)\.

## Contents

![Refer to caption](https://arxiv.org/html/2608.13719v1/overview_revised.png)Figure 6:Overview of approach\.Our method uses support\-aware Mutual Information \([Equation3](https://arxiv.org/html/2608.13719#S4.E3)\) to subsampleKM​IK\_\{MI\}scenariosxx, which are evaluated on the cheap proxy system to get metricsysy\_\{s\}, and a surrogate modelqθq\_\{\\theta\}trained using data evaluated on target system\. We subsequently optimizeμC​V​\(x\)\\mu\_\{CV\}\(x\)\([Equation2](https://arxiv.org/html/2608.13719#S4.E2)\)over these sub sampled candidates to select the next batch of evaluation points as likely failure scenarios\. The process is repeated until evaluation budgetBBis exhausted\. Figure on the left shows the surrogate model learnt after 40 samples of evaluation for the target system shown\.
## Appendix ALocal minimum\-variance property of the control\-variate estimator

We show that the estimator in[Equation2](https://arxiv.org/html/2608.13719#S4.E2)is the minimum\-variance estimator within the class of local affine control\-variate estimators\. Specifically, our method does not reduce surrogate bias, but uses proxy system information in a meaningful way to accelerate efficiency of failure discovery\. Our proof follows proof of Theorem 1 from\[[17](https://arxiv.org/html/2608.13719#bib.bib26)\]\. Fix a candidate scenariox∈𝒳x\\in\\mathcal\{X\}, and letBx⊆𝒳B\_\{x\}\\subseteq\\mathcal\{X\}denote the local neighborhood used to estimate the target failure statistic aroundxx\. LetYr∼p⁡\(yr∣x′\)Y\_\{r\}\\sim p\(y\_\{r\}\\mid x^\{\\prime\}\),Ys∼p⁡\(ys∣x′\)Y\_\{s\}\\sim p\(y\_\{s\}\\mid x^\{\\prime\}\), andx′∼Bxx^\{\\prime\}\\sim B\_\{x\}\. In practice, the target samplesYrY\_\{r\}are replaced by surrogate predictionsY^r∼qθ​\(x′\)\\hat\{Y\}\_\{r\}\\sim q\_\{\\theta\}\(x^\{\\prime\}\), so the following result is interpreted conditional on the learned surrogateqθq\_\{\\theta\}\.

Suppose we havennpaired local samples

\{\(y^ri,ysi\)\}i=1n,\\\{\(\\hat\{y\}\_\{r\}^\{i\},y\_\{s\}^\{i\}\)\\\}\_\{i=1\}^\{n\},\(4\)andkkadditional proxy\-only samples

\{ysj\}j=n\+1n\+k\.\\\{y\_\{s\}^\{j\}\\\}\_\{j=n\+1\}^\{n\+k\}\.\(5\)Consider the class of local affine control\-variate estimators

μCV​\(β\)=μC​V​\(x\)=1n​∑i=1n\(y^ri−𝜷​ysi\)\+1k​∑j=1kβ​ysi\\mu\_\{\\mathrm\{CV\}\}\(\\beta\)=\\mu\_\{CV\}\(x\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\(\\hat\{y\}\_\{r\}^\{i\}\-\\boldsymbol\{\\beta\}y\_\{s\}^\{i\}\)\+\\frac\{1\}\{k\}\\sum\_\{j=1\}^\{k\}\\beta y\_\{s\}^\{i\}\(6\)Assuming that the paired and proxy\-only samples are drawn independently from the same local neighborhood distribution overBxB\_\{x\}, this estimator is unbiased for the local target statistic:

𝔼Bx​\[μ^CV​\(β\)\]\\displaystyle\\mathbb\{E\}\_\{B\_\{x\}\}\\left\[\\hat\{\\mu\}\_\{\\mathrm\{CV\}\}\(\\beta\)\\right\]=𝔼Bx​\[Yr−β⊤​Ys\]\+𝔼Bx​\[β⊤​Ys\]\\displaystyle=\\mathbb\{E\}\_\{B\_\{x\}\}\\left\[Y\_\{r\}\-\\beta^\{\\top\}Y\_\{s\}\\right\]\+\\mathbb\{E\}\_\{B\_\{x\}\}\\left\[\\beta^\{\\top\}Y\_\{s\}\\right\]\(7\)=𝔼Bx​\[Yr\]\.\\displaystyle=\\mathbb\{E\}\_\{B\_\{x\}\}\[Y\_\{r\}\]\.\(8\)Thus, the choice ofβ\\betaaffects the variance but not the expectation\.

For a fixedβ\\beta, independence of the paired and proxy\-only sample sets gives

Var⁡\(μ^CV​\(β\)\)=1n​Var​\(Yr−β⊤​Ys\)\+1k​Var​\(β⊤​Ys\)\.\\mathrm\{Var\}\\left\(\\hat\{\\mu\}\_\{\\mathrm\{CV\}\}\(\\beta\)\\right\)=\\frac\{1\}\{n\}\\mathrm\{Var\}\\left\(Y\_\{r\}\-\\beta^\{\\top\}Y\_\{s\}\\right\)\+\\frac\{1\}\{k\}\\mathrm\{Var\}\\left\(\\beta^\{\\top\}Y\_\{s\}\\right\)\.\(9\)Expanding the two variance terms, we obtain

Var​\(μCV​\(β\)\)\\displaystyle\\mathrm\{Var\}\\left\(\\mu\_\{\\mathrm\{CV\}\}\(\\beta\)\\right\)=1n​Var​\(Yr\)−2n​β⊤​Cov​\(Ys,Yr\)\+1n​β⊤​Var​\(Ys\)​β\+1k​β⊤​Var​\(Ys\)​β\\displaystyle=\\frac\{1\}\{n\}\\mathrm\{Var\}\(Y\_\{r\}\)\-\\frac\{2\}\{n\}\\beta^\{\\top\}\\mathrm\{Cov\}\(Y\_\{s\},Y\_\{r\}\)\+\\frac\{1\}\{n\}\\beta^\{\\top\}\\mathrm\{Var\}\(Y\_\{s\}\)\\beta\+\\frac\{1\}\{k\}\\beta^\{\\top\}\\mathrm\{Var\}\(Y\_\{s\}\)\\beta\(10\)=1n​Var​\(Yr\)−2n​β⊤​Cov​\(Ys,Yr\)\+n\+kn​k​β⊤​Var​\(Ys\)​β\.\\displaystyle=\\frac\{1\}\{n\}\\mathrm\{Var\}\(Y\_\{r\}\)\-\\frac\{2\}\{n\}\\beta^\{\\top\}\\mathrm\{Cov\}\(Y\_\{s\},Y\_\{r\}\)\+\\frac\{n\+k\}\{nk\}\\beta^\{\\top\}\\mathrm\{Var\}\(Y\_\{s\}\)\\beta\.\(11\)This is a convex quadratic function ofβ\\betawheneverVar⁡\(Ys\)\\mathrm\{Var\}\(Y\_\{s\}\)is positive semidefinite, and is strictly convex whenVar⁡\(Ys\)\\mathrm\{Var\}\(Y\_\{s\}\)is positive definite\. Differentiating[Equation11](https://arxiv.org/html/2608.13719#A1.E11)with respect toβ\\betaand setting the derivative equal to zero gives

−2n​Cov​\(Ys,Yr\)\+2​\(n\+k\)n​k​Var​\(Ys\)​β=0\.\-\\frac\{2\}\{n\}\\mathrm\{Cov\}\(Y\_\{s\},Y\_\{r\}\)\+\\frac\{2\(n\+k\)\}\{nk\}\\mathrm\{Var\}\(Y\_\{s\}\)\\beta=0\.\(12\)Therefore, the variance\-minimizing coefficient is

β⋆=kn\+k​Var​\(Ys\)−1​Cov​\(Ys,Yr\)\.\\beta^\{\\star\}=\\frac\{k\}\{n\+k\}\\mathrm\{Var\}\(Y\_\{s\}\)^\{\-1\}\\mathrm\{Cov\}\(Y\_\{s\},Y\_\{r\}\)\.\(13\)Substitutingβ⋆\\beta^\{\\star\}into[Equation6](https://arxiv.org/html/2608.13719#A1.E6)gives the local control\-variate estimator used in\. Hence,μCV\\mu\_\{\\mathrm\{CV\}\}is the minimum\-variance estimator among all estimators of the affine local control\-variate form in[Equation6](https://arxiv.org/html/2608.13719#A1.E6)\.

The corresponding optimal variance is

Var⁡\(μCV​\(β⋆\)\)=1n​\(Var⁡\(Yr\)−kn\+k​Cov​\(Yr,Ys\)​Var​\(Ys\)−1​Cov​\(Yr,Ys\)\)\.\\mathrm\{Var\}\\left\(\\mu\_\{\\mathrm\{CV\}\}\(\\beta^\{\\star\}\)\\right\)=\\frac\{1\}\{n\}\\left\(\\mathrm\{Var\}\(Y\_\{r\}\)\-\\frac\{k\}\{n\+k\}\\mathrm\{Cov\}\(Y\_\{r\},Y\_\{s\}\)\\mathrm\{Var\}\(Y\_\{s\}\)^\{\-1\}\\mathrm\{Cov\}\(Y\_\{r\},Y\_\{s\}\)\\right\)\.\(14\)In the scalar case, this can be written as

Var⁡\(μCV​\(β⋆\)\)=Var⁡\(F\)n​\(1−kn\+k​ρ2​\(Yr,Ys\)\),\\mathrm\{Var\}\\left\(\\mu\_\{\\mathrm\{CV\}\}\(\\beta^\{\\star\}\)\\right\)=\\frac\{\\mathrm\{Var\}\(F\)\}\{n\}\\left\(1\-\\frac\{k\}\{n\+k\}\\rho^\{2\}\(Y\_\{r\},Y\_\{s\}\)\\right\),\(15\)whereρ⁡\(Yr,Ys\)\\rho\(Y\_\{r\},Y\_\{s\}\)is the local Pearson correlation between the target and proxy statistics\. Sinceρ2​\(Yr,Ys\)∈\[0,1\]\\rho^\{2\}\(Y\_\{r\},Y\_\{s\}\)\\in\[0,1\], we have

Var⁡\(μ^CV​\(β⋆\)\)≤Var⁡\(Yr\)n,\\mathrm\{Var\}\\left\(\\hat\{\\mu\}\_\{\\mathrm\{CV\}\}\(\\beta^\{\\star\}\)\\right\)\\leq\\frac\{\\mathrm\{Var\}\(Y\_\{r\}\)\}\{n\},\(16\)which is the variance of the direct local Monte Carlo estimator using only thennsurrogate induced\-statistic samples overYrY\_\{r\}\. Therefore, whenever the proxy statistic is locally correlated with the surrogate induced statistic, the control\-variate estimator strictly reduces variance\. When the local correlation is zero, it matches the target\-only estimator in variance\.

Applying this result to our setting,YrY\_\{r\}corresponds to the local target failure statistic induced by the surrogateqθq\_\{\\theta\}, andYsY\_\{s\}corresponds to the local proxy statistic induced byp⁡\(ys∣x\)p\(y\_\{s\}\\mid x\)\. Thus, under the local sampling assumptions above and conditional on the learned surrogate, the coefficient in[Equation13](https://arxiv.org/html/2608.13719#A1.E13)yields the minimum\-variance local affine estimator of the target failure statistic\. This justifies usingμCV\\mu\_\{\\mathrm\{CV\}\}as a proxy\-corrected failure predictor for selecting scenarios likely to satisfyμr​\(x\)≥γ\\mu\_\{r\}\(x\)\\geq\\gamma\.

#### Note:

For MC sampled datax′∼Bxx^\{\\prime\}\\sim B\_\{x\}, the resultingy^r∼qθ​\(x\)\\hat\{y\}\_\{r\}\\sim q\_\{\\theta\}\(x\)may give biased results in practice as the data used to trainqθ​\(x\)q\_\{\\theta\}\(x\)is not unbiased, and acquired through active learning, hence this justification is only valid for the case when surrogate induced statistic is an unbiased representation of the target statistic\. To bridge this practical gap, we focus on selecting right hyperparameter for the training and architecture ofqθq\_\{\\theta\}\. Newer training paradigms that calibrate the loss of trainingqθq\_\{\\theta\}, such that the resulting estimator is equivalent to an estimator trained on unbiased data will be investigated as a part of our future work\.

## Appendix BMotivation for MI\-adjusted data acquisition

We combine MI with failure discovery as two distinct objectives in data acquisition\. This setup emerges naturally under active inference based data acquisition proposed by\[[16](https://arxiv.org/html/2608.13719#bib.bib1)\]\. In[AppendixD](https://arxiv.org/html/2608.13719#A4), we discuss the construction of various terms that appear in[Section4\.2](https://arxiv.org/html/2608.13719#S4.SS2)in detail\.

## Appendix CDefining locality in the feature space

This section discusses the details of construction of local neighborhood regionℬx\\mathcal\{B\}\_\{x\}in detail for the case where𝒳\\mathcal\{X\}is a continuous space\.

The neighborhoodℬx\\mathcal\{B\}\_\{x\}can be defined in multiple ways, and controls the granularity ofμC​V​\(x\)\\mu\_\{CV\}\(x\)\. For example, consider a continuous region, with the neighborhoodℬx\\mathcal\{B\}\_\{x\}defined using a Euclidean ball with radiusRRasℬx=\{x′\|∥x−x′∥22≤R\}\\mathcal\{B\}\_\{x\}=\\\{x^\{\\prime\}\|\\lVert x\-x^\{\\prime\}\\rVert^\{2\}\_\{2\}\\leq R\\\}\. Consider,R∈\[Rmin,Rmax\]R\\in\[R\_\{\\text\{min\}\},R\_\{\\text\{max\}\}\], such that forR=RmaxR=R\_\{\\text\{max\}\},ℬx≡𝒳\\mathcal\{B\}\_\{x\}\\equiv\\mathcal\{X\}\. The correspondingμC​V​\(x\)\\mu\_\{CV\}\(x\)is equivalent to a global average CV estimator, represented in\[[17](https://arxiv.org/html/2608.13719#bib.bib26)\]\. On the other hand, prediction forR→0R\\to 0collapses to a point\-wise local mean prediction\. For cases where we cannot manually specify the regionℬx\\mathcal\{B\}\_\{x\}, we propose a sensitivity aware heuristic for adaptively adjustingR∈\[Rmin,Rmax\]R\\in\[R\_\{\\text\{min\}\},R\_\{\\text\{max\}\}\], such that if the local sensitivity of prediction is high,RRshould be smaller and vice versa\. We define a sensitivity terms⁡\(x\)s\(x\)that measures sensitivity of prediction using local gradient information and variance of the surrogate model atxx:

s⁡\(x\)=‖∇xμq​\(x\)‖\(σq​\(x\)\+ϵ\),s~​\(x\)=s⁡\(x\)−minx′∈𝒳⁡s⁡\(x′\)maxx′∈𝒳⁡s⁡\(x′\)−minx′∈𝒳⁡s⁡\(x′\)\+ϵ\.s\(x\)=\\frac\{\\left\\\|\\nabla\_\{x\}\\mu\_\{q\}\(x\)\\right\\\|\}\{\\left\(\\sigma\_\{q\}\(x\)\+\\epsilon\\right\)\},\\quad\\quad\\tilde\{s\}\(x\)=\\frac\{s\(x\)\-\\min\_\{x^\{\\prime\}\\in\\mathcal\{X\}\}s\(x^\{\\prime\}\)\}\{\\max\_\{x^\{\\prime\}\\in\\mathcal\{X\}\}s\(x^\{\\prime\}\)\-\\min\_\{x^\{\\prime\}\\in\\mathcal\{X\}\}s\(x^\{\\prime\}\)\+\\epsilon\}\.\(17\)Heres~​\(x\)∈\[0,1\]\\tilde\{s\}\(x\)\\in\[0,1\]is a normalized version ofs⁡\(x\)s\(x\)for practical application, andσq2​\(x\)=Varqθ​\(x\)​\[yr\]\\sigma\_\{q\}^\{2\}\(x\)=\\text\{Var\}\_\{q\_\{\\theta\}\(x\)\}\[y\_\{r\}\]\. We construct our heuristic as:

Ra​\(x\)=Rmin\+\(Rmax−Rmin\)1\+exp⁡\(s~​\(x\)\)\.R\_\{a\}\(x\)=R\_\{\\min\}\+\\frac\{\\left\(R\_\{\\max\}\-R\_\{\\min\}\\right\)\}\{1\+\\exp\\left\(\\tilde\{s\}\(x\)\\right\)\}\.\(18\)[Figure7](https://arxiv.org/html/2608.13719#A3.F7)showsμC​V\\mu\_\{CV\}as a function ofxxfor a 1D toy example, with local regionℬx\\mathcal\{B\}\_\{x\}chosen using fixed radius values and adaptiveRa​\(x\)R\_\{a\}\(x\)chosen using[Equation18](https://arxiv.org/html/2608.13719#A3.E18)\. We observe that the prediction withRaR\_\{a\}most accurately predicts the ground\-truth target mean as opposed to fixed region alternatives\.

![Refer to caption](https://arxiv.org/html/2608.13719v1/1d.png)Figure 7:1D synthetic task\.Left\-right: \(1\) Randomly sampled 10 scenarios and observed values of synthetic target, shown against ground truth target and proxy system\. \(2\) Scenarios identified as high MI due to being outside support/novel candidates \([Section4\.2](https://arxiv.org/html/2608.13719#S4.SS2)\)\. \(3\)μ^C​V​\(x\)\\hat\{\\mu\}\_\{CV\}\(x\)estimated using different fixed local regions and adaptive local region selection plotted against ground truth target\. Adaptively selecting local region shows minimum deviation from ground truth\.
## Appendix DConstruction of terms for MI estimation

To estimate MI, we need to estimate entropyHH, for which, we model the distribution overRxR\_\{x\}asp⁡\(Rx\|x,𝒟t\)=∑k=1Kp⁡\(Rx\|Zx=k\)​p​\(Zx=k\|x,𝒟t\)p\(R\_\{x\}\|x,\\mathcal\{D\}\_\{t\}\)=\\sum\_\{k=1\}^\{K\}p\(R\_\{x\}\|Z\_\{x\}=k\)p\(Z\_\{x\}=k\|x,\\mathcal\{D\}\_\{t\}\)\. Here, priorp⁡\(Zx=k\|𝒟t,x\)p\(Z\_\{x\}=k\|\\mathcal\{D\}\_\{t\},x\)controls probability of assignment of a pointxxto a cluster, and can be either observed directly for probabilistic clustering approaches such as GMMs, or can be manually designed, as shown below for K\-means clustering used in this work\. We define the likelihoodp⁡\(Rx\|Zx=k\)p\(R\_\{x\}\|Z\_\{x\}=k\)to be high for scenarios far from existing clusters as:

p⁡\(Rx=1∣Zx=k\)=qk​\(x\),p⁡\(Rx=1∣Zx=new,x,Dt\)=qnew​\(x\),p\(R\_\{x\}=1\\mid Z\_\{x\}=k\)=q\_\{k\}\(x\),\\qquad p\(R\_\{x\}=1\\mid Z\_\{x\}=\\mathrm\{new\},x,D\_\{t\}\)=q\_\{\\mathrm\{new\}\}\(x\),\(19\)Here0≤qk​\(x\)≤εexist0\\leq q\_\{k\}\(x\)\\leq\\varepsilon\_\{\\mathrm\{exist\}\}is a distance weighted distribution that is00close to an existing cluster, and is capped at a small valueεexist\\varepsilon\_\{\\mathrm\{exist\}\}far away\. On the other hand,qnew​\(x\)q\_\{\\mathrm\{new\}\}\(x\)is defined as:

qnew​\(x\)=σ⁡\(dmin​\(x\)−ργ\),σ⁡\(z\)=11\+e−z,q\_\{\\mathrm\{new\}\}\(x\)=\\sigma\\left\(\\frac\{d\_\{\\min\}\(x\)\-\\rho\}\{\\gamma\}\\right\),\\qquad\\sigma\(z\)=\\frac\{1\}\{1\+e^\{\-z\}\},\(20\)wheredmin​\(x\)d\_\{\\min\}\(x\)is the distance fromxxto the nearest observed cluster boundary,γ\\gammais a softness parameter andρ\\rhois a novelty radius which controls the minimum distance threshold for a scenarioxxto be considered novel\.[Equation19](https://arxiv.org/html/2608.13719#A4.E19)and[Equation20](https://arxiv.org/html/2608.13719#A4.E20)specify a model for geometric distance based support discovery such thatp⁡\(Rx=1\)p\(R\_\{x\}=1\)has high probability mass around novel scenarios, leading to the discovery of new clusters\. We use data adaptive approaches in practice to estimateγ,ρ\\gamma,\\rhoandKKto mitigate hyperparameter dependency\.KKis estimated from a range of\[Kmin,Kmax\]\[K\_\{\\text\{min\}\},K\_\{\\text\{max\}\}\]using the elbow heuristic\. We deriveγ,ρ\\gamma,\\rhodirectly from the geometry of the current training support𝒟t\\mathcal\{D\}\_\{t\}\. Specifically, we compute the nearest\-neighbour distance for each support pointxi∈𝒟tx\_\{i\}\\in\\mathcal\{D\}\_\{t\}as

dnn,i=minj≠i⁡‖xi−xj‖,d\_\{\\mathrm\{nn\},i\}=\\min\_\{j\\neq i\}\\\|x\_\{i\}\-x\_\{j\}\\\|,\(21\)and set

ρ=median⁡\(\{dnn,i\}i=1M\),γ=std⁡\(\{dnn,i\}i=1M\),\\rho=\\mathrm\{median\}\\\!\\left\(\\\{d\_\{\\mathrm\{nn\},i\}\\\}\_\{i=1\}^\{M\}\\right\),\\qquad\\gamma=\\mathrm\{std\}\\\!\\left\(\\\{d\_\{\\mathrm\{nn\},i\}\\\}\_\{i=1\}^\{M\}\\right\),\(22\)
### Defining probabilityp⁡\(Zx∣x,𝒟t\)p\(Z\_\{x\}\\mid x,\\mathcal\{D\}\_\{t\}\)

Given cluster centers\{ck\}k=1K\\\{c\_\{k\}\\\}\_\{k=1\}^\{K\}with per\-cluster RMS spreadlk=1\|k\|​∑i∈k‖xi−ck‖2l\_\{k\}=\\sqrt\{\\frac\{1\}\{\|k\|\}\\sum\_\{i\\in k\}\\\|x\_\{i\}\-c\_\{k\}\\\|^\{2\}\}, the prior over support stateZxZ\_\{x\}atxxis defined using a distance based kernelwkw\_\{k\}:

wk​\(x\)=exp⁡\(−‖x−ck‖22​lk2\),W⁡\(x\)=∑k=1Kwk​\(x\)w\_\{k\}\(x\)=\\exp\\\!\\left\(\-\\frac\{\\\|x\-c\_\{k\}\\\|^\{2\}\}\{2\\,l\_\{k\}^\{2\}\}\\right\),\\qquad W\(x\)=\\sum\_\{k=1\}^\{K\}w\_\{k\}\(x\)\(23\)
p⁡\(Z=zk∣x\)=wk​\(x\)W⁡\(x\)\+1,p⁡\(Z=znew∣x\)=1W⁡\(x\)\+1p\(Z=z\_\{k\}\\mid x\)=\\frac\{w\_\{k\}\(x\)\}\{W\(x\)\+1\},\\qquad p\(Z=z\_\{\\mathrm\{new\}\}\\mid x\)=\\frac\{1\}\{W\(x\)\+1\}\(24\)
[Figure6](https://arxiv.org/html/2608.13719#Ax1.F6)shows the reconstruction of failure regions using a GP surrogate model by performing data acquisition using our approach on a tor 2D synthetic task\. The task consists of two low prob failures shown in yellow, which we aim to be discover inB=40B=40samples\.[Figure6](https://arxiv.org/html/2608.13719#Ax1.F6)shows that our overall approach succesfully applies to uncover both regions, showing that our MI strategy helps in searching for diverse failures, andμCV\\mu\_\{\\text\{CV\}\}helps to bridge sim and real gap\.

## Appendix EnuPlan ablations

### E\.1Ablation results for size of initial dataset

We conduct ablation for initial size of sampling withN0=5,10,20,50N\_\{0\}=5,10,20,50initial data points for surrogate model training for our method, results forB=100B=100samples reported in[Table4](https://arxiv.org/html/2608.13719#A5.T4)\. Clearly, forN0=50N\_\{0\}=50we observe lowest cumulative TTC and highest diversity as well as failure count, however, we observe that the difference betweenN0=5,10,20N\_\{0\}=5,10,20is marginal, which tells us that in the lack of sufficient initial data, our method’s performance is not substantially affected\. For another 20 samples, withB=140B=140, the run initialized withN0=20N\_\{0\}=20reports1\.05,0\.81\.05,0\.8and0\.330\.33for TTC average, coverage and positive samples, indicating that the lack of sufficient initial samples in high dimensional scenarios can be compensated for by increasing the testing budget and collecting more sequential data\.

Table 4:Ablation over different initialization sizes on thenuPlantask, 100 samples, four seeds, avg±\\pmstd reported\.SettingAvg\. Cumulative TTCCumulative CoveragePositive Samplesninit=5n\_\{\\text\{init\}\}=51\.09±0\.061\.09\\pm 0\.060\.58±0\.040\.58\\pm 0\.040\.27±0\.020\.27\\pm 0\.02ninit=10n\_\{\\text\{init\}\}=101\.16±0\.131\.16\\pm 0\.130\.48±0\.090\.48\\pm 0\.090\.25±0\.010\.25\\pm 0\.01ninit=20n\_\{\\text\{init\}\}=201\.06±0\.041\.06\\pm 0\.040\.75±0\.020\.75\\pm 0\.020\.26±0\.030\.26\\pm 0\.03ninit=50n\_\{\\text\{init\}\}=500\.99±0\.060\.99\\pm 0\.061\.10±0\.131\.10\\pm 0\.130\.30±0\.050\.30\\pm 0\.05
### E\.2Ablation results forμCV\\mu\_\{\\text\{CV\}\}

[Table5](https://arxiv.org/html/2608.13719#A5.T5)shows results for ablation over[Equation2](https://arxiv.org/html/2608.13719#S4.E2)withβ=0\\beta=0, to remove the dependency on sim samples\. We observe that withβ=0\\beta=0, there is marginal improvement in positive samples and cumulative TTC, however, this comes at the cost of coverage, which is especially compromised for samples with TTC=0\.0=0\.0, i\.e\., severe failure scenarios\. This is due to the fact that while proxy scenarios, in this case constructed from open loop simulation, are a noisy estimate of the target system, having larger sampling capacity on proxy helps identify more regions where the target system might be failing, which may be missed in limited budget target only estimation\.

Table 5:Ablation comparing our approach with and without sim samples by settingβ=0\\beta=0on thenuPlantask\. Results reported for 20 initial samples,B=150B=150, avg and std for four seeds\.SettingAvg\. Cumulative TTCCumulative CoveragePositive Samplesγ=0\.3\\gamma=0\.3nominal1\.04±0\.03321\.04\\pm 0\.03320\.98±0\.060\.98\\pm 0\.060\.32±0\.010\.32\\pm 0\.01β=0\\beta=01\.03±0\.051\.03\\pm 0\.050\.95±0\.050\.95\\pm 0\.050\.35±0\.010\.35\\pm 0\.01γ=0\.0\\gamma=0\.0nominal1\.04±0\.031\.04\\pm 0\.030\.87±0\.050\.87\\pm 0\.050\.29±0\.010\.29\\pm 0\.01β=0\\beta=01\.03±0\.051\.03\\pm 0\.050\.81±0\.080\.81\\pm 0\.080\.31±0\.010\.31\\pm 0\.01

## Appendix FQuadruped

Additional results forQuadruped, showing scatterplot of our method andBAMSin[Figure8](https://arxiv.org/html/2608.13719#A6.F8), and[Figure9](https://arxiv.org/html/2608.13719#A6.F9)showing heatmap of learnt failure versus success region, providing qualitative evidence that our method discovers more diverse failures\. We also provide video demos of two failure and one non\-fail command velocity tracking tasks in the supplementary video attached, where the scenarios are selected as those predicted fail and non fail by our method\.

Figure 8:Scatterplot of failure scenariosx=\(vx,vy,wz\)x=\(v\_\{x\},v\_\{y\},w\_\{z\}\)\(y≥0\.7y\\geq 0\.7\) forQuadrupedtask, generated by our method \(red\), andBAMS\(blue\)\. Our method shows higher diversity of scenarios unlikeBAMSthat tends to concentrate around failures discovered in early iterations\.Figure 9:Heatmaps showing learnt surrogate model predictions projected over scenario space forQuadrupedtask\. Projection value corresponds to median of third variable reported in parentheses\. Scattered points correspond to initial 10 and subsequent 20 scenarios collected by our approach\.
## Appendix GSIMPLER

Baseline comparison forSIMPLERshown in[Figure10](https://arxiv.org/html/2608.13719#A7.F10)\. We also show videos of fail and non fail tasks, where the successful task is provided as a reference, and fail tasks are sampled by our approach consisting of one task where both proxy and target system fail, and one where only target system fails, showing that our approach can discover failures unseen in simulation as well\.

Figure 10:Baseline comparisonforSIMPLERtask, showing avg±\\pmstd across three seeds forB=70B=70samples, with1010initial random samples for training all surrogate models, and 60 additional samples acquired in batches ofb=3b=3\.
## Appendix HVisualization

In[Figure12](https://arxiv.org/html/2608.13719#A8.F12)we show additional examples forSIMPLERtask\. We also show videos of failures forQuadrupedtask, which correspond to high velocity invxv\_\{x\}andvyv\_\{y\}, as tracking larger requires crossing the obstacles used to construct a boundary, leading to obstacle avoidance getting activated and velocity becoming zero\.

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/failure1.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/failure2.png)

Figure 11:Failures forQuadrupedtask\. Quadrupeds shown at the end of tracking duration, failure scenarios correspond toQuadrupedreaching near one of the boxed and stopping due to obstacle avoidance\.![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/screenshot1.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/screenshot2.png)

![Refer to caption](https://arxiv.org/html/2608.13719v1/figs/screenshot3.png)

Figure 12:More explains of failure scenarios uncovered by our approach in theSIMPLERtask\.
## Appendix IExperiment details

### I\.1Model specifications

We choose randomly sampled data to initialize the various surrogate models for each setup\. FornuPlanandSIMPLER, we use BNN surrogate models for our method, which consist of a MLP with MC dropouts at each layer \(p⁡\(dropout\)=0\.1p\(\\text\{dropout\}\)=0\.1\), and forQuadruped, we use MLP, which is suitable choice due to low observed stochasticity in the velocity tracking policy\. For each setup and across all baselines that use a NN based architecture, we train usingAdamoptimizer with lr=1e\-3, with 1000, 800 and 3000 epochs fornuPlan, SIMPLERandQuadrupedtasks respectively\. These hyperparameter are selected based on training accuracy over the initial random sampling dataset\.[Table6](https://arxiv.org/html/2608.13719#A9.T6)summarizes the model architectures for all tasks\.nuPlanhas inherent over\-parameterization in scenario due to encoding so we benefit from a compressive architecture whereasSIMPLERbenefits from over\-parameterization\.

Table 6:Summary of neural network architectures across tasks\.TaskArchitectureInputHidden layersActivationnuPlansurrogateMLP38496→24→696\\to 24\\to 6ELU \+ LNnuPlanBNN\-C head\+ sigmoid——σ\\sigmaSIMPLERBNNMLP1996→24→696\\to 24\\to 6TanhSIMPLERclassifier for GP\-CMLP \+ sigmoid1996→24→696\\to 24\\to 6σ\\sigmaQuadrupedMLPMLP38→88\\to 8ReLU
### I\.2Data specifications

FornuPlan, proxy dataset is directly available and used as it is, whereasSIMPLERandQuadrupedare setups where querying proxy data can be costly as well\. Hence, we use a BNN and MLP for constructing a surrogate model for proxy systems forSIMPLERandQuadruped, using larger initial datasets of 50 and 2000 datapoints collected offline\. ForSIMPLER, the initial dataset for proxy surrogate model is quite less, hence, at each step of acquisition for target system, we also update proxy system with 1 datapoint\.BAMSexpects a cost of collecting proxy and target data as an input, which is calibrated to match the frequency of data collection used by our approach\. ForBNN\-CinnuPlan, we use the same architecture as ours\. Note that the baseline is originally supposed to work with GPs, which can be incompatible for large size ofnuPlan, hence we use a BNN there\.

### I\.3Scenario design and failure metrics for each task

FornuPlan, scenarios are driving logs from nuPlan database, which are encoded into 384\-dimensional vectors, we measure TTC as failure metric generated by nuPlan, and proxy and target system correspond to open loop and closed loop simulations respectively\.

ForSIMPLERtask, we perturb visual specifications and object placements, generating a 19 dimensional scenario specification\. The target object \(a Coke can\) is placed at a 2D tabletop positionxobj∈\[−0\.5,−0\.1\]x\_\{\\text\{obj\}\}\\in\[\-0\.5,\-0\.1\]m andyobj∈\[0\.0,0\.4\]y\_\{\\text\{obj\}\}\\in\[0\.0,0\.4\]m, spanning the robot’s reachable workspace\. Camera viewpoint variation is introduced via lateral translationsΔ​xcam,Δ​ycam∈\[−0\.025,0\.025\]\\Delta x\_\{\\text\{cam\}\},\\,\\Delta y\_\{\\text\{cam\}\}\\in\[\-0\.025,0\.025\]m relative to the nominal camera pose\. Scene appearance is controlled by brightnessb∈\[0\.70,1\.10\]b\\in\[0\.70,1\.10\]and contrastc∈\[0\.90,1\.25\]c\\in\[0\.90,1\.25\], modelling illumination variation between the proxy and target domains\. Finally, distractor objects are placed at positionsxdist∈\[−0\.5,−0\.1\]x\_\{\\text\{dist\}\}\\in\[\-0\.5,\-0\.1\]m,ydist∈\[0\.0,0\.4\]y\_\{\\text\{dist\}\}\\in\[0\.0,0\.4\]m with uniformly sampled yawθdist∈\[−π,π\]\\theta\_\{\\text\{dist\}\}\\in\[\-\\pi,\\pi\]rad, introducing clutter that occludes the target object and disrupts grasping\. Across three task variants,pick\_horizontal,pick\_vertical, andpick\_standing, and choosing distractor objects from a selection of 8 objects makes a 19 dimensional scenario representation\.

Failure here is recorded as failure to grasp or lift the coke can, and is generated by the environment\. Target and proxy systems correspond to slight difference in visual specifications, and target system has extra visual noise acting as occlusion, leading to worse performance on several scenarios\.

ForQuadruped, the scenario is 3\-d and corresponds to command velocityvx,vy,wzv\_\{x\},v\_\{y\},w\_\{z\}in the range\[−0\.8,1\.0\]×\[−0\.8,0\.8\]×\[−0\.8,0\.8\]\[\-0\.8,1\.0\]\\times\[\-0\.8,0\.8\]\\times\[\-0\.8,0\.8\]and proxy system consists of a simulation with RL policy for command velocity tracking, whereas target system consists of the actual hardware with Unitree sports mode used for command velocity tracking\. We initialize the quadruped at the same location within a square boundary constructed using cardboard boxes\. However, we observe that failure scenarios are agnostic to initialization provided they are initialized at one of the four corners\.