AdaStop: Cost-Aware Early Stopping for DNN Test Selection

# AdaStop: Cost-Aware Early Stopping for DNN Test Selection
Source: [https://arxiv.org/html/2607.05461](https://arxiv.org/html/2607.05461)
Bonan Shen1, Wei\-Jung Huang1, Xin Liu1, Jiazhou Gao1and Tao Ning1

###### Abstract

Existing methods for testing deep neural networks \(DNNs\) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget\. In practice, choosing that budget is difficult: too little testing misses failures, while too much incurs unnecessary labeling costs\. This work studies the stopping problem in DNN testing\. We formulate testing as a cost–benefit decision process in which labeling an input incurs costccand discovering a fault yields valuevv\. Based on this formulation, we introduceAdaStop, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the thresholdτ=c/v\\tau=c/v\. Experiments across multiple datasets, architectures, and selection strategies show that6565–84%84\\%of faults can be discovered using only99–31%31\\%of the labeling budget\.

## IIntroduction

Deep neural networks \(DNNs\) are increasingly deployed in domains where reliability is critical, including autonomous vehicles\[[24](https://arxiv.org/html/2607.05461#bib.bib24)\], medical diagnosis\[[25](https://arxiv.org/html/2607.05461#bib.bib25)\], and financial systems\[[26](https://arxiv.org/html/2607.05461#bib.bib26)\]\. Ensuring reliability requires systematic testing to identify test inputs on which the model makes incorrect predictions—commonly referred to asfaults\. In many settings, however, verifying predictions requires ground\-truth labels obtained through costly human annotation by domain experts\[[27](https://arxiv.org/html/2607.05461#bib.bib27)\]\.

This creates a fundamental and practical challenge:when should we stop labeling test inputs?Stopping too early risks missing faults that could lead to system failures; stopping too late wastes labeling resources on inputs that are unlikely to reveal new faults\.

Existing test selection methods primarily focus on theselectionproblem—determining which inputs to label first—while the complementarystoppingproblem has received comparatively little attention\. We identify three key limitations:

1. 1\.Lack of principled stopping criterion\.Methods like DeepGini\[[9](https://arxiv.org/html/2607.05461#bib.bib9)\], TestRank\[[10](https://arxiv.org/html/2607.05461#bib.bib10)\], ATS\[[11](https://arxiv.org/html/2607.05461#bib.bib11)\], and DeepSample\[[12](https://arxiv.org/html/2607.05461#bib.bib12)\]typically evaluate performance under predetermined labeling budgets without providing guidance on when testing should terminate\.
2. 2\.Arbitrary budget selection\.In the absence of stopping criteria, practitioners must choose labeling budgets heuristically\. However, appropriate budgets depend on factors such as model quality and dataset characteristics, which are often unknown beforehand\.
3. 3\.Unaddressed cost–benefit trade\-off\.Labeling each input incurs cost, while discovering a fault provides value\. Existing approaches focus on optimizing theorderof test selection, such as prioritizing inputs that are more likely to reveal model errors, but do not explicitly consider this cost–benefit trade\-off\[[9](https://arxiv.org/html/2607.05461#bib.bib9)\]\.

Although stopping criteria exist in related domains, such as the SAFE procedure\[[18](https://arxiv.org/html/2607.05461#bib.bib18)\]for document screening and convergence\-based stopping\[[13](https://arxiv.org/html/2607.05461#bib.bib13)\]for active learning, these methods have not been adapted to DNN test selection, where the goal is efficient fault discovery\.

A key empirical observation motivates our approach: test selection often exhibitsdiminishing returns\. Inputs with higher uncertainty, which are typically selected first, tend to reveal faults at higher rates\. As testing progresses, the remaining inputs yield progressively fewer faults per labeled instance\.

This behavior suggests that the value of additional labeling decreases over time, creating a natural point at which further testing becomes inefficient\. Based on this observation, we proposeAdaStop, a cost\-aware framework that terminates testing when the marginal fault discovery ratep​\(t\)p\(t\)falls below a cost\-adjusted thresholdτ=c/v\\tau=c/v, whereccdenotes the labeling cost andvvrepresents the value of discovering a fault\.

The AdaStop framework consists of three components: \(1\) an uncertainty\-based selection strategy \(DeepGini\) to prioritize inputs likely to reveal faults, \(2\) a sliding\-window estimator of the marginal fault discovery rate, and \(3\) a cost\-aware stopping rule that terminates labeling once the estimated rate falls belowτ\\tau\.

Contributions\.\(1\) We frame DNN test selection as a sequential cost\-benefit optimization, deriving the optimal stopping conditionτ=c/v\\tau=c/v\. \(2\) We propose a cost\-aware threshold\-based stopping rule and compare it against practical alternatives, including patience, consecutive non\-faults, confidence\-based, and cumulative\-rate stopping\. \(3\) We conduct a comprehensive evaluation across 3 datasets, 4 architectures, 3 quality levels, and 8 strategies, achieving 65–84% recall with 70–91% budget savings\. \(4\) We provide an open\-source implementation for community adoption\.

## IIProblem Formulation

### II\-ASetting and Objective

Consider a DNN classifierM:𝒳→𝒴M:\\mathcal\{X\}\\rightarrow\\mathcal\{Y\}and an unlabeled test pool𝒰=\{x1,…,xn\}\\mathcal\{U\}=\\\{x\_\{1\},\\ldots,x\_\{n\}\\\}\. Afaultis an input whereM​\(x\)≠y∗M\(x\)\\neq y^\{\*\}\(the ground\-truth label\)\. Labeling costsc\>0c\>0per query; each fault discovered has valuev\>0v\>0\.

Test selection proceeds sequentially: at steptt, select inputxtx\_\{t\}from the remaining pool via strategyss, query the oracle for labelyty\_\{t\}, record outcomert=𝟏​\[M​\(xt\)≠yt\]r\_\{t\}=\\mathbf\{1\}\[M\(x\_\{t\}\)\\neq y\_\{t\}\], and decide whether to continue\. Thenet valueat stopping timeTTis:

V​\(T\)=v⋅F​\(T\)−c⋅TV\(T\)=v\\cdot F\(T\)\-c\\cdot T\(1\)whereF​\(T\)=∑t=1TrtF\(T\)=\\sum\_\{t=1\}^\{T\}r\_\{t\}is the number of faults found\.

### II\-BOptimal Stopping Threshold

At steptt, the marginal cost of labeling one more input iscc, and the marginal benefit isv⋅p​\(t\)v\\cdot p\(t\), wherep​\(t\)=𝔼​\[rt\]p\(t\)=\\mathbb\{E\}\[r\_\{t\}\]is the probability of discovering a fault\. The marginal net value of the next label is:

Δ​V​\(t\)=v⋅p​\(t\)−c\\Delta V\(t\)=v\\cdot p\(t\)\-c\(2\)
The optimal decision rule follows directly:continueifv⋅p​\(t\)\>cv\\cdot p\(t\)\>c\(positive marginal value\);stopifv⋅p​\(t\)≤cv\\cdot p\(t\)\\leq c\. Rearranging the stopping condition gives:

τ=cv\\boxed\{\\tau=\\frac\{c\}\{v\}\}\(3\)
Proposition 1 \(Optimal Stopping\)\.Under diminishing returns \(fault ratep​\(t\)p\(t\)is non\-increasing\), the optimal policy stops at the firstTTwherep​\(T\)≤τ=c/vp\(T\)\\leq\\tau=c/v\. Oncep​\(t\)p\(t\)drops belowτ\\tau, it remains below for all subsequent steps, so stopping immediately is optimal\.□\\square

The thresholdτ\\tauis thebreak\-even fault rate\. For example, if labeling costsc=$​1c=\\mathdollar 1and each fault is worthv=$​20v=\\mathdollar 20, thenτ=0\.05\\tau=0\.05: we stop when fewer than 5% of labels are expected to reveal faults\. At this point, the expected value of the next label exactly equals its cost\. The threshold automatically adapts—safety\-critical scenarios \(lowc/vc/v\) continue longer; expensive\-labeling scenarios \(highc/vc/v\) stop earlier\.

### II\-CFault Rate Estimation

Sincep​\(t\)p\(t\)is not directly observable, we estimate it from recent history using a sliding window:

p^​\(t\)=1W​∑i=t−W\+1tri\\hat\{p\}\(t\)=\\frac\{1\}\{W\}\\sum\_\{i=t\-W\+1\}^\{t\}r\_\{i\}\(4\)whereWWis the window size\. This estimator addresses three challenges: \(1\)non\-stationarity—the fault rate decreases as testing progresses, so we use only recent samples rather than the full history; \(2\)noise—individual outcomes are binary, so averaging overWWsamples provides smoothing; \(3\)responsiveness—the window adapts to rate changes faster than a cumulative average\. The choice ofWWinvolves a bias\-variance trade\-off: smallWWis responsive but noisy \(risking premature stopping\); largeWWis stable but slow to detect rate drops\. We analyze this empirically in Section[VI](https://arxiv.org/html/2607.05461#S6)\.

### II\-DThe Diminishing Returns Property

The optimal stopping condition relies on a key empirical observation:test selection exhibits diminishing returns\. When using uncertainty\-based selection \(e\.g\., DeepGini\), high\-uncertainty inputs are selected first\. Since high uncertainty correlates with higher fault probability, the fault ratep​\(t\)p\(t\)decreases asttincreases\. This ensures a natural stopping point wherep​\(t\)p\(t\)crosses belowτ\\tau\. Without this property, the stopping problem would be ill\-defined—the rate could fluctuate unpredictably\. We validate this assumption formally with Mann\-Kendall monotonicity tests in Section[VI](https://arxiv.org/html/2607.05461#S6)\.

## IIIRelated Work

### III\-ATest Selection for DNNs

Uncertainty\-based methodsprioritize inputs where the model is least confident\. DeepGini\[[9](https://arxiv.org/html/2607.05461#bib.bib9)\]uses Gini impurity of softmax outputs; FAST\[[15](https://arxiv.org/html/2607.05461#bib.bib15)\]addresses over\-confidence through guided feature selection; CertPri\[[17](https://arxiv.org/html/2607.05461#bib.bib17)\]uses movement cost with formal robustness guarantees\.Coverage and diversity\-based methodsaim for diverse test inputs: NLC\[[19](https://arxiv.org/html/2607.05461#bib.bib19)\]captures neuron output distributions; DeepGD\[[16](https://arxiv.org/html/2607.05461#bib.bib16)\]combines Gini score with geometric diversity via NSGA\-II\.Learning\-based methodsinclude TestRank\[[10](https://arxiv.org/html/2607.05461#bib.bib10)\]\(GNN\-based ranking\) and ATS\[[11](https://arxiv.org/html/2607.05461#bib.bib11)\]\(RL\-based adaptive selection\)\.Sampling\-based methodsinclude DeepSample\[[12](https://arxiv.org/html/2607.05461#bib.bib12)\]and DeepReduce\[[22](https://arxiv.org/html/2607.05461#bib.bib22)\]\. All these methods use fixed budgets without principled stopping criteria\.

### III\-BStopping Criteria in Other Domains

Active learning uses convergence\-based criteria\[[13](https://arxiv.org/html/2607.05461#bib.bib13),[14](https://arxiv.org/html/2607.05461#bib.bib14),[20](https://arxiv.org/html/2607.05461#bib.bib20)\]—stopping when the model stops improving\. These are fundamentally different from our setting: we are not training a model, but discovering faults in a fixed model\. Document screening uses discovery\-based criteria: SAFE\[[18](https://arxiv.org/html/2607.05461#bib.bib18)\]monitors discovery rates; Chao’s estimator\[[21](https://arxiv.org/html/2607.05461#bib.bib21)\]predicts total relevant documents\. Mittal et al\.\[[23](https://arxiv.org/html/2607.05461#bib.bib23)\]formulate model evaluation as an MDP but minimize estimation error, not fault discovery efficiency\. We adapt the discovery\-based stopping concept, adding a cost\-benefit formulation specific to DNN testing\.

### III\-CResearch Gap

Existing methods typically assume a fixed labeling budget and do not provide principled guidance on when additional testing is no longer worthwhile\. Table[I](https://arxiv.org/html/2607.05461#S3.T1)summarizes the comparison\.

TABLE I:Comparison of Related Work

## IVProposed Approach

### IV\-AFramework Overview

Figure[1](https://arxiv.org/html/2607.05461#S4.F1)illustrates the AdaStop framework\. The system iteratively selects inputs via uncertainty\-based ranking, obtains labels from an oracle, estimates the current fault rate, and stops when the rate falls below the cost\-justified threshold\.

Test Pool𝒰\\mathcal\{U\}ModelMMτ\\tau,WW,cc,vvSelection\(DeepGini\)Oracle\(Label\)RateEstimatorStop when:p^​\(t\)<τ\\hat\{p\}\(t\)<\\tauTest Setℒ\\mathcal\{L\}xtx\_\{t\}rtr\_\{t\}continueFigure 1:AdaStop Framework\. The system iteratively selects inputs via DeepGini, obtains labels, estimates the fault rate, and stops whenp^​\(t\)<τ\\hat\{p\}\(t\)<\\tau\.
### IV\-BSelection Strategy

AdaStop uses DeepGini\[[9](https://arxiv.org/html/2607.05461#bib.bib9)\]as its default selection strategy, prioritizing inputs by Gini impurity of softmax outputs:

Gini​\(x\)=1−∑i=1Cpi​\(x\)2\\text\{Gini\}\(x\)=1\-\\sum\_\{i=1\}^\{C\}p\_\{i\}\(x\)^\{2\}\(5\)wherepi​\(x\)p\_\{i\}\(x\)is the predicted probability for classiiandCCis the number of classes\. At each step, we select the input with the highest Gini score from the remaining pool, as uncertain predictions are more likely to be incorrect\. We choose DeepGini for its simplicity \(requires only softmax outputs\), effectiveness \(4–5×\\timesover random\), and wide adoption\. AdaStop’s stopping framework isstrategy\-agnosticand is compatible with any strategy that produces a ranked ordering \(Section[VI\-F](https://arxiv.org/html/2607.05461#S6.SS6)\)\.

### IV\-CStopping Criteria

We discuss four practical stopping criteria for different scenarios:

Threshold\-based \(default\)\.Stop whenp^​\(t\)<τ\\hat\{p\}\(t\)<\\tau, with a minimum sample constraintNminN\_\{\\min\}to ensure reliable estimates\. Directly implements the cost\-benefit optimality condition\. The thresholdτ=c/v\\tau=c/vshould reflect the testing scenario’s cost\-benefit ratio\.

Patience\-based\.Whenp^​\(t\)\\hat\{p\}\(t\)first drops belowτ\\tau, continue for an additional patience window ofkklabels and stop unless the estimated rate rises aboveτ\\tauagain\. This criterion adds a small safety margin against transient fluctuations at the stopping boundary\.

Consecutive non\-faults\.Stop after observingkkconsecutive labels with no faults\. This criterion is simple to implement and does not require setting an explicit cost\-benefit threshold, but it is less directly tied to the optimality condition\.

Confidence\-based\.For risk\-averse scenarios, stop when the upper bound of a Wilson confidence interval\[[8](https://arxiv.org/html/2607.05461#bib.bib8)\]falls belowτ\\tau:CIupper​\(p^\)<τ\\text\{CI\}\_\{\\text\{upper\}\}\(\\hat\{p\}\)<\\tau\. This reduces premature stopping at the cost of more budget\.

### IV\-DComplete Algorithm

Algorithm[1](https://arxiv.org/html/2607.05461#alg1)presents the complete AdaStop procedure\.

Algorithm 1AdaStop: Cost\-Aware Test Selection1:Input:Test pool

𝒰\\mathcal\{U\}, model

MM, threshold

τ\\tau, window

WW, min samples

NminN\_\{\\min\}
2:Output:Labeled test set

ℒ\\mathcal\{L\}
3:Compute Gini scores

Gi←1−∑jPi​j2G\_\{i\}\\leftarrow 1\-\\sum\_\{j\}P\_\{ij\}^\{2\}for all

xi∈𝒰x\_\{i\}\\in\\mathcal\{U\}
4:

ℛ←𝒰\\mathcal\{R\}\\leftarrow\\mathcal\{U\},

ℒ←∅\\mathcal\{L\}\\leftarrow\\emptyset,

H←\[\]H\\leftarrow\[\]
5:for

t=1,2,…t=1,2,\\ldotsdo

6:

xt←arg⁡maxxi∈ℛ⁡Gix\_\{t\}\\leftarrow\\arg\\max\_\{x\_\{i\}\\in\\mathcal\{R\}\}G\_\{i\};

ℛ←ℛ∖\{xt\}\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\setminus\\\{x\_\{t\}\\\}
7:

rt←𝟏​\[M​\(xt\)≠Oracle​\(xt\)\]r\_\{t\}\\leftarrow\\mathbf\{1\}\[M\(x\_\{t\}\)\\neq\\textsc\{Oracle\}\(x\_\{t\}\)\]; append

rtr\_\{t\}to

HH
8:

ℒ←ℒ∪\{\(xt,rt\)\}\\mathcal\{L\}\\leftarrow\\mathcal\{L\}\\cup\\\{\(x\_\{t\},r\_\{t\}\)\\\}
9:

p^←mean\(H\[max\(1,\|H\|−W\+1\):\|H\|\]\)\\hat\{p\}\\leftarrow\\text\{mean\}\(H\[\\max\(1,\|H\|\-W\+1\):\|H\|\]\)
10:if

\|ℒ\|≥Nmin\|\\mathcal\{L\}\|\\geq N\_\{\\min\}and

p^<τ\\hat\{p\}<\\tauthen

11:return

ℒ\\mathcal\{L\}\{Cost\-aware stop\}

12:endif

13:endfor

## VExperimental Setup

### V\-AResearch Questions

- •RQ1: Can AdaStop realize high recall with a small budget fraction?
- •RQ2: What is the impact ofτ\\tauon recall\-budget balance?
- •RQ3: What are the differences between various stopping criteria?
- •RQ4: How sensitive is AdaStop to window sizeWW?
- •RQ5: Does AdaStop have good generalization properties with respect to datasets and architectures?
- •RQ6: Is AdaStop strategy\-agnostic?
- •RQ7: How sensitive is AdaStop to model quality?

### V\-BDatasets, Model Architectures, and Configuration

We evaluate across three datasets:CIFAR\-10\[[1](https://arxiv.org/html/2607.05461#bib.bib1)\]\(10K test images\),SVHN\[[2](https://arxiv.org/html/2607.05461#bib.bib2)\]\(26K test images\), andFashionMNIST\[[3](https://arxiv.org/html/2607.05461#bib.bib3)\]\(10K test images\); four architectures:ResNet\-20\[[4](https://arxiv.org/html/2607.05461#bib.bib4)\],VGG\-16\[[5](https://arxiv.org/html/2607.05461#bib.bib5)\],DenseNet\-121\[[6](https://arxiv.org/html/2607.05461#bib.bib6)\], andShuffleNetV2\[[7](https://arxiv.org/html/2607.05461#bib.bib7)\]; and three model quality levels for ResNet\-20 on CIFAR\-10: low \(∼\\sim70%, 2,961 faults\), mid \(∼\\sim79%, 2,117 faults\), and high \(∼\\sim88%, 1,196 faults\)\. Table[II](https://arxiv.org/html/2607.05461#S5.T2)summarizes the configurations\.

TABLE II:Dataset–Architecture ConfigurationsDefault parameters:τ=0\.05\\tau\{=\}0\.05,W=20W\{=\}20,Nmin=50N\_\{\\min\}\{=\}50\. We compare against DeepGini at fixed budgets \(k∈\{1,2,5,10,20,50,100\}%k\\in\\\{1,2,5,10,20,50,100\\\}\\%\), an oracle \(perfect fault\-first ordering\), and five alternative stopping criteria: patience\-5, consecutive non\-faults \(k=50,100k\{=\}50,100\), confidence\-90%, and cumulative rate\. For RQ6, we test eight selection strategies: DeepGini, entropy, margin, boundary distance, random, TestRank\[[10](https://arxiv.org/html/2607.05461#bib.bib10)\], ATS\[[11](https://arxiv.org/html/2607.05461#bib.bib11)\], and DeepGD\[[16](https://arxiv.org/html/2607.05461#bib.bib16)\]\.

Metrics:\(1\)Budget used \(%\):fraction of the test pool labeled before stopping\. \(2\)Fault recall \(%\):fraction of total faults discovered\. \(3\)Efficiency:ratio of faults found to labels used\. \(4\)Net value:v×faults−c×budgetv\\times\\text\{faults\}\-c\\times\\text\{budget\}withv=20v\{=\}20,c=1c\{=\}1\.

### V\-CExperimental Procedure

For each configuration, we compute model predictions and Gini scores, sort inputs by score, iterate through the ranked list while updating the sliding\-window estimate, and apply the stopping rule onceNminN\_\{\\min\}samples have been collected\. For baseline comparisons, we run DeepGini at each fixed budget and record the same metrics\.

## VIExperimental Results

### VI\-ARQ1: Effectiveness

Table[III](https://arxiv.org/html/2607.05461#S6.T3)compares AdaStop against exhaustive testing on CIFAR\-10/ResNet\-20\.

TABLE III:AdaStop vs\. Exhaustive Testing \(τ=0\.05\\tau\{=\}0\.05, CIFAR\-10\)Figure[2](https://arxiv.org/html/2607.05461#S6.F2)shows the budget\-recall trade\-off\. AdaStop automatically finds an effective operating point on the budget\-recall curve without manual budget tuning\.

01010202030304040505060607070808090901001000505010010079\.0% recall23\.5% budgetBudget Used \(%\)Fault Recall \(%\)DeepGini@kk%AdaStop \(τ\\tau=0\.05\)Oracle \(perfect\)Figure 2:Budget\-recall trade\-off\. AdaStop \(star\) identifies stopping points close to the optimal point on the budget–recall curve\.Figure[3](https://arxiv.org/html/2607.05461#S6.F3)compares net value across methods\. The oracle achieves peak net value at exactly 11\.96% budget \(the error rate\), where all faults are found with zero waste\. DeepGini peaks at 50% and then decreases as costs outpace discovery\. AdaStop achieves 90% of DeepGini’s peak value while using 27 percentage points less budget\. Notably, exhaustive testing yieldslowernet value \(13,920\) than AdaStop \(16,552\), demonstrating that more testing is not always better\.

0101020203030404050506060707080809090100100010 00010\\,00020 00020\\,000Oracle peak16,552Budget Used \(%\)Net Value \(v⋅faults−c⋅budgetv\\cdot\\text\{faults\}\-c\\cdot\\text\{budget\}\)Oracle \(perfect\)DeepGini@kk%AdaStop \(τ\\tau=0\.05\)Figure 3:Net value comparison \(v=20v\{=\}20,c=1c\{=\}1\)\. Oracle peaks at 11\.96% budget\. DeepGini peaks at 50% then decreases\. Exhaustive testing \(100%\) yields thelowestnet value among non\-trivial budgets\.RQ1:AdaStop achieves 79\.0% fault recall using only 23\.5% of the budget, with a 3\.35×\\timesefficiency improvement and 18\.9% higher net value than exhaustive testing\.

### VI\-BRQ2: Threshold Selection

Table[IV](https://arxiv.org/html/2607.05461#S6.T4)shows threshold sensitivity\.

TABLE IV:Threshold Selection Results \(CIFAR\-10, ResNet\-20\)RQ2:Thresholdsτ=0\.01\\tau\{=\}0\.01–0\.050\.05converge to the same stopping point \(23\.5% budget, 79\.0% recall\), whileτ=0\.10\\tau\{=\}0\.10–0\.200\.20converge to an earlier point \(13\.2% budget, 54\.7% recall\)\. This stepped behavior indicates the fault rate exhibits sharp drops rather than smooth decline, creating natural stopping plateaus where multiple threshold values trigger at similar points\.

### VI\-CRQ3: Stopping Criteria Comparison

Table[V](https://arxiv.org/html/2607.05461#S6.T5)compares stopping criteria, all using DeepGini for selection\.

TABLE V:Stopping Criteria Comparison \(CIFAR\-10, ResNet\-20\)The criteria form a clear aggressive\-to\-conservative spectrum\. Threshold is most efficient; patience\-5 adds a small safety margin; consecutive\-50 achieves the best net value; and consecutive\-100 and confidence\-90 are conservative, achieving≥\\geq98\.8% recall at higher cost\. The cumulative rate criterion never triggers because the cumulative average is too slow to reflect marginal rate decline\.

RQ3:For maximum efficiency, use threshold\-based stopping\. For best net value, consecutive\-50 is optimal\. For near\-complete recall, confidence\-90 provides 99\.0% recall with 40% savings\.

### VI\-DRQ4: Sensitivity to Window Size

TABLE VI:Window Size Sensitivity \(τ=0\.05\\tau\{=\}0\.05, CIFAR\-10\)Small windows \(W=10W\{=\}10\) cause premature stopping;W=20W\{=\}20andW=50W\{=\}50converge to the same point, showing robustness for moderate sizes\. The minimum\-samples parameterNminN\_\{\\min\}is also robust: values of 20, 50, and 100 all produce identical results because the natural stopping point \(2,348 samples\) far exceedsNminN\_\{\\min\}\.

### VI\-ERQ5: Generalization

#### VI\-E1Cross\-Dataset

AdaStop consistently achieves substantial budget savings \(77–91%\) across all datasets while maintaining 65–79% recall\.

TABLE VII:Cross\-Dataset Results \(ResNet\-20,τ=0\.05\\tau\{=\}0\.05\)
#### VI\-E2Cross\-Architecture

TABLE VIII:Cross\-Architecture Results \(CIFAR\-10,τ=0\.05\\tau\{=\}0\.05\)Across architectures, the 18–30% budget range and the 67–84% recall range confirm that the stopping criterion adapts to varying error rates without parameter changes\.

RQ5:AdaStop generalizes well: 65–84% recall with 70–91% savings across 3 datasets and 4 architectures\.

### VI\-FRQ6: Strategy\-Agnostic Property

TABLE IX:Strategy\-Agnostic Evaluation \(CIFAR\-10,τ=0\.05\\tau\{=\}0\.05\)Six of the eight strategies produce remarkably consistent stopping behavior: 23–24% budget, 79–81% recall, and 0\.39–0\.40 efficiency\. This consistency occurs because all six strategies exploit model uncertainty to order inputs, producing similar diminishing\-returns curves that crossτ\\tauat nearly the same point\.

RQ6:AdaStop is strategy\-agnostic—all uncertainty\-based strategies produce consistent stopping points\. DeepGD stops earlier \(12%\) due to a steeper initial rate decline from its diversity component; random stops almost immediately because its unprioritized fault rate fluctuates enough to crossτ\\tausoon after the initial window\.

### VI\-GRQ7: Effect of Model Quality

TABLE X:Model Quality Sensitivity \(CIFAR\-10, ResNet\-20\)As model quality improves \(fewer faults\), AdaStop uses less budget and achieves higher savings\. For the low\-quality model \(30% error rate\), the fault rate remains aboveτ\\taufor longer, while for the high\-quality model \(12% error rate\), it drops belowτ\\tauat 23\.5%\. The stopping point therefore varies naturally with fault density\.

RQ7:AdaStop adapts to model quality: more budget for worse models \(64\.7%\) and less for better ones \(23\.5%\), with consistent efficiency \(∼\\sim0\.40–0\.44\)\.

### VI\-HDiminishing Returns Validation

The diminishing returns property is validated via APFD scores and Mann\-Kendall monotonicity tests \(Table[XI](https://arxiv.org/html/2607.05461#S6.T11)\)\.

TABLE XI:Diminishing Returns: APFD and Mann\-Kendall TestsAll guided strategies show strong decreasing trends \(MKτ\\taufrom−0\.51\-0\.51to−0\.74\-0\.74, allp<10−4p<10^\{\-4\}\), confirming the diminishing returns assumption\. Random selection shows near\-zero trend \(\|τ\|<0\.04\|\\tau\|<0\.04\), confirming no meaningful trend as expected—without prioritization, the fault rate remains roughly constant at the base error rate\.

### VI\-ISummary of Key Findings

1. 1\.Cost\-aware stopping improves labeling efficiency: AdaStop withτ=0\.05\\tau\{=\}0\.05achieves 79\.0% recall using 23\.5% of the budget—a 3\.35×\\timesefficiency gain with 18\.9% higher net value than exhaustive testing\.
2. 2\.Threshold exhibits stepped behavior:τ=0\.01\\tau\{=\}0\.01–0\.050\.05converge to 23\.5% budget;τ=0\.10\\tau\{=\}0\.10–0\.200\.20converge to 13\.2%, reflecting sharp fault rate drops\.
3. 3\.Multiple criteria suit different needs: threshold for efficiency, consecutive\-50 for best net value, confidence\-90 for near\-complete recall\.
4. 4\.Window size matters:W≥20W\\geq 20is required for stable estimates;W=10W\{=\}10causes premature stopping \(30% recall vs\. 79%\)\.
5. 5\.Generalizes broadly: consistent 65–84% recall with 70–91% savings across 3 datasets, 4 architectures, and 8 strategies\.
6. 6\.Adapts to model quality: uses 23–65% budget depending on error rate, without parameter changes\.

### VI\-JLimitations

Our evaluation focuses on image classification across three vision datasets; generalization to NLP, tabular, or multi\-modal domains remains to be validated\. AdaStop assumes labels are obtained one at a time; batch labeling scenarios \(where multiple inputs are sent for annotation simultaneously\) would require extending the framework\. The thresholdτ=c/v\\tau=c/vrequires practitioners to estimate the labeling cost and fault value; in practice these may be uncertain, though the relative and consecutive stopping criteria partially address this by not requiring explicit cost\-benefit ratios\. Finally, while the diminishing returns assumption is validated empirically via Mann\-Kendall tests across three datasets, it may not hold for models with multi\-modal failure clusters or adversarial inputs where faults appear in unexpected regions of the uncertainty space\.

## VIIConclusion

We presented AdaStop, a cost\-aware framework for determining when to stop labeling in DNN test selection\. By formalizing the stopping problem as a cost\-benefit optimization, we derived the optimal stopping thresholdτ=c/v\\tau=c/vand implemented it through a sliding window fault rate estimator\. Our comprehensive evaluation across 3 datasets \(CIFAR\-10, SVHN, FashionMNIST\), 4 architectures \(ResNet\-20, VGG\-16, DenseNet\-121, ShuffleNetV2\), 3 model quality levels, and 8 selection strategies demonstrates that AdaStop consistently achieves 65–84% fault recall with 70–91% budget savings\. The framework is strategy\-agnostic, producing consistent stopping behavior regardless of the underlying selection method\. Among the stopping criteria evaluated, consecutive non\-faults \(k=50k\{=\}50\) achieves the best net value, while threshold\-based stopping offers the highest efficiency\. The diminishing returns assumption underlying our approach is formally validated via Mann\-Kendall tests across all configurations\. Future work will extend AdaStop to non\-vision domains \(NLP, tabular data\), batch labeling scenarios\.

## References

- \[1\]A\. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” Tech\. Rep\., University of Toronto, 2009\.
- \[2\]Y\. Netzer et al\., “Reading Digits in Natural Images with Unsupervised Feature Learning,” inNIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011\.
- \[3\]H\. Xiao, K\. Rasul, and R\. Vollgraf, “Fashion\-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms,” arXiv:1708\.07747, 2017\.
- \[4\]K\. He, X\. Zhang, S\. Ren, and J\. Sun, “Deep Residual Learning for Image Recognition,” inProc\. CVPR, 2016\.
- \[5\]K\. Simonyan and A\. Zisserman, “Very Deep Convolutional Networks for Large\-Scale Image Recognition,” inProc\. ICLR, 2015\.
- \[6\]G\. Huang, Z\. Liu, L\. van der Maaten, and K\. Q\. Weinberger, “Densely Connected Convolutional Networks,” inProc\. CVPR, 2017\.
- \[7\]N\. Ma, X\. Zhang, H\.\-T\. Zheng, and J\. Sun, “ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design,” inProc\. ECCV, 2018\.
- \[8\]E\. B\. Wilson, “Probable Inference, the Law of Succession, and Statistical Inference,”J\. Amer\. Statist\. Assoc\., vol\. 22, no\. 158, pp\. 209–212, 1927\.
- \[9\]Y\. Feng et al\., “DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networks,” inProc\. ISSTA, 2020\.
- \[10\]Z\. Li et al\., “TestRank: Bringing order into unlabeled test instances for deep learning tasks,” inProc\. NeurIPS, 2021\.
- \[11\]X\. Gao et al\., “Adaptive Test Selection for Deep Neural Networks,” inProc\. ICSE, 2022\.
- \[12\]A\. Guerriero, R\. Pietrantuono, and S\. Russo, “DeepSample: DNN Sampling\-Based Testing for Operational Accuracy Assessment,” inProc\. ICSE, 2024\.
- \[13\]M\. Bloodgood and K\. Vijay\-Shanker, “A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User\-Adjustable Stopping,” inProc\. CoNLL, 2009\.
- \[14\]H\. Ishibashi and H\. Hino, “Stopping Criterion for Active Learning Based on Deterministic Generalization Bounds,” inProc\. AISTATS, 2020\.
- \[15\]J\. Chen et al\., “FAST: Boosting Uncertainty\-based Test Prioritization Methods for Neural Networks via Feature Selection,” inProc\. ASE, 2024\.
- \[16\]Z\. Aghababaeyan et al\., “DeepGD: A Multi\-Objective Black\-Box Test Selection Approach for Deep Neural Networks,” inProc\. FSE, 2024\.
- \[17\]H\. Zheng et al\., “CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space,” inProc\. ASE, 2023\.
- \[18\]J\. Boetje and R\. van de Schoot, “The SAFE Procedure: A Practical Stopping Heuristic for Active Learning\-Based Screening in Systematic Reviews and Meta\-Analyses,”Systematic Reviews, vol\. 13, art\. 81, 2024\.
- \[19\]Y\. Yuan, Q\. Pang, and S\. Wang, “Revisiting Neuron Coverage for DNN Testing: A Layer\-Wise and Distribution\-Aware Criterion,” inProc\. ICSE, 2023\.
- \[20\]H\. Ishibashi and H\. Hino, “Stopping Criterion for Active Learning Based on Error Stability,” arXiv:2104\.01836, 2021\.
- \[21\]M\. P\. Bron et al\., “Using Chao’s Estimator as a Stopping Criterion for Technology\-Assisted Review,” arXiv:2404\.01176, 2024\.
- \[22\]J\. Zhou et al\., “Cost\-Effective Testing of a Deep Learning Model through Input Reduction,” inProc\. ISSRE, 2020\.
- \[23\]D\. Mittal et al\., “Adaptive Labeling for Efficient Out\-of\-distribution Model Evaluation,” inProc\. NeurIPS, 2024\.
- \[24\]Y\. Tian et al\., “DeepTest: Automated Testing of Deep\-Neural\-Network\-Driven Autonomous Cars,” inProc\. ICSE, 2018\.
- \[25\]A\. Esteva et al\., “Dermatologist\-level classification of skin cancer with deep neural networks,”Nature, vol\. 542, pp\. 115–118, 2017\.
- \[26\]A\. M\. Ozbayoglu et al\., “Deep Learning for Financial Applications: A Survey,” arXiv:2002\.05786, 2020\.
- \[27\]B\. Settles, “Active Learning Literature Survey,” Tech\. Rep\. 1648, University of Wisconsin\-Madison, 2010\.