Hard Cases, Bad Labels: Testing Error Exposure and Error Location in Uncertainty Sampling Under Bounded Label Noise

# Hard Cases, Bad Labels Testing Error Exposure and Error Location in Uncertainty Sampling Under Bounded Label Noise
Source: [https://arxiv.org/html/2608.13601](https://arxiv.org/html/2608.13601)
\(July 2026\)

###### Abstract

Active learning can reduce labeling cost by requesting examples that appear informative, but the most uncertain examples may also be the hardest to label correctly\. This motivates two distinct explanations for failure: uncertainty sampling may acquire more corrupted labels, and errors concentrated in difficult regions may remain more harmful even after acquired corruption is aligned\. This study tests both explanations by comparing margin\-based uncertainty sampling with random sampling under clean labels, random classification noise \(RCN\), and bounded difficulty\-dependent noise on three public binary tabular datasets\. The design uses 100 paired seeds, nine expected noise rates from 0 to 0\.30, annotation budgets from 20 to 120, and logistic regression with regularization re\-selected by cross\-validation at every budget\. A leave\-one\-seed\-out exposure\-matched RCN control tests whether performance differences remain after aligning mean final acquired corruption; a clean\-label extension reaches budget 400\. Under clean labels, uncertainty sampling improved normalized balanced\-accuracy area under the learning curve \(AULC\) by 1\.09 to 1\.77 percentage points on all three datasets, with every comparison surviving Holm correction\. Difficulty\-dependent noise reduced this advantage more than RCN at six of eight rates on Breast Cancer Wisconsin, but at no tested rate on Banknote Authentication or MAGIC Gamma Telescope\. The exposure\-matched analysis found no Holm\-corrected evidence for the predicted additional harm of structured error location; two MAGIC comparisons were significant in the opposite direction\. No mean AULC advantage crossed below zero through 0\.30 noise\. On clean MAGIC data, uncertainty sampling improved balanced accuracy while reducing average precision and true\-positive rate at fixed false\-positive rates\. Uncertainty sampling was label\-efficient, but its apparent robustness depended on dataset, budget, noise structure, and metric\.

## 1 Introduction

Many machine\-learning projects begin with more unlabeled examples than can be reviewed by experts\. Active learning addresses this problem by allowing a model to choose which examples should be labeled next\[[18](https://arxiv.org/html/2608.13601#bib.bib5)\]\. A common method, margin\-based uncertainty sampling, requests examples closest to the learner’s current decision boundary\[[11](https://arxiv.org/html/2608.13601#bib.bib1)\]\. These examples can be informative because their labels may change where the boundary is placed\.

The same examples can also be difficult for annotators\. Human label quality may vary by annotator, domain, and individual example\[[22](https://arxiv.org/html/2608.13601#bib.bib6),[5](https://arxiv.org/html/2608.13601#bib.bib7),[8](https://arxiv.org/html/2608.13601#bib.bib20)\]\. If uncertainty sampling repeatedly requests ambiguous cases, it may acquire a higher concentration of incorrect labels than random sampling\. This creates a tension: the examples with the greatest potential information may also carry the greatest annotation risk\.

This tension contains two mechanisms\. First,*exposure*: uncertainty sampling may acquire a larger fraction of corrupted labels because it queries where error probability is elevated\. Second,*location*: after acquired corruption is aligned, a structured placement of errors may still be more damaging than a random placement\. Standard end\-to\-end comparisons combine these mechanisms\. Distinguishing them matters because exposure can motivate repeated labeling or escalation, whereas a residual location effect would implicate the acquisition rule itself\.

This study provides a controlled paired benchmark on three binary tabular datasets\. It distinguishes constant\-probability random classification noise \(RCN\) from a bounded difficulty\-dependent process whose error probability increases near a fixed simulation\-only reference boundary\. It then compares the structured process with an independent RCN condition calibrated to align mean final acquired corruption\. This control does not perfectly isolate location within each seed or across the full trajectory; it tests whether a residual performance difference remains after exposure alignment\. The goal is not to reproduce all human disagreement or propose a new query strategy\. It is to measure when a simple uncertainty baseline helps, when its advantage changes, and which explanations survive replication across datasets\.

### 1\.1 Contributions

This study makes four bounded contributions:

- •It evaluates uncertainty and random sampling across complete low\-budget learning curves using paired seeds and budget\-specific regularization selection\.
- •It compares constant\-probability RCN with a reproducible, difficulty\-dependent noise process while reporting results separately by dataset\.
- •It uses a leave\-one\-seed\-out exposure\-matched control to test whether performance differences remain after aligning mean final acquired corruption\.
- •It identifies a metric reversal on clean MAGIC data: uncertainty sampling improves balanced accuracy while reducing average precision and TPR at every pre\-specified FPR limit\.

### 1\.2 Research Questions and Hypotheses

The central research question is:

> How do random classification noise and bounded difficulty\-dependent label noise affect the relative performance of uncertainty sampling and random sampling across annotation budgets, and do performance differences remain after aligning acquired corruption exposure?

Four hypotheses were fixed before scientific outcomes were accessed:

- •H1:Under clean labels, uncertainty sampling has higher normalized balanced\-accuracy AULC than random sampling\.
- •H2:As noise increases, the uncertainty\-minus\-random AULC advantage shrinks more under difficulty\-dependent noise than under RCN\.
- •H3:The uncertainty\-minus\-random performance difference changes across annotation budgets\. This trajectory analysis was pre\-specified as descriptive and received no pointwise hypothesis tests\.
- •H4:At budget 120, uncertainty sampling under difficulty\-dependent noise performs worse than under RCN matched to the fraction of corrupted labels it acquires\.

## 2 Related Work

### 2\.1 Active Learning and Evaluation Across Budgets

Lewis and Gale demonstrated that sequentially selecting examples could reduce the labeled data needed for text classification\[[11](https://arxiv.org/html/2608.13601#bib.bib1)\]\. Settles later organized uncertainty, disagreement, and representativeness methods within the broader active\-learning literature\[[18](https://arxiv.org/html/2608.13601#bib.bib5)\]\. However, an active method’s measured advantage can depend on training choices, randomness, dataset, and budget\. Munjal et al\. showed that regularization and training settings can change active\-learning conclusions\[[14](https://arxiv.org/html/2608.13601#bib.bib8)\], while Ji et al\. emphasized repeated, controlled evaluation\[[10](https://arxiv.org/html/2608.13601#bib.bib9)\]\. Bae et al\. further showed that a method effective at one label budget can underperform random sampling in another\[[3](https://arxiv.org/html/2608.13601#bib.bib10)\]\. These findings motivate paired seeds, budget\-dependent regularization selection, and complete learning\-curve analysis\.

### 2\.2 Random and Input\-Dependent Label Noise

Under RCN, each binary label is independently flipped with one constant probability below one half\[[1](https://arxiv.org/html/2608.13601#bib.bib2)\]\. Massart noise permits this probability to depend on the input while retaining an upper bound below one half\[[13](https://arxiv.org/html/2608.13601#bib.bib3),[2](https://arxiv.org/html/2608.13601#bib.bib4)\]\. General Massart noise can be adversarial\. The present study uses a narrower, non\-adversarial process in which error probability is a fixed monotone function of difficulty relative to an estimated reference boundary\. Because that boundary is not the Bayes boundary and need not match the active learner’s boundary, the process is called*bounded difficulty\-dependent noise*\. It is inspired by bounded input\-dependent noise but is not claimed to simulate arbitrary Massart noise or inherit theoretical guarantees from that setting\.

Recent work has addressed active learning with imperfect annotation\. Nuggehalli et al\. study active learning when imbalance and label noise occur together\[[15](https://arxiv.org/html/2608.13601#bib.bib11)\]\. Shafir et al\. propose a noise\-aware low\-budget framework and evaluate multiple noise processes on image benchmarks\[[19](https://arxiv.org/html/2608.13601#bib.bib12)\]\. The present experiment complements this work with standard logistic regression, tabular datasets, explicit exposure matching, and paired comparisons across the complete annotation path\.

### 2\.3 Metrics Under Class Imbalance

AUROC averages performance across false\-positive operating regions that may not match actual use\. Precision–recall analysis can expose differences that appear small in ROC space, particularly with class imbalance\[[6](https://arxiv.org/html/2608.13601#bib.bib13),[17](https://arxiv.org/html/2608.13601#bib.bib14)\]\. This study therefore excludes AUROC and instead reports balanced accuracy, average precision, and true\-positive rate \(TPR\) at pre\-specified false\-positive\-rate \(FPR\) limits\. MAGIC is additionally evaluated at 1% and 2% FPR because its documentation states that accepting a background event as signal is more costly than rejecting a signal event and identifies 1%, 2%, 5%, and 10% as relevant operating thresholds\[[4](https://arxiv.org/html/2608.13601#bib.bib17)\]\.

## 3 Methods

### 3\.1 Datasets and Splits

Three public UCI binary\-classification datasets were selected to vary in size, dimensionality, class balance, and domain \(Table[1](https://arxiv.org/html/2608.13601#S3.T1)\)\.

Table 1:Benchmark datasets, mean pool positive\-class rates under the stratified splits, and pre\-specified fixed\-FPR operating points\.For each of 100 seeds, each dataset received an 80/20 stratified split into an active\-learning pool and a clean test set\. Numerical features were standardized using the pool features only\. Test labels were excluded from training, query selection, cross\-validation, and synthetic\-noise construction\. All comparisons remained paired within dataset and seed; raw metric values were never pooled across datasets\.

### 3\.2 Learner and Query Strategies

The learner was logistic regression withL2L\_\{2\}regularization, theliblinearsolver, and no class weighting, implemented with scikit\-learn\[[16](https://arxiv.org/html/2608.13601#bib.bib21)\]\. At every annotation budget and separately for each query strategy, inverse regularization strength was selected from

C∈\{10−3,10−2,10−1,1,10,100,1000\}C\\in\\\{10^\{\-3\},10^\{\-2\},10^\{\-1\},1,10,100,1000\\\}
using three\-fold seeded stratified cross\-validation on only the labels acquired at that point\. Cross\-validation optimized balanced accuracy and broke exact ties in favor of the smallerCC\. The model was then retrained from scratch\.

Each trial began with the same stratified set of 20 trusted clean labels, including at least three examples from each class\. Models acquired labels in batches of five through budget 120, producing 21 recorded checkpoints\. Random sampling selected uniformly without replacement\. Uncertainty sampling selected the five remaining examples with the smallest current learner margin

milearner=\|2​p^i−1\|,m\_\{i\}^\{\\text\{learner\}\}=\\left\|2\\widehat\{p\}\_\{i\}\-1\\right\|,
with ties broken by original pool index\.

### 3\.3 Independent Reference Difficulty

Synthetic difficulty\-dependent error required a measure independent of the logistic active learner\. For each split, a random\-forest reference model generated one clean out\-of\-fold probability for every pool example using five\-fold stratified cross\-fitting\. Each forest used 500 trees, minimum leaf size five, square\-root feature subsampling, bootstrap sampling, and a fixed seed rule\. No example’s difficulty was predicted by a forest trained on that example\. Reference difficulty was

di=1−\|2​piOOF−1\|,d\_\{i\}=1\-\\left\|2p\_\{i\}^\{\\mathrm\{OOF\}\}\-1\\right\|,
where larger values indicate proximity to the reference boundary\. The reference was frozen before any active\-learning query and used only to construct synthetic annotation error\. It was unavailable to the learner and was not used for evaluation\. Consequently,did\_\{i\}measures proximity to the random\-forest reference boundary, not necessarily to the logistic learner’s own boundary\. This distinction limits mechanistic interpretations of the structured\-noise results\.

### 3\.4 Noise Processes

The clean initial set was never corrupted\. For the remaining pool, expected noise rates were

ρ∈\{0,\.025,\.05,\.075,\.10,\.15,\.20,\.25,\.30\}\.\\rho\\in\\\{0,\.025,\.05,\.075,\.10,\.15,\.20,\.25,\.30\\\}\.
Under RCN, every eligible label flipped independently:

Bi∼Bernoulli⁡\(ρ\),y~i=yi⊕Bi\.B\_\{i\}\\sim\\operatorname\{Bernoulli\}\(\\rho\),\\qquad\\widetilde\{y\}\_\{i\}=y\_\{i\}\\oplus B\_\{i\}\.
Under bounded difficulty\-dependent noise, eligible flip probabilities were

qi=ρ\+γρ​\(di−d¯\),q\_\{i\}=\\rho\+\\gamma\_\{\\rho\}\(d\_\{i\}\-\\bar\{d\}\),
γρ=0\.90​min⁡\{ρd¯,0\.45−ρ1−d¯\}\.\\gamma\_\{\\rho\}=0\.90\\min\\left\\\{\\frac\{\\rho\}\{\\bar\{d\}\},\\frac\{0\.45\-\\rho\}\{1\-\\bar\{d\}\}\\right\\\}\.
This centered linear construction preserves mean expected errorρ\\rho, increases monotonically with difficulty, and keeps every probability below 0\.45\. A constant\-difficulty pool falls back to RCN\. Labels then flipped independently asBi∼Bernoulli⁡\(qi\)B\_\{i\}\\sim\\operatorname\{Bernoulli\}\(q\_\{i\}\)\. Within each paired condition, random and uncertainty sampling shared the same pre\-generated noise map, allowing their acquired corruption exposure to differ only because they queried different examples\. RCN and difficulty\-dependent maps at the same dataset, seed, and expected rate used deterministic but independent random streams\. H2 therefore compares independently realized noise processes rather than applying two thresholds to one shared uniform draw\.

### 3\.5 Exposure\-Matched and Extended Controls

H2 can reflect two mechanisms: uncertainty sampling may acquire more corrupted labels, or corrupted labels near the boundary may be more harmful even at the same exposure\. H4 separated these possibilities with an exposure\-matched RCN control\. For each dataset, source noise rate, and seed, the RCN target was the mean final acquired\-corruption fraction from the other 99 difficulty\-dependent uncertainty trials\. The leave\-one\-seed\-out target was capped at 0\.45 and used an independent noise stream\. This generated 2,400 matched\-control trials\. The control aligns final exposure in expectation across seeds rather than matching it exactly within each seed, and it does not match the exposure trajectory before budget 120\. H4 therefore tests whether a residual performance difference remains after mean final exposure alignment; it does not perfectly isolate error location\.

A secondary clean\-label experiment extended Breast Cancer Wisconsin from budget 120 to 400\. Both query strategies were run for the same 100 seeds, producing 200 trials and 35 retained checkpoints\. Its first 21 checkpoints were required to match the main experiment exactly\.

### 3\.6 Metrics and Statistical Analysis

The primary outcome was normalized area under the balanced\-accuracy learning curve from budgets 20 to 120, calculated by trapezoidal integration\. Secondary outcomes were final balanced accuracy, average precision, TPR at fixed FPR, and acquired\-corruption fraction\. At FPR limitα\\alpha, the reported TPR was

maxτ:FPR⁡\(τ\)≤α⁡TPR⁡\(τ\)\.\\max\_\{\\tau:\\operatorname\{FPR\}\(\\tau\)\\leq\\alpha\}\\operatorname\{TPR\}\(\\tau\)\.
The seed within dataset was the paired analysis unit\. Mean paired differences received 10,000 paired\-seed percentile bootstrap resamples and 95% confidence intervals\[[7](https://arxiv.org/html/2608.13601#bib.bib22)\]\. Two\-sided Wilcoxon signed\-rank tests\[[20](https://arxiv.org/html/2608.13601#bib.bib18)\]were secondary to estimates and intervals\. Holm correction\[[9](https://arxiv.org/html/2608.13601#bib.bib19)\]was performed separately within H1 \(3 tests\), H2 \(24 tests\), and H4 \(24 tests\)\. H3 budget intervals and final secondary metrics were descriptive and received no pointwisepp\-values\.

H2 used the difference\-in\-differences

Dρ=\(AULCU−AULCR\)difficulty−\(AULCU−AULCR\)RCN,D\_\{\\rho\}=\\left\(\\mathrm\{AULC\}\_\{U\}\-\\mathrm\{AULC\}\_\{R\}\\right\)\_\{\\mathrm\{difficulty\}\}\-\\left\(\\mathrm\{AULC\}\_\{U\}\-\\mathrm\{AULC\}\_\{R\}\\right\)\_\{\\mathrm\{RCN\}\},
where negative values match the predicted direction\. H4 used difficulty\-dependent minus exposure\-matched RCN balanced accuracy at budget 120\. AULC robustness boundaries were defined as the first adjacent linear zero\-crossing of the mean paired uncertainty\-minus\-random curve over\[0,0\.30\]\[0,0\.30\]; absent crossings were reported as censored rather than extrapolated\.

The analysis plan and implementation were frozen internally in the repository after all three experiment components passed count\-only structural audits and before scientific outcomes were accessed\. This was a repository\-recorded freeze, not an external preregistration\. The final inventory contained 12,800 trials and 271,600 retained budget rows\.

## 4 Results

### 4\.1 H1: Clean\-Label Efficiency

H1 was supported on all three datasets \(Figure[1](https://arxiv.org/html/2608.13601#S4.F1)\)\. Uncertainty sampling increased normalized balanced\-accuracy AULC by 1\.45 percentage points on Banknote \(95% CI 1\.28 to 1\.64; Holm\-adjustedp=1\.89×10−17p=1\.89\\times 10^\{\-17\}\), 1\.77 points on Breast Cancer \(95% CI 1\.47 to 2\.11; adjustedp=5\.85×10−16p=5\.85\\times 10^\{\-16\}\), and 1\.09 points on MAGIC \(95% CI 0\.44 to 1\.70; adjustedp=1\.53×10−4p=1\.53\\times 10^\{\-4\}\)\.

![Refer to caption](https://arxiv.org/html/2608.13601v1/x1.png)Figure 1:Paired clean\-label uncertainty\-minus\-random normalized balanced\-accuracy AULC differences\. Points are means across 100 paired seeds; bars are 95% paired\-bootstrap intervals\. Positive values favor uncertainty sampling\.
### 4\.2 H2: Noise Structure

H2 received strong but dataset\-specific support \(Figure[2](https://arxiv.org/html/2608.13601#S4.F2)\)\. On Breast Cancer, the difficulty\-minus\-RCN difference\-in\-differences was negative and Holm\-significant at six consecutive rates from 0\.025 through 0\.20\. Mean effects ranged from−0\.84\-0\.84to−1\.26\-1\.26percentage points\. The 0\.25 comparison was inconclusive\. At 0\.30, the bootstrap interval remained below zero, but the Wilcoxon result did not survive Holm correction\.

Banknote showed no corrected difference at any rate: six of eight point estimates were negative, but every interval crossed zero\. MAGIC showed only two negative point estimates and no corrected differences\. Therefore, the hypothesis that difficulty\-dependent noise generally reduces uncertainty sampling’s advantage more than RCN was not supported across datasets\.

![Refer to caption](https://arxiv.org/html/2608.13601v1/x2.png)Figure 2:H2 difference\-in\-differences in normalized balanced\-accuracy AULC\. Negative values indicate that the uncertainty\-minus\-random advantage was smaller under difficulty\-dependent noise than under RCN\. Shading shows 95% paired\-bootstrap intervals; orange points survived Holm correction within the 24\-test H2 family\.Despite the relative degradation on Breast Cancer, no point\-estimate AULC curve crossed below zero within the tested domain\. All six dataset\-process crossing estimates were right\-censored above 0\.30\. Thus, the results show reduced advantage in some conditions, not general failure of uncertainty sampling through 30% expected noise\.

### 4\.3 H3: Budget Dependence and the Extended Clean Curve

Budget\-level results were descriptive\. Banknote’s mean balanced\-accuracy advantage was positive at every post\-initial checkpoint under every process and rate\. Breast Cancer remained mostly positive, but under difficulty\-dependent noise at rate 0\.30 it was positive at 13 of 20 post\-initial checkpoints and ended at−0\.24\-0\.24percentage points \(95% CI−1\.33\-1\.33to 0\.91\)\. MAGIC was more variable across budgets\. No budget\-level interval on any dataset was strictly below zero\.

The extended Breast Cancer experiment found its first persistent positive mean difference at budget 25, the earliest post\-seed checkpoint \(Figure[3](https://arxiv.org/html/2608.13601#S4.F3)\)\. The mean advantage peaked at 2\.47 percentage points at budget 35, declined as random sampling caught up, and remained positive at budget 400 \(0\.27 points; 95% CI 0\.06 to 0\.49\)\. The earliest persistent\-positive point estimate occurred at budget 25, but the corresponding bootstrap interval spanned 25 to 400\. The curve, rather than a precise onset estimate, is therefore the interpretable result\.

![Refer to caption](https://arxiv.org/html/2608.13601v1/x3.png)Figure 3:Descriptive paired uncertainty\-minus\-random balanced\-accuracy difference on clean Breast Cancer Wisconsin data through budget 400\. The line is the mean across 100 paired seeds and the band is a 95% paired\-bootstrap interval\.
### 4\.4 H4: Error Location Beyond Exposure

The matched control aligned mean final acquired exposure closely \(Table[2](https://arxiv.org/html/2608.13601#S4.T2)\)\. Across the eight nonzero source rates, every paired\-bootstrap interval for the exposure difference included zero, and the 0\.45 target cap was never activated\. This is evidence of successful alignment at the dataset–rate level, not exact matching within every seed\.

Table 2:Exposure alignment for H4 across nonzero source rates\. Differences are difficulty\-dependent minus exposure\-matched RCN acquired corruption fractions at budget 120, summarized across the eight rates per dataset\. The ranges summarize means and do not establish statistical equivalence\.H4 was not supported \(Figure[4](https://arxiv.org/html/2608.13601#S4.F4)\)\. No comparison was Holm\-significant in the predicted negative direction\. Banknote remained near zero\. Breast Cancer had negative intervals at rates 0\.025 and 0\.05, but neither corresponding rank test survived correction\.

MAGIC instead moved mainly in the opposite direction\. Difficulty\-dependent noise outperformed exposure\-matched RCN by 1\.38 percentage points at rate 0\.10 \(95% CI 0\.57 to 2\.21; adjustedp=0\.0128p=0\.0128\) and by 1\.89 points at rate 0\.15 \(95% CI 0\.95 to 2\.85; adjustedp=0\.00634p=0\.00634\)\. These results reject a simple universal claim that errors concentrated near the reference boundary must be more harmful than randomly located errors with similar mean final exposure\. They do not establish equivalence or identify why MAGIC moved in the opposite direction\.

![Refer to caption](https://arxiv.org/html/2608.13601v1/x4.png)Figure 4:H4 balanced\-accuracy differences at budget 120\. Values are difficulty\-dependent uncertainty sampling minus uncertainty sampling under exposure\-matched RCN\. Negative values match H4’s predicted direction\. Shading shows 95% paired\-bootstrap intervals; orange points indicate two\-sided Holm\-significant results and occur in the opposite direction\.
### 4\.5 Metric Dependence on MAGIC

The secondary final\-budget metrics exposed a major tradeoff \(Figure[5](https://arxiv.org/html/2608.13601#S4.F5)\)\. Under clean MAGIC labels at budget 120, uncertainty sampling improved balanced accuracy by 1\.12 percentage points \(95% CI 0\.51 to 1\.71\) but reduced average precision by 2\.21 points \(95% CI−2\.90\-2\.90to−1\.51\-1\.51\)\. It also reduced TPR by 3\.20 points at 1% FPR, 5\.19 points at 2% FPR, 8\.31 points at 5% FPR, and 9\.23 points at 10% FPR\. Each descriptive interval was entirely negative\.

![Refer to caption](https://arxiv.org/html/2608.13601v1/x5.png)Figure 5:Descriptive clean\-label MAGIC differences at budget 120\. Positive values favor uncertainty sampling; negative values favor random sampling\. Bars show paired means and error bars show 95% paired\-bootstrap intervals\. These secondary intervals were not adjusted as a confirmatory test family\.

## 5 Discussion

### 5\.1 What the Results Establish

The clean\-label result is the most consistent conclusion\. Across all three datasets, uncertainty sampling used labels more efficiently according to the pre\-specified primary AULC metric\. The extended experiment further suggests that this benefit is largest when labels are scarce and narrows as the budget grows\. Random sampling is therefore a serious baseline, but it was not stronger in the tested clean low\-budget regime\.

The noise comparison is less universal\. On Breast Cancer, difficulty\-dependent noise reduced the active\-learning advantage more than RCN across a substantial rate range even though both processes had the same expected pool\-wide corruption rate\. Banknote and MAGIC did not reproduce that pattern\. Possible explanations include dataset geometry, the alignment between the random\-forest reference and the evolving logistic learner, and the examples reached by each query path\. The current design does not distinguish among them\.

The matched control further weakens a one\-mechanism explanation\. After mean final acquired exposure was aligned, the predicted additional harm did not survive correction\. MAGIC produced two corrected results in the opposite direction\. These findings do not prove that location never matters, that the conditions are equivalent, or that exposure alone explains H2\. H4 evaluated one final budget and aligned exposure across seeds rather than within every seed or checkpoint\. The justified conclusion is narrower: a universal additional location penalty was not detected by this control\.

Finally, the MAGIC metric reversal prevents a broad statement that uncertainty sampling was simply “better\.” Balanced accuracy evaluates hard predictions at a particular threshold, while average precision and fixed\-FPR TPR evaluate different aspects of ranking and operating behavior\. A strategy can improve one while reducing another\. For MAGIC, the source documentation identifies low background\-acceptance regions as relevant, making the fixed\-FPR results directly important alongside balanced accuracy\[[4](https://arxiv.org/html/2608.13601#bib.bib17)\]\.

### 5\.2 Practical Implications

For small tabular projects, uncertainty sampling is a reasonable clean\-label baseline, especially when the annotation budget is very limited\. It should not be deployed with one aggregate metric and assumed to be robust\. A practical evaluation should:

- •compare against random sampling at every relevant label budget;
- •tune regularization using only the labels available at that budget;
- •repeat paired runs rather than rely on one split or query path;
- •measure the errors actually acquired, not only the pool\-wide rate; and
- •report metrics at operating points that match the intended use\.

When uncertain examples may receive unreliable labels, repeated labeling, expert escalation, or noise\-aware query rules are more defensible next steps than assuming uncertainty sampling will either always fail or always remain efficient\.

## 6 Limitations

The conclusions are limited to three binary tabular datasets, one logistic learner, and two basic query strategies\. Deep models, multiclass problems, structured data, and alternative uncertainty or diversity methods may behave differently\.

The noise is synthetic\. The bounded difficulty\-dependent process is tied to an out\-of\-fold random\-forest boundary, not a measurement of real human disagreement or a worst\-case bounded\-noise adversary\. The reference boundary need not match the logistic learner’s boundary, so “difficulty” is a simulation construct rather than an observed annotator property\. The benchmark labels may themselves contain unknown errors\. The trusted clean seed of 20 labels is useful for stable cross\-validation but may not exist in every application\.

Expected noise was tested only through 0\.30\. Since all point\-estimate AULC crossings were right\-censored, this study cannot estimate where an actual failure crossover would occur\. The matched control tested its confirmatory comparison only at budget 120 and matched mean exposure across seeds rather than exact exposure within each seed or checkpoint\. It therefore cannot fully separate exposure from location across the trajectory\. Cross\-validation also operated on very small labeled samples at early budgets; re\-selecting regularization was more realistic than fixing it, but early selections may still be variable\.

Fixed\-FPR estimates depend on finite test sets and are especially discrete at very low FPR\. The MAGIC test splits contained 1,338 background examples, so a 1% empirical FPR limit permits at most 13 false positives\. Secondary metric intervals and budget trajectories were descriptive rather than multiplicity\-adjusted confirmatory families\. Finally, the analysis plan was frozen after implementation\-only pilots and structural audits rather than registered publicly before all computation\. The repository records this sequence, but an external preregistration would provide stronger protection in a future study\.

## 7 Conclusion

Uncertainty sampling improved clean\-label balanced\-accuracy label efficiency on all three tested datasets and retained a positive mean AULC advantage through 30% expected noise\. Difficulty\-dependent noise reduced this advantage more than RCN on Breast Cancer, but the effect did not generalize to Banknote or MAGIC\. Exposure\-matched controls provided no corrected evidence for the predicted universal additional harm of structured error location and produced two opposite\-direction results on MAGIC\. This null pattern does not prove equivalence or an exposure\-only mechanism\. MAGIC also showed that balanced\-accuracy gains can coexist with reductions in average precision and fixed\-FPR TPR\.

The defensible conclusion is therefore conditional: uncertainty sampling can be label\-efficient, but its apparent robustness depends on dataset structure, annotation budget, noise process, and evaluation metric\. Active\-learning claims should be made across complete learning curves and realistic operating points, not from a single final score\.

## Reproducibility and Data Availability

All datasets are publicly available from the UCI Machine Learning Repository\. The frozen configuration, deterministic experiment implementation, structural audits, statistical analysis plan, automated tests, and reporting code are available at[https://github\.com/dev\-juy/hard\-cases\-bad\-labels](https://github.com/dev-juy/hard-cases-bad-labels)\. The retained main, matched\-control, extended\-clean, and analysis artifacts are hash\-bound to manifests\. The scientific analysis contains 12,800 trials and 271,600 retained budget rows\. The main and matched\-control components used 100 seeds for each dataset; the extended clean\-label component used 100 paired seeds on Breast Cancer Wisconsin only\.

## Deviations and Reporting Clarifications

The frozen configuration retained the internal identifiermargin\_dependent\_massart\_style\. The manuscript uses “bounded difficulty\-dependent noise” because the simulated probability depends on distance to a random\-forest reference boundary rather than the logistic learner’s margin, and because the process is narrower than general Massart noise\. This is a terminology clarification; the noise function, trials, hypotheses, and statistical tests were not changed\.

The H1–H4 numbering is preserved from the frozen analysis plan\. The exposure range in Table[2](https://arxiv.org/html/2608.13601#S4.T2)is a post\-analysis descriptive summary of the already\-produced H4 output; it introduces no additional test\. No new scientific trials or post\-outcome model variants were added during manuscript revision\.

## Acknowledgments

The author thanks Daniel Kane for feedback connecting the simulated processes to random classification and Massart\-style noise, Christian Shelton for feedback on budget\-dependent regularization and operating\-point metrics, and the Summit Research Scholars teaching fellows and peer reviewers for comments on runtime, reference\-model independence, exposure matching, and budget coverage\. These contributors did not determine the final analyses or claims, and any remaining errors are the author’s\.

## References

- \[1\]D\. Angluin and P\. Laird\(1988\)Learning from noisy examples\.Machine Learning2,pp\. 343–370\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1022873112823),[Link](https://link.springer.com/article/10.1023/A:1022873112823)Cited by:[§2\.2](https://arxiv.org/html/2608.13601#S2.SS2.p1.1)\.
- \[2\]P\. Awasthi, M\. Balcan, N\. Haghtalab, and R\. Urner\(2015\)Efficient learning of linear separators under bounded noise\.InProceedings of the 28th Conference on Learning Theory,Proceedings of Machine Learning Research, Vol\.40,pp\. 167–190\.External Links:[Link](https://proceedings.mlr.press/v40/Awasthi15b.html)Cited by:[§2\.2](https://arxiv.org/html/2608.13601#S2.SS2.p1.1)\.
- \[3\]W\. Bae, D\. J\. Sutherland, and G\. L\. Oliveira\(2025\)Uncertainty herding: one active learning method for all label budgets\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UgPoHhYQ2U)Cited by:[§2\.1](https://arxiv.org/html/2608.13601#S2.SS1.p1.1)\.
- \[4\]R\. Bock\(2004\)MAGIC gamma telescope\.UCI Machine Learning Repository\.External Links:[Document](https://dx.doi.org/10.24432/C52C8B),[Link](https://archive.ics.uci.edu/dataset/159/magic+gamma+telescope)Cited by:[§2\.3](https://arxiv.org/html/2608.13601#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2608.13601#S3.T1.1.4.3.1),[§5\.1](https://arxiv.org/html/2608.13601#S5.SS1.p4.1)\.
- \[5\]H\. Cheng, Z\. Zhu, X\. Li, Y\. Gong, X\. Sun, and Y\. Liu\(2021\)Learning with instance\-dependent label noise: a sample sieve approach\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2VXyy9mIyU3)Cited by:[§1](https://arxiv.org/html/2608.13601#S1.p2.1)\.
- \[6\]J\. Davis and M\. Goadrich\(2006\)The relationship between precision–recall and ROC curves\.InProceedings of the 23rd International Conference on Machine Learning,pp\. 233–240\.External Links:[Document](https://dx.doi.org/10.1145/1143844.1143874),[Link](https://doi.org/10.1145/1143844.1143874)Cited by:[§2\.3](https://arxiv.org/html/2608.13601#S2.SS3.p1.1)\.
- \[7\]B\. Efron\(1979\)Bootstrap methods: another look at the jackknife\.The Annals of Statistics7\(1\),pp\. 1–26\.External Links:[Document](https://dx.doi.org/10.1214/aos/1176344552),[Link](https://doi.org/10.1214/aos/1176344552)Cited by:[§3\.6](https://arxiv.org/html/2608.13601#S3.SS6.p3.1)\.
- \[8\]B\. Frénay and M\. Verleysen\(2014\)Classification in the presence of label noise: a survey\.IEEE Transactions on Neural Networks and Learning Systems25\(5\),pp\. 845–869\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2013.2292894),[Link](https://doi.org/10.1109/TNNLS.2013.2292894)Cited by:[§1](https://arxiv.org/html/2608.13601#S1.p2.1)\.
- \[9\]S\. Holm\(1979\)A simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.External Links:[Link](https://www.jstor.org/stable/4615733)Cited by:[§3\.6](https://arxiv.org/html/2608.13601#S3.SS6.p3.1)\.
- \[10\]Y\. Ji, D\. Kaestner, O\. Wirth, and C\. Wressnegger\(2023\)Randomness is the root of all evil: more reliable evaluation of deep active learning\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 3943–3952\.External Links:[Link](https://openaccess.thecvf.com/content/WACV2023/html/Ji_Randomness_Is_the_Root_of_All_Evil_More_Reliable_Evaluation_WACV_2023_paper.html)Cited by:[§2\.1](https://arxiv.org/html/2608.13601#S2.SS1.p1.1)\.
- \[11\]D\. D\. Lewis and W\. A\. Gale\(1994\)A sequential algorithm for training text classifiers\.InProceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3–12\.External Links:[Document](https://dx.doi.org/10.1007/978-1-4471-2099-5%5F1),[Link](https://arxiv.org/abs/cmp-lg/9407020)Cited by:[§1](https://arxiv.org/html/2608.13601#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.13601#S2.SS1.p1.1)\.
- \[12\]V\. Lohweg\(2012\)Banknote authentication\.UCI Machine Learning Repository\.External Links:[Document](https://dx.doi.org/10.24432/C55P57),[Link](https://archive.ics.uci.edu/dataset/267/banknote+authentication)Cited by:[Table 1](https://arxiv.org/html/2608.13601#S3.T1.1.3.2.1)\.
- \[13\]P\. Massart and É\. Nédélec\(2006\)Risk bounds for statistical learning\.The Annals of Statistics34\(5\),pp\. 2326–2366\.External Links:[Document](https://dx.doi.org/10.1214/009053606000000786),[Link](https://arxiv.org/abs/math/0702683)Cited by:[§2\.2](https://arxiv.org/html/2608.13601#S2.SS2.p1.1)\.
- \[14\]P\. Munjal, N\. Hayat, M\. Hayat, J\. Sourati, and S\. Khan\(2022\)Towards robust and reproducible active learning using neural networks\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 223–232\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/html/Munjal_Towards_Robust_and_Reproducible_Active_Learning_Using_Neural_Networks_CVPR_2022_paper.html)Cited by:[§2\.1](https://arxiv.org/html/2608.13601#S2.SS1.p1.1)\.
- \[15\]S\. Nuggehalli, J\. Zhang, L\. K\. Jain, and R\. D\. Nowak\(2025\)Improved algorithm for deep active learning under imbalance via optimal separation\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 46815–46836\.External Links:[Link](https://proceedings.mlr.press/v267/nuggehalli25a.html)Cited by:[§2\.2](https://arxiv.org/html/2608.13601#S2.SS2.p2.1)\.
- \[16\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and É\. Duchesnay\(2011\)Scikit\-learn: machine learning in Python\.Journal of Machine Learning Research12,pp\. 2825–2830\.External Links:[Link](https://jmlr.org/papers/v12/pedregosa11a.html)Cited by:[§3\.2](https://arxiv.org/html/2608.13601#S3.SS2.p1.1)\.
- \[17\]T\. Saito and M\. Rehmsmeier\(2015\)The precision–recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets\.PLOS ONE10\(3\),pp\. e0118432\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0118432),[Link](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432)Cited by:[§2\.3](https://arxiv.org/html/2608.13601#S2.SS3.p1.1)\.
- \[18\]B\. Settles\(2009\)Active learning literature survey\.Technical reportTechnical Report1648,University of Wisconsin–Madison\.External Links:[Link](https://research.cs.wisc.edu/techreports/2009/TR1648.pdf)Cited by:[§1](https://arxiv.org/html/2608.13601#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.13601#S2.SS1.p1.1)\.
- \[19\]N\. Shafir, G\. Hacohen, and D\. Weinshall\(2025\)Active learning with a noisy annotator\.arXiv preprint arXiv:2504\.04506\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.04506),[Link](https://arxiv.org/abs/2504.04506)Cited by:[§2\.2](https://arxiv.org/html/2608.13601#S2.SS2.p2.1)\.
- \[20\]F\. Wilcoxon\(1945\)Individual comparisons by ranking methods\.Biometrics Bulletin1\(6\),pp\. 80–83\.External Links:[Document](https://dx.doi.org/10.2307/3001968)Cited by:[§3\.6](https://arxiv.org/html/2608.13601#S3.SS6.p3.1)\.
- \[21\]W\. Wolberg, O\. Mangasarian, N\. Street, and W\. Street\(1993\)Breast cancer wisconsin \(diagnostic\)\.UCI Machine Learning Repository\.External Links:[Document](https://dx.doi.org/10.24432/C5DW2B),[Link](https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic)Cited by:[Table 1](https://arxiv.org/html/2608.13601#S3.T1.1.2.1.1)\.
- \[22\]Y\. Yan, R\. Rosales, G\. Fung, and J\. G\. Dy\(2011\)Active learning from crowds\.InProceedings of the 28th International Conference on Machine Learning,pp\. 1161–1168\.External Links:[Link](https://icml.cc/2011/papers/596_icmlpaper.pdf)Cited by:[§1](https://arxiv.org/html/2608.13601#S1.p2.1)\.