Unmasking Removal-Budget Confounding: A Matched Operating-Point Evaluation Framework for Adaptive Data Cleaning

arXiv cs.LG Papers
adaptive-data-cleaning evaluation-framework removal-budget machine-learning benchmarking

Summary

Introduces an operating-point-aware evaluation framework for adaptive data cleaning that corrects for removal-budget confounding. Experiments on CIFAR-10 and ImageNet-100 show that apparent performance gains vanish when operating points are matched, suggesting that many improvements reflect smaller removal budgets rather than genuine corruption discrimination.

arXiv:2608.06511v1 Announce Type: new Abstract: Adaptive data-cleaning methods replace manual filtering thresholds with data-driven partitions. However, changing the partition granularity, the number of groups used to segment samples by estimated corruption risk, can implicitly shift the decision boundary and alter the overall number of removed samples. This creates a bias known as removal-budget confounding, where apparent gains in metrics like precision or false-positive rate reflect a smaller removal budget rather than superior corruption discrimination. To address this evaluation bias, we introduce an operating-point-aware evaluation framework that evaluates methods using matched-budget and matched-recall controls alongside threshold-independent metrics (AUROC and AUPRC). We test this framework on a multi-cue adaptive cleaner redesign featuring a reweighted learning-difficulty cue, an auxiliary Euclidean-distance cue, and increased partition granularity intended to isolate clean-but-difficult samples. While naive evaluations (assessing configurations at their own induced operating points) suggest substantial performance improvements for the redesign, these gains disappear once operating points are equalized. False-positive decomposition reveals that clean-but-difficult samples primarily drive error counts at low corruption rates, become threshold-dependent at moderate corruption, and contribute negligibly under severe corruption. Experiments on CIFAR-10 and ImageNet-100 demonstrate that most performance differences observed in naive evaluation shrink or vanish at low-to-moderate corruption when operating points are matched. True ranking advantages only remain in specific low-prevalence settings and in high-recall regions under severe corruption. These findings highlight that adaptive cleaning methods must be benchmarked at matched operating points to ensure performance gains reflect genuine corruption discrimination.
Original Article
View Cached Full Text

Cached at: 08/10/26, 08:01 AM 

# Unmasking Removal-Budget Confounding: A Matched Operating-Point Evaluation Framework for Adaptive Data Cleaning
Source: [https://arxiv.org/abs/2608.06511](https://arxiv.org/abs/2608.06511)
[View PDF](https://arxiv.org/pdf/2608.06511)

> Abstract:Adaptive data\-cleaning methods replace manual filtering thresholds with data\-driven partitions\. However, changing the partition granularity, the number of groups used to segment samples by estimated corruption risk, can implicitly shift the decision boundary and alter the overall number of removed samples\. This creates a bias known as removal\-budget confounding, where apparent gains in metrics like precision or false\-positive rate reflect a smaller removal budget rather than superior corruption discrimination\. To address this evaluation bias, we introduce an operating\-point\-aware evaluation framework that evaluates methods using matched\-budget and matched\-recall controls alongside threshold\-independent metrics \(AUROC and AUPRC\)\. We test this framework on a multi\-cue adaptive cleaner redesign featuring a reweighted learning\-difficulty cue, an auxiliary Euclidean\-distance cue, and increased partition granularity intended to isolate clean\-but\-difficult samples\. While naive evaluations \(assessing configurations at their own induced operating points\) suggest substantial performance improvements for the redesign, these gains disappear once operating points are equalized\. False\-positive decomposition reveals that clean\-but\-difficult samples primarily drive error counts at low corruption rates, become threshold\-dependent at moderate corruption, and contribute negligibly under severe corruption\. Experiments on CIFAR\-10 and ImageNet\-100 demonstrate that most performance differences observed in naive evaluation shrink or vanish at low\-to\-moderate corruption when operating points are matched\. True ranking advantages only remain in specific low\-prevalence settings and in high\-recall regions under severe corruption\. These findings highlight that adaptive cleaning methods must be benchmarked at matched operating points to ensure performance gains reflect genuine corruption discrimination\.

## Submission history

From: JungHua Wang \[[view email](https://arxiv.org/show-email/e9335786/2608.06511)\] **\[v1\]**Thu, 6 Aug 2026 18:50:35 UTC \(367 KB\)

Similar Articles

CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models

arXiv cs.AI

This paper introduces CaRE, a compute-aware evaluation protocol for masked diffusion language models that standardizes step counts, metrics, and stochasticity. It demonstrates that previous comparisons conflate algorithmic improvements with evaluation artifacts, showing temperature explains most MAUVE variance and compute-matched comparisons reverse published rankings.