The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

# The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark
Source: [https://arxiv.org/html/2610.00118](https://arxiv.org/html/2610.00118)
Soumyadeep RoyEmail:[roysam422002@gmail\.com](mailto:[email protected])Affiliation:Department of Data Science, St\. Xavier’s College \(Autonomous\), Kolkata, 700016, India

###### Abstract

Customer churn prediction on the IBM Telco Customer Churn benchmark \(n=7,043n=7\{,\}043\) routinely reports test accuracies above 95%, with the most cited published study reporting 99\.01%\. We audit this benchmark for four trustworthiness failures invisible to the accuracy\- andF1F\_\{1\}\-centred reporting that dominates the literature\. First, pre\-splitSmoteinflates churn\-classF1F\_\{1\}by 13\.1 percentage points across ten classifiers and fifteen seeds \(Wilcoxonp<10−4p<10^\{\-4\}per classifier\); the same leaky pipeline ordering paired with class weighting yields no inflation, isolating the effect toSmote’s geometric construction\. We measure the mechanism directly:≈36%\\approx 36\\%of synthetic training points are nearest\-neighbour interpolations of test\-set instances\. Second, theTotalChargesfield is approximately determined bytenure×MonthlyCharges\\texttt\{tenure\}\\times\\texttt\{MonthlyCharges\}\(R2=0\.999R^\{2\}=0\.999\); removing it changes accuracy by less than0\.20\.2pp, yet TreeSHAP ranks it ninth in mean absolute attribution – a pattern that materially corrupts SHAP\-based interpretation\. We propose anR2\>0\.95R^\{2\}\>0\.95pre\-modelling diagnostic\. Third, in a 15\-seed calibration audit, isotonic regression is the strongest default; temperature scaling fails on class\-weighted tree ensembles whose predicted\-probability distribution is bimodal\. Fourth, the cost\-optimal decision threshold \(under a $50 retention offer and 24\-month CLV proxy\) is approximately 5–10×\\timeslower than theF1F\_\{1\}\-optimal threshold, saving∼\\sim$77k per1,0001\{,\}000customers\. We replicate F1 and F2 on Iranian Telecom Churn \(within domain\) and Bank Customer Churn \(across domain\): F1 generalises; F2 generalises only within telecom\. We synthesise these findings into a four\-component reporting checklist – pipeline disclosure, redundancy diagnostic, calibration audit, and cost\-sensitive thresholds – and release a reproducible implementation\.

###### keywords

Customer churn prediction, Trustworthy machine learning, Data leakage, Class imbalance, Probability calibration, Cost\-sensitive learning, Decision curve analysis, SHAP, Reproducibility

## 1Introduction

Customer churn prediction is a textbook decision support task: a probabilistic prediction of subscriber attrition is compared with a threshold to trigger a costed retention action\. The economic stakes are substantial\. Acquiring a new subscriber typically costs five to seven times more than retaining an existing one[Reichheld \(1996\)](https://arxiv.org/html/2610.00118#bib.bib1), and small reductions in monthly churn translate to substantial enterprise value over the customer lifetime[Gupta et al\. \(2004\)](https://arxiv.org/html/2610.00118#bib.bib2)\.

Over the past decade the supervised machine learning community has returned repeatedly to a small set of public churn benchmarks, and the headline numbers in this literature have drifted upward\. On the IBM Telco Customer Churn dataset[IBM \(2018\)](https://arxiv.org/html/2610.00118#bib.bib3), the most\-cited published study reports a test accuracy of 99\.01% with a kernel support vector machine trained onSmote\-Ennresampled data[Nguyen Nhu et al\. \(2022\)](https://arxiv.org/html/2610.00118#bib.bib4), and several further studies report accuracies in the 95–99% band on the same 7,043\-row table\.

These numbers should give the reader pause\. By our own measurement \(Section[3](https://arxiv.org/html/2610.00118#S3)\), the dataset contains 18 distinct feature configurations that appear with conflicting churn labels \(irreducible label noise on the encoded feature space\)\. On this encoding, a classifier that does not overfit to noise and does not leak information from the test set should not exceed1−18/7043≈99\.74%1\-18/7043\\approx 99\.74\\%accuracy\. We are deliberate that this is an empirical irreducible\-error estimate atn=7,043n=7\{,\}043, not a strict information\-theoretic ceiling: a sufficiently overfit model can exceed it on any particular split\. We do not claim that Nguyen et al\.’s[Nguyen Nhu et al\. \(2022\)](https://arxiv.org/html/2610.00118#bib.bib4)99\.01% accuracy is itself attributable to the failure modes we audit – they used a kernel SVM withSmote\-Ennpreprocessing, which we did not reproduce directly – but the proximity of published headline numbers to this informal bound, combined with the absence of any calibration, redundancy, or cost analysis in the high\-accuracy literature, is sufficient motivation for a forensic look at the pipeline choices that produced them\.

This paper documents four trustworthiness failures on this benchmark\. Each is invisible to accuracy\- andF1F\_\{1\}\-centred reporting practices\. Each would mislead a decision\-maker who relied on published numbers to size a retention budget or to interpret model explanations\.

##### \(F1\) Pre\-splitSmoteinflates results, but class weighting on the same leaky pipeline does not\.

The dominant remedy for the dataset’s mild class imbalance \(1:2\.771:2\.77negative\-to\-positive\) in the published literature is theSmotefamily of synthetic oversamplers[Chawla et al\. \(2002\)](https://arxiv.org/html/2610.00118#bib.bib5)applied to the full dataset before the train–test split\. Across 10 classifier families and 15 seeds, this single ordering choice inflates churn\-classF1F\_\{1\}by 13\.1 percentage points relative to a leakage\-free pipeline\. The same leaky ordering paired with cost\-sensitive class weighting yields no comparable inflation\. The existing pipeline\-leakage literature[Vandewiele et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib6);[Kapoor and Narayanan \(2023\)](https://arxiv.org/html/2610.00118#bib.bib7);[Kabane \(2024\)](https://arxiv.org/html/2610.00118#bib.bib8)has reported pre\-splitSmoteas a single confounded unit; our factorial design separates the placement of the preprocessing step from the nature of the imbalance correction\. The asymmetry replicates on the Iranian Telecom Churn benchmark to within 0\.4 pp\.

##### \(F2\)TotalChargesis algebraically redundant and SHAP over\-attributes to it\.

The benchmark’sTotalChargesfield, the cumulative dollar amount billed to a customer, is approximately the product oftenureandMonthlyCharges:R2=0\.9991R^\{2\}=0\.9991, Pearsonr=0\.9996r=0\.9996, mean residual $0\.15\. Removing the feature reduces test accuracy by 0\.18 pp and churn\-classF1F\_\{1\}by 0\.53 pp, both within seed\-to\-seed variation\. Yet TreeSHAP ranksTotalChargesninth in mean absolute attribution, on par with features that contribute real predictive signal\. We do not claim this is data leakage in the conventional sense – the model’s accuracy is honest about the feature’s marginal contribution – but the SHAP\-based interpretation is corrupted in a way that matters for decision support\. We propose a one\-lineR2\>0\.95R^\{2\}\>0\.95pre\-modelling diagnostic to flag candidate redundant features, and we replicate the pattern on the Iranian Telecom Churn benchmark, where three features exceed the same threshold\.

##### \(F3\) Isotonic regression is the strongest calibration default for class\-weighted tree ensembles; temperature scaling fails on bimodal output\.

We measure Expected Calibration Error and Brier score under four post\-hoc calibration methods \(raw, Platt scaling, isotonic regression, temperature scaling\) on the top five classifiers across 15 seeds\. Isotonic regression is the strongest single\-method default on this benchmark, best on four of five models\. Platt scaling is a strong fallback on the worst\-calibrated raw models \(BalancedRF,CatBoost, where it halves ECE alongside isotonic\) but is mildly worse than raw on already\-well\-calibrated models likeGBM\. Temperature scaling, the recommended default in the deep\-learning calibration literature[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9), fails to correct the bimodal predicted\-probability distribution induced by class weighting on tree ensembles\. To our knowledge no published paper on this benchmark reports any calibration audit; ours establishes isotonic regression as a sensible default and explicitly identifies the regime where temperature scaling fails\.

##### \(F4\) The cost\-optimal decision threshold is approximately 5–10×\\timeslower than theF1F\_\{1\}\-optimal threshold\.

Almost all published Telco churn studies report classification metrics at the default threshold of0\.50\.5or, less commonly, at theF1F\_\{1\}\-optimal threshold\. Neither is meaningful for a retention manager, who faces a concrete cost matrix\. We construct a dollar\-cost framework using the dataset’s ownMonthlyChargesfield – false negatives cost 24 months of customer lifetime value, false positives and true positives both cost a fixed $50 retention offer – and sweep the threshold to find the cost\-minimising operating point\. Across five seeds, the cost\-optimal threshold sits consistently in\[0\.04,0\.07\]\[0\.04,0\.07\], saving$77,646±$19,698\\$77\{,\}646\\pm\\$19\{,\}698per 1,000 customers relative toF1F\_\{1\}\-optimal thresholding \(95% CI\[$56,675,$107,634\]\[\\$56\{,\}675,\\$107\{,\}634\]\)\. Decision Curve Analysis[Vickers and Elkin \(2006\)](https://arxiv.org/html/2610.00118#bib.bib10)confirms that the calibrated GBM dominates the trivial “treat all” and “treat none” policies across the operationally relevant threshold range\.

##### Contributions and scope\.

The paper makes four contributions to the trustworthy decision support literature\.

1. 1\.We document the SMOTE\-specific inflation effect \(F1\) on the most widely cited churn benchmark across 1,200 model fits, and isolate it from generic pipeline\-ordering leakage by contrastingSmotewith class weighting on the same leaky pipeline\. The asymmetry replicates on the Iranian Telecom Churn benchmark to within 0\.4 pp\.
2. 2\.We measureTotalChargesalgebraic redundancy on Telco \(F2\), demonstrate that SHAP attribution is misleading on this feature, propose a one\-lineR2R^\{2\}diagnostic, and replicate the pattern on the Iranian benchmark, where three features exceed the sameR2\>0\.95R^\{2\}\>0\.95threshold\.
3. 3\.We provide the first published calibration audit of this benchmark \(F3\), recommending isotonic regression as the default for class\-weighted binary tree ensembles, identifying Platt scaling as a strong fallback on poorly\-calibrated raw models, and documenting the regime\-specific failure of temperature scaling on bimodal class\-weighted output\.
4. 4\.We construct a dollar\-cost decision framework \(F4\) using the dataset’s own pricing, quantify the dollar value of getting the threshold right \(∼\\sim$78k per 1,000 customers\), and demonstrate model dominance over trivial policies via Decision Curve Analysis\.

We close with a four\-component reporting checklist – pipeline ordering disclosure, feature\-redundancy diagnostic, calibration audit, and cost\-sensitive threshold reporting – that we recommend as a minimum trustworthiness standard for tabular churn prediction studies\. All code, intermediate tables, and figures necessary to reproduce every number in the paper are released at[https://github\.com/Sam422002/telco\-churn\-trustworthy\-ml](https://github.com/Sam422002/telco-churn-trustworthy-ml)under an OSI\-approved permissive license\.

##### Paper structure\.

Section[2](https://arxiv.org/html/2610.00118#S2)situates our findings in the churn\-prediction and pipeline\-leakage literatures\. Section[3](https://arxiv.org/html/2610.00118#S3)describes the benchmark, the duplicate\-row and missingness audit, the four pipelines, and the classifier bank\. Section[4](https://arxiv.org/html/2610.00118#S4)presents the four empirical findings, the SHAP analysis, and the cross\-benchmark replication\. Section[5](https://arxiv.org/html/2610.00118#S5)discusses the SMOTE\-versus\-class\-weight mechanism, the choice of calibration method, and the implications of feature redundancy for SHAP\-based interpretation\. Section[6](https://arxiv.org/html/2610.00118#S6)states the reporting checklist and concludes\.

## 2Related Work

Our contribution sits at the intersection of three literatures: applied churn\-prediction studies on the IBM Telco benchmark, the methodological literature on data leakage in tabular machine learning, and the calibration and cost\-sensitive decision support literatures that inform how a probabilistic prediction is actually used\. We treat each in turn\.

### 2\.1High\-Accuracy Telco Churn Prediction

The IBM Telco Customer Churn dataset[IBM \(2018\)](https://arxiv.org/html/2610.00118#bib.bib3)has been the de facto reference benchmark for tabular churn prediction since its release as a Cognos Analytics sample and its subsequent re\-publication on Kaggle\. The published Telco churn literature exhibits several recurring methodological choices: ensembling of tree\-based learners, synthetic oversampling of the minority class with theSmotefamily[Chawla et al\. \(2002\)](https://arxiv.org/html/2610.00118#bib.bib5), and reporting of overall accuracy andF1F\_\{1\}on a single held\-out split\. Nguyen et al\.[Nguyen Nhu et al\. \(2022\)](https://arxiv.org/html/2610.00118#bib.bib4)report 99% accuracy on this benchmark using a kernel support vector machine withSmote\-Ennpreprocessing, in what is to date among the most cited published results on the dataset, and several further studies report accuracies in the 95–99% range on the same 7,043\-row table\.

We highlight three properties of this literature that are directly relevant to our audit\. First, the preprocessing pipeline is rarely reported in sufficient detail to determine whether resampling and scaling were performed before or after the train–test split; in the most\-cited studies, the available information is consistent with pre\-split application\. Second,TotalChargesis consistently reported as a top\-three or top\-five feature in SHAP, permutation, or gain\-based importance rankings, but we are aware of no published study on this benchmark that examines its algebraic relationship withtenureandMonthlyCharges\. Third, calibration is not reported in any of the high\-accuracy studies we surveyed, and the cost of a false retention recommendation versus a missed churn event is treated implicitly through the choice of metric \(typically the default0\.50\.5threshold\) rather than explicitly through a cost matrix\.

A separate strand of churn\-prediction research considers profit\-aware or cost\-sensitive evaluation[Verbeke et al\. \(2012\)](https://arxiv.org/html/2610.00118#bib.bib11);[Verbraken et al\. \(2013\)](https://arxiv.org/html/2610.00118#bib.bib12), typically using the Expected Maximum Profit \(EMP\) framework\. These studies make important contributions but operate largely independently of the high\-accuracy literature: papers reporting 99% accuracy on Telco do not typically report EMP or expected cost, and EMP\-aware studies do not typically engage with the pipeline\-leakage and calibration questions we audit here\. Our dollar\-cost analysis \(Section[4\.4](https://arxiv.org/html/2610.00118#S4.SS4)\) is methodologically simpler than EMP – a fixed retention offer cost and a 24\-month CLV proxy – and is intended as a minimum\-viable cost\-sensitive analysis that any churn study could include without adopting the full EMP machinery\.

### 2\.2Data Leakage in Tabular Machine Learning

Data leakage – the inadvertent inclusion of information at training time that would not be available at deployment – is the dominant explanation in the methodological literature for irreproducible or implausibly high accuracy on tabular benchmarks\. Kapoor and Narayanan[Kapoor and Narayanan \(2023\)](https://arxiv.org/html/2610.00118#bib.bib7)catalogue 294 papers across 17 fields with documented leakage and identify a small number of recurring failure modes: target leakage, train–test contamination, feature leakage from post\-outcome variables, and preprocessing leakage from steps applied before the train–test split\. They emphasise that the fix – moving preprocessing inside cross\-validation – is well known but rarely implemented in published code\.

The specific case of pre\-split synthetic oversampling has received focused attention\. Vandewiele et al\.[Vandewiele et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib6)demonstrate on a perinatal benchmark that applyingSmotebefore splitting produces nearest\-neighbour pairs that span the train–test boundary, artificially inflating test accuracy\. Kabane[Kabane \(2024\)](https://arxiv.org/html/2610.00118#bib.bib8)replicates the effect on the European credit\-card fraud benchmark and provides a controlled comparison of pre\-split versus pipeline\-internalSmote\.

The existing audits treat “pre\-splitSmote” as the unit of analysis, conflating two confounded choices: the placement of the preprocessing step \(before vs\. after the split\) and the nature of the imbalance correction \(synthetic oversampling vs\. class weighting\)\. We disentangle these in Section[4\.1](https://arxiv.org/html/2610.00118#S4.SS1)by running the same leaky pipeline ordering with bothSmoteand class\-weighted training and showing that the inflation is specific toSmote\.

### 2\.3Calibration and Cost\-Sensitive Decision Support

A churn prediction is, in operational terms, a probability that is compared to a threshold to trigger a retention action\.

The probability\-calibration literature distinguishes between discrimination \(the ability to rank positives above negatives, measured by ROC\-AUC\) and calibration \(the agreement between predicted probabilities and empirical frequencies, measured by Expected Calibration Error or Brier score\)\. Modern tree ensembles and neural networks are well known to be miscalibrated by default[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9), and post\-hoc calibration methods – Platt scaling[Platt \(1999\)](https://arxiv.org/html/2610.00118#bib.bib13), isotonic regression[Zadrozny and Elkan \(2002\)](https://arxiv.org/html/2610.00118#bib.bib14), and temperature scaling[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9)– are by now standard remedies\. The choice among them is not, however, settled\. Platt scaling assumes a sigmoid miscalibration shape and works best on binary problems with monotonic miscalibration; isotonic regression is more flexible but data\-hungry; temperature scaling has a single parameter and is the recommended default in the deep\-learning literature for multi\-class softmax outputs\. We are aware of no calibration audit of the Telco benchmark and no systematic comparison of calibration methods on class\-weighted tree ensembles for binary churn, which is the regime in which this benchmark is used in practice\.

The cost\-sensitive learning literature[Elkan \(2001\)](https://arxiv.org/html/2610.00118#bib.bib15)provides the formal apparatus for selecting a decision threshold when false positives and false negatives have different costs\. Decision Curve Analysis[Vickers and Elkin \(2006\)](https://arxiv.org/html/2610.00118#bib.bib10)is a graphical tool from medical decision\-making that plots net benefit against threshold probability and identifies the threshold range over which a model dominates the trivial “treat all” and “treat none” policies\. DCA has been applied extensively in clinical risk prediction but is essentially absent from the churn\-prediction literature, where operational evaluation, when reported at all, typically uses lift charts or top\-decile capture rates rather than threshold\-explicit cost analysis\. Our Section[4\.4](https://arxiv.org/html/2610.00118#S4.SS4)brings DCA and a simple explicit cost matrix into the churn setting\.

## 3Dataset and Methods

This section describes the benchmark, the diagnostic audit we perform before any modelling, the four preprocessing pipelines that isolate specific pipeline\-ordering effects, the classifier bank, the evaluation protocol, and the reproducibility setup\.

### 3\.1Datasets

We use the IBM Telco Customer Churn dataset[IBM \(2018\)](https://arxiv.org/html/2610.00118#bib.bib3), distributed as a Cognos Analytics sample and most commonly accessed via its Kaggle mirror\. The dataset containsn=7,043n=7\{,\}043customer records of a fictional but plausibly parameterised telecommunications provider, with 21 columns: a customer identifier \(customerID\), the binary churn label, three numeric features \(tenurein months,MonthlyChargesin US dollars,TotalChargesin US dollars\), and 16 categorical service and demographic features\. The class distribution is moderately imbalanced at5,1745\{,\}174negatives to1,8691\{,\}869positives, an imbalance ratio of1:2\.771:2\.77\(Table[1](https://arxiv.org/html/2610.00118#S3.T1), row 4\)\.

##### Replication benchmark\.

For the cross\-benchmark replication in Section[4\.6](https://arxiv.org/html/2610.00118#S4.SS6)we additionally use the Iranian Telecom Churn dataset[Jafari\-Marandi et al\. \(2020\)](https://arxiv.org/html/2610.00118#bib.bib16),n=3,150n=3\{,\}150, 13 numeric features, churn rate 15\.71% \(imbalance ratio1:5\.361:5\.36\)\. This benchmark differs from IBM Telco in operator \(a different telecommunications provider in a different country\), sample size \(less than half of Telco\), and feature space \(all\-numeric, no categorical columns\), but addresses the same underlying decision problem \(month\-to\-month subscriber retention\)\. It is, to our knowledge, the cleanest publicly\-available second churn benchmark for testing whether our findings generalise within the telecommunications domain\.

##### The whitespaceTotalChargesquirk\.

TotalChargesis loaded bypandasas anobject\(string\) dtype rather thanfloat64because 11 of the 7,043 rows contain a single whitespace character in this field rather than a numeric value\. All 11 such rows havetenure= 0 andChurn= “No”\. These are brand\-new customers who have never been billed\. A naivepd\.to\_numeric\(\)call without explicit handling will silently coerce these 11 rows toNaN, after which the standard practice of dropping rows with missing values removes 11 customers who are, structurally, the most informative cases for studying churn onset\. We handle this case explicitly in our loader \(whitespace→0\.0\\rightarrow 0\.0forTotalChargeswheretenure= 0\) and flag the quirk because none of the high\-accuracy studies we surveyed mention it; reproducibility of those results depends on a choice the studies do not document\.

### 3\.2Diagnostic Audit

Before any modelling we run four diagnostics on the raw data\. Their results are summarised in Table[1](https://arxiv.org/html/2610.00118#S3.T1)and form the basis of our subsequent analyses\.

##### Duplicate\-row audit\.

We define three duplicate categories\.*Exact duplicates*are rows that match on all features and on the label;*feature\-only duplicates*are rows that match on all features but may differ in the label;*ambiguous tuples*are feature configurations that appear more than once in the data with conflicting labels\. On Telco we find 22 exact duplicates \(0\.31%\), 40 feature\-only duplicates \(0\.57%\), and 18 ambiguous tuples \(Table[1](https://arxiv.org/html/2610.00118#S3.T1)\)\. The maximum number of copies of any single feature tuple is 4\. Ambiguous tuples place an empirical irreducible\-error estimate on deterministic test accuracy atn=7,043n=7\{,\}043: if 18 distinct feature configurations are inherently unpredictable from the available features, a classifier that does not overfit to noise should not exceed1−18/7043≈99\.74%1\-18/7043\\approx 99\.74\\%accuracy on this encoding\. We treat this as a sanity\-check estimate rather than a strict bound, since overfitting to noise on a particular split can in principle exceed it on an in\-sample test\.

##### Missingness and identifier audit\.

Beyond the 11 whitespaceTotalChargesentries described above, the dataset has no missing values\. ThecustomerIDcolumn has 7,043 unique values across 7,043 rows; no row appears twice under the same identifier\. We dropcustomerIDbefore any modelling because it carries no predictive signal\.

##### Tenure\-redundancy diagnostic \(F2\)\.

We test the hypothesis thatTotalChargesis approximately the product oftenureandMonthlyCharges\. Restricting to rows withtenure\>0\>0\(to exclude the 11 zero\-tenure new customers and avoid trivial agreement at zero\), we compute the residual

ri=TotalChargesi−tenurei⋅MonthlyChargesir\_\{i\}\\;=\\;\\texttt\{TotalCharges\}\_\{i\}\\;\-\\;\\texttt\{tenure\}\_\{i\}\\cdot\\texttt\{MonthlyCharges\}\_\{i\}\(1\)and report its summary statistics in Table[1](https://arxiv.org/html/2610.00118#S3.T1)\. The fit is essentially exact:R2=0\.9991R^\{2\}=0\.9991, Pearsonr=0\.9996r=0\.9996, mean residual $0\.15, standard deviation $67\.20, maximum absolute residual $373\.25 \(likely reflecting mid\-cycle plan changes during a billing month\)\.TotalChargesis, to two decimal places, an algebraic function of two other columns of the same row\. This is the empirical basis for the F2 finding in Section[4\.2](https://arxiv.org/html/2610.00118#S4.SS2)\.

##### Class\-balance diagnostic\.

The negative\-to\-positive ratio is1:2\.771:2\.77\. We characterise this as mild imbalance, in line with the heuristic threshold of1:101:10below which class weighting and threshold tuning are typically sufficient and synthetic oversampling is not strictly required[Branco et al\. \(2016\)](https://arxiv.org/html/2610.00118#bib.bib17)\. The implication for our pipeline design is that we can run bothSmoteand class\-weighted training on the same data and compare them directly \(Section[3\.4](https://arxiv.org/html/2610.00118#S3.SS4)\)\.

Table 1:Diagnostic audit of the IBM Telco Customer Churn benchmark \(n=7,043n=7\{,\}043\)\. Quirks flagged in the “Note” column are, to our knowledge, not mentioned in any high\-accuracy published study on this benchmark\.DiagnosticValueNote*Class balance*Total rows7,043Negatives \(No churn\)5,174 \(73\.46%\)Positives \(Churn\)1,869 \(26\.54%\)Imbalance ratio \(neg:pos\)1:2\.771:2\.77mild*Duplicate\-row audit*Unique feature tuples7,003 \(99\.43%\)Exact duplicates \(feat\. \+ label\)22 \(0\.31%\)Feature\-only duplicates40 \(0\.57%\)Ambiguous tuples \(label noise\)18irreducible noiseMax copies of any feature tuple4Empirical irred\.\-error est\.≈99\.74%\\approx 99\.74\\%1−18/70431\-18/7043*Identifiers and missingness*customerIDunique7,043 / 7,043TotalChargesraw dtypeobject \(string\)whitespace quirkWhitespaceTotalCharges11Tenure of whitespace rows\{0\}new customersChurn of whitespace rows\{No: 11\}never billed*Tenure\-redundancy diagnostic \(F2\)*R2R^\{2\}\(tenure×\\timesMC→\\toTC\)0\.9991Pearsonrr0\.9996Mean residual \($\)0\.15Std\. residual \($\)67\.20Max\|\\lvertresidual\|\\rvert\($\)373\.25*Marginal correlations*corr\(tenure,TC\)0\.8262corr\(MC,TC\)0\.6512corr\(tenure,MC\)0\.2479

### 3\.3Feature Encoding

We dropcustomerID\(identifier\) andChurn\(label\) from the feature matrix and one\-hot encode the 16 categorical features withdrop\_first=Trueto avoid the dummy\-variable trap\. We do not engineer interaction features, do not bin numeric features, and do not apply log transforms; the goal of this study is to audit the standard pipeline, not to optimise predictive performance\. The encoded feature matrix has 30 columns\. For theP\-Droppipeline \(Section[3\.4](https://arxiv.org/html/2610.00118#S3.SS4)\) we additionally dropTotalCharges, yielding 29 columns\.

### 3\.4Pipelines

We compare four preprocessing pipelines that differ only in the ordering of three operations: train–test split \(Split\), feature scaling \(Scale\), and synthetic minority oversampling \(Smote\)\. All four use the same encoded feature matrix, the same80/2080/20stratified test split, and the same set of classifiers\.

- •P\-Clean\(correct\)\.Split→\\toScale\(fit on train only\)→\\toSmote\(fit on train only\)\. This is the leakage\-free reference\.
- •P\-Leak\(incorrect, as in the high\-accuracy literature\)\.Smote→\\toScale→\\toSplit\. Synthetic samples can land in both train and test; scaler statistics are computed on data the test set has seen\.
- •P\-Scale\(intermediate, isolates scaler leakage\)\.Scale→\\toSplit→\\toSmote\(on train\)\. Scaler is fit on full data, but synthetic oversampling is correct\.
- •P\-Drop\(probe for the F2 redundancy finding\)\. Same ordering asP\-Clean, but withTotalChargesremoved from the feature matrix before any operation\.

The four\-pipeline design supports two contrasts\. TheP\-LeakversusP\-Cleancontrast isolates the effect induced by ordering ofSmoteandScalerelative to the split \(the classical leakage from Vandewiele et al\.[Vandewiele et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib6)\)\. TheP\-CleanversusP\-Dropcontrast isolates the predictive contribution ofTotalChargesabove and beyondtenureandMonthlyCharges\. TheP\-Scalepipeline serves as a control to verify that scaler leakage alone \(without resampling leakage\) is small, which we confirm in Section[4\.1](https://arxiv.org/html/2610.00118#S4.SS1)\.

##### Imbalance handling\.

For each pipeline we run two strategies:Smoteoversampling to a1:11:1class ratio on the \(appropriate, by pipeline\) training data, and class weighting \(Cw\) withclass\_weight=‘‘balanced’’or the equivalentscale\_pos\_weightfor boosters\. The factorial design over pipelines and strategies \(4×\\times2 = 8 combinations\) is what allows us to disentangle SMOTE\-specific inflation from generic pipeline\-ordering effects in Section[4\.1](https://arxiv.org/html/2610.00118#S4.SS1)\.

### 3\.5Classifier Bank

We evaluate ten classifiers spanning four model families:

- •*Linear:*Logistic Regression \(LogReg\)\.
- •*Single tree:*Decision Tree \(DT\)\.
- •*Tree ensembles:*Random Forest \(RF\), Extra Trees \(ExtraTrees\), Balanced Random Forest \(BalancedRF\), Gradient Boosting \(GBM\), LightGBM, XGBoost, CatBoost\.
- •*Neural:*Multilayer Perceptron \(MLP, two hidden layers of 64 and 32 units\)\.

We use scikit\-learn defaults forLogReg,DT,RF,ExtraTrees,GBM, andMLP;imbalanced\-learnforBalancedRF; and the official implementations of LightGBM, XGBoost, and CatBoost\. Tree ensembles are configured with300300trees \(100100forGBMfor runtime parity\), random seed control on every estimator, and GPU acceleration where available\. All other hyperparameters are library defaults\. We do not perform hyperparameter tuning; this is deliberate, because tuned models would obscure the pipeline\-ordering signal that is the subject of the audit\.

We exclude support vector classifiers, Gaussian naïve Bayes, andkk\-nearest neighbours from the bank\. SVC was projected at over two hours per fit on the 8,278\-rowSmote\-augmented training set, which would have made the 1,200\-fit factorial benchmark intractable; the omission does not affect any of the comparative findings, since F1 inflation is observed across all classifier families that are included\.

### 3\.6Evaluation Protocol

For each \(pipeline, strategy, classifier\) combination we report five metrics on the held\-out test set: overall accuracy, churn\-classF1F\_\{1\}, churn\-class recall, churn\-class precision, and ROC\-AUC\. We emphasise churn\-class metrics because the operationally relevant event is the minority class\.

##### Multi\-seed factorial design\.

The full benchmark in Section[4\.1](https://arxiv.org/html/2610.00118#S4.SS1)runs 10 classifiers×\\times15 random seeds×\\times4 pipelines×\\times2 imbalance strategies = 1,200 model fits\. Seeds are drawn from\{0,1,…,14\}\\\{0,1,\\dots,14\\\}and used both for the train–test split and for the classifier’s own internal randomness\. Reporting the mean and standard deviation across the 15 seeds provides a stable estimate of each pipeline’s effect that is robust to seed\-specific outliers\.

##### Calibration evaluation\.

For the calibration analysis \(Section[4\.3](https://arxiv.org/html/2610.00118#S4.SS3)\) we restrict to the top five classifiers by accuracy onP\-Clean\+Cwand evaluate four post\-hoc calibration methods: raw \(no calibration\), Platt scaling[Platt \(1999\)](https://arxiv.org/html/2610.00118#bib.bib13), isotonic regression[Zadrozny and Elkan \(2002\)](https://arxiv.org/html/2610.00118#bib.bib14), and temperature scaling[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9)\. Platt and isotonic are fit on a held\-out calibration split of the training set \(20% of train\); temperature scaling is fit on the same calibration split using LBFGS on the negative log\-likelihood\. We report Expected Calibration Error \(15\-bin equal\-width\) and Brier score[Brier \(1950\)](https://arxiv.org/html/2610.00118#bib.bib18)on the test set\.

##### Cost\-sensitive evaluation\.

For Section[4\.4](https://arxiv.org/html/2610.00118#S4.SS4)we define a customer\-specific cost matrix using the dataset’sMonthlyChargescolumn\. A false negative \(a churning customer not flagged for retention\) incurs a cost equal to the customer’s 24\-month customer lifetime value,24⋅MonthlyCharges24\\cdot\\texttt\{MonthlyCharges\}; a true positive or false positive \(any customer flagged for retention\) incurs a fixed retention\-offer cost of $50; a true negative incurs zero cost\. We sweep the decision threshold over\[0\.01,0\.99\]\[0\.01,0\.99\]in 99 steps and report the cost\-minimising threshold per seed across five seeds\{7,13,21,42,99\}\\\{7,13,21,42,99\\\}\. Decision Curve Analysis[Vickers and Elkin \(2006\)](https://arxiv.org/html/2610.00118#bib.bib10)is computed on the canonical seed = 42,P\-Clean\+Cw, GBM model\.

##### SHAP analysis\.

SHAP values are computed with TreeSHAP[Lundberg et al\. \(2020\)](https://arxiv.org/html/2610.00118#bib.bib19)on the canonicalP\-Clean\+CwGBM model with seed = 42\. We report mean absolute SHAP value rankings on the test set and check business\-intuition alignment against eight a priori expected risk factors \(Section[4\.5](https://arxiv.org/html/2610.00118#S4.SS5)\)\.

##### Cross\-benchmark replication protocol\.

For Section[4\.6](https://arxiv.org/html/2610.00118#S4.SS6)we re\-run theP\-Clean/P\-Leak/P\-Scalepipelines under both imbalance strategies on the Iranian Telecom benchmark, with the same 10\-classifier bank, 15 random seeds, and metric set, yielding3×2×15×10=9003\\times 2\\times 15\\times 10=900model fits\. TheP\-Droppipeline is omitted for Iranian because its feature space is all\-numeric and contains no a priori derived feature analogous toTotalCharges; instead, we run the redundancy diagnostic on every numeric column to identify candidate redundant features post hoc\.

### 3\.7Reproducibility

All experiments are run on a hosted notebook environment with a single NVIDIA T4 GPU\. Random seeds are fixed at the level of the train–test split, the resampler, and the classifier; with these fixed, every number in this paper is deterministic up to the non\-determinism of GPU floating\-point reduction in CatBoost and XGBoost \(which we measure to be below10−310^\{\-3\}in accuracy across re\-runs\)\. The full code, intermediate CSV tables, and figures are released at[https://github\.com/Sam422002/telco\-churn\-trustworthy\-ml](https://github.com/Sam422002/telco-churn-trustworthy-ml)under an OSI\-approved permissive license\.

## 4Results

We present four empirical findings\. Section[4\.1](https://arxiv.org/html/2610.00118#S4.SS1)quantifies pre\-splitSmoteleakage \(F1\) across 1,200 model fits and isolates it from class\-weighted training on the same leaky pipeline ordering\. Section[4\.2](https://arxiv.org/html/2610.00118#S4.SS2)validatesTotalChargesas a redundant feature \(F2\) using theP\-Clean–P\-Dropcontrast\. Section[4\.3](https://arxiv.org/html/2610.00118#S4.SS3)reports the calibration audit \(F3\)\. Section[4\.4](https://arxiv.org/html/2610.00118#S4.SS4)presents the dollar\-cost threshold analysis \(F4\)\. Section[4\.5](https://arxiv.org/html/2610.00118#S4.SS5)reports SHAP\-based business alignment and a SHAP\-specific over\-attribution finding for redundant features\. Section[4\.6](https://arxiv.org/html/2610.00118#S4.SS6)reports the cross\-benchmark replication on the Iranian Telecom Churn dataset\.

### 4\.1Pipeline\-Ordering Audit \(F1\)

We report the 1,200\-fit factorial benchmark first as pipeline\-aggregated means \(Table[2](https://arxiv.org/html/2610.00118#S4.T2)\), then broken down by classifier \(Table[3](https://arxiv.org/html/2610.00118#S4.T3)\), and finally as the SMOTE\-versus\-class\-weight contrast that drives Section[5\.1](https://arxiv.org/html/2610.00118#S5.SS1)\(Table[5](https://arxiv.org/html/2610.00118#S4.T5)\)\.

##### Pipeline\-aggregated effect\.

Aggregated over 10 classifiers, 15 random seeds, and both imbalance strategies, the pipeline contrasts are summarised in Table[2](https://arxiv.org/html/2610.00118#S4.T2)\. The leakage\-free referenceP\-Cleanyields a mean accuracy of0\.76690\.7669and churn\-classF1F\_\{1\}of0\.57440\.5744\. The classical leaky pipelineP\-Leakinflates these to0\.80350\.8035and0\.70550\.7055– absolute differences of\+3\.66 percentage points in accuracy and \+13\.11 percentage points inF1F\_\{1\}, with churn recall inflated by 11\.46 pp and ROC\-AUC by 5\.38 pp\. The intermediateP\-Scalepipeline \(scaler leakage only, no resampling leakage\) produces results essentially identical toP\-Clean, confirming that theP\-Leakeffect is driven by theSmote\-on\-full\-data ordering rather than by the scaler\. Of the 1,200 fits, zero failed\.

Table 2:Pipeline\-aggregated mean test metrics on the IBM Telco Customer Churn benchmark \(10 classifiers×\\times15 seeds×\\times2 imbalance strategies = 300 fits per pipeline\)\.Δ\\Deltarows give the absolute difference ofP\-LeakandP\-Droprelative toP\-Clean, in percentage points\. Accuracy andF1F\_\{1\}are computed at the default threshold of0\.50\.5\.
##### Per\-classifier inflation\.

Table[3](https://arxiv.org/html/2610.00118#S4.T3)disaggregates the accuracy inflation by classifier \(averaged over both imbalance strategies and 15 seeds\)\. The inflation is positive for every classifier in the bank but varies by an order of magnitude:LogRegshows the smallest accuracy inflation \(\+0\.96\+0\.96pp\) andMLPthe largest \(\+4\.82\+4\.82pp\)\. The corresponding SMOTE\-specific churn\-classF1F\_\{1\}inflation per classifier – which is the operationally relevant quantity – is reported in Table[4](https://arxiv.org/html/2610.00118#S4.T4)\.

Table 3:Per\-classifier accuracy inflation on Telco Churn\. Each cell is the mean over 15 seeds and 2 imbalance strategies \(30 fits per cell\)\.*Acc\. inflation*isP\-Leakaccuracy minusP\-Cleanaccuracy in percentage points; the*F2 drop*column isP\-DropminusP\-Cleanaccuracy\. Sorted by accuracy inflation\.
##### Significance of the per\-classifier inflation\.

We ran a paired Wilcoxon signed\-rank test on the churn\-classF1F\_\{1\}deltasP\-Leak−\-P\-Cleanper seed, restricted to theSmotestrategy, for each of the ten classifiers \(15 seeds each\)\. Results are reported in Table[4](https://arxiv.org/html/2610.00118#S4.T4)\. Every classifier shows a positive, highly significant SMOTE\-specificF1F\_\{1\}inflation:p<10−4p<10^\{\-4\}for all ten classifiers\. The SMOTE\-specificF1F\_\{1\}inflation per classifier ranges from\+15\.13±1\.36\+15\.13\\pm 1\.36pp \(LogReg\) to\+30\.23±2\.04\+30\.23\\pm 2\.04pp \(MLP\), with most tree ensembles in the\[\+23,\+30\]\[\+23,\+30\]pp band\.

Table 4:Per\-classifier SMOTE\-specific churn\-classF1F\_\{1\}inflation on Telco\. Each row is the paired\-by\-seed mean±\\pmstandard deviation across 15 seeds, restricted to theSmoteimbalance strategy\. The Wilcoxon signed\-rank test \(one\-sided, alternative “P\-Leak\>\>P\-Clean”\) reaches its lower\-boundp=3\.05⋅10−5p=3\.05\\cdot 10^\{\-5\}on every classifier\. Sorted byΔ​F1\\Delta F\_\{1\}\.
##### The SMOTE\-versus\-class\-weight isolation\.

The principal finding of the pipeline\-ordering audit is that the inflation in Table[2](https://arxiv.org/html/2610.00118#S4.T2)is not produced by pipeline ordering alone\. We re\-run the same factorial design with the imbalance strategy as a between\-cell factor and report the four \(pipeline, strategy\) means in Table[5](https://arxiv.org/html/2610.00118#S4.T5)\.

Table 5:The SMOTE\-versus\-class\-weight contrast on the leaky pipeline\. Each cell is the mean over 10 classifiers and 15 seeds \(150 fits per cell\) at default threshold0\.50\.5\.The structure of Table[5](https://arxiv.org/html/2610.00118#S4.T5)is unambiguous\. The leaky pipelineP\-Leakproduces a 0\.8404 accuracy / 0\.8393F1F\_\{1\}result*only*when the imbalance correction isSmote; running the identical leaky ordering with class weighting yields 0\.7666 / 0\.5716, statistically indistinguishable fromP\-Clean\.*The\+13\+13ppF1F\_\{1\}inflation is the interaction of two choices, not either one alone*: pre\-split applicationandsynthetic minority oversampling\. Either remedy – movingSmoteinside the train side of the split, or replacing it with class weighting – removes the inflation entirely\. This sharpens the existing pipeline\-leakage literature[Vandewiele et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib6);[Kapoor and Narayanan \(2023\)](https://arxiv.org/html/2610.00118#bib.bib7);[Kabane \(2024\)](https://arxiv.org/html/2610.00118#bib.bib8)\.

### 4\.2Feature\-Redundancy Finding \(F2\)

The audit in Section[3\.2](https://arxiv.org/html/2610.00118#S3.SS2)established thatTotalCharges≈tenure×MonthlyCharges\\texttt\{TotalCharges\}\\approx\\texttt\{tenure\}\\times\\texttt\{MonthlyCharges\}withR2=0\.9991R^\{2\}=0\.9991\. TheP\-Clean–P\-Dropcontrast in Table[2](https://arxiv.org/html/2610.00118#S4.T2)shows that removingTotalChargesreduces accuracy by 0\.18 pp,F1F\_\{1\}by 0\.53 pp, recall by 0\.73 pp, and ROC\-AUC by 0\.23 pp\. All four differences are within the seed\-to\-seed standard deviation\. Within the resolution of the experiment,*the model loses no performance from removing the feature*\.

We therefore propose “feature\-redundancy diagnostic” rather than “redundancy leakage” as the appropriate label\. We adopt the operational testΔdrop​accuracy<σseed\\Delta\_\{\\text\{drop\}\}\\,\\text\{accuracy\}<\\sigma\_\{\\text\{seed\}\}as a practical definition of redundancy on a candidate feature, and recommend reporting it for any feature whoseR2R^\{2\}regressed on a small set of others exceeds 0\.95\.

### 4\.3Calibration Audit \(F3\)

We report Expected Calibration Error \(ECE, 15\-bin equal\-width\) and Brier score under four post\-hoc calibration methods on the top five classifiers byP\-Clean\+Cwaccuracy, averaged over 15 random seeds\. ECE means and standard deviations are reported in Table[6](https://arxiv.org/html/2610.00118#S4.T6)\.

Table 6:Multi\-seed Expected Calibration Error \(ECE, 15\-bin equal\-width\) on the top five classifiers under four calibration methods, onP\-Clean\+Cw\. Each cell is mean±\\pmstandard deviation across 15 random seeds \(seeds0,…,140,\\ldots,14\)\. Best per row in bold\. ECE is lower\-is\-better\.Three findings emerge\. First,isotonic regression is the strongest single\-method default: best on four of five models, with a 70% ECE reduction on the worst\-calibrated raw models\. Second,Platt scaling is a strong fallbackon poorly calibrated models but mildly worse than raw on well\-calibrated models likeGBM\. Third,temperature scaling[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9)fails on class\-weighted tree ensembleswith the largest raw ECE \(BalancedRF:0\.1092→0\.11270\.1092\\to 0\.1127;CatBoost:0\.1129→0\.11600\.1129\\to 0\.1160\): class weighting pushes a tree’s leaf\-level vote distributions to a bimodal shape that a single\-parameter logit rescaling cannot correct\.

### 4\.4Dollar\-Cost Threshold Analysis \(F4\)

We sweep the decision threshold over\[0\.01,0\.99\]\[0\.01,0\.99\]for the canonicalP\-Clean\+CwGBMmodel with Platt calibration \(Table[7](https://arxiv.org/html/2610.00118#S4.T7)\)\.

Table 7:Threshold\-strategy comparison on the canonicalP\-Clean\+Cw\+Gbm\+Platt model, seed = 42\. Total cost is computed on the full test set \(ntest=1,409n\_\{\\text\{test\}\}=1\{,\}409customers\)\.Across five seeds, the cost\-optimal threshold is consistently in\[0\.04,0\.07\]\[0\.04,0\.07\]; mean savings are$77,646±$19,698\\$77\{,\}646\\pm\\$19\{,\}698per 1,000 customers\(95% CI\[$56,675,$107,634\]\[\\$56\{,\}675,\\$107\{,\}634\]\)\. The 95% CI does not cross zero:F1F\_\{1\}\-optimal thresholding is the worse choice on every seed in our sample\. Decision Curve Analysis[Vickers and Elkin \(2006\)](https://arxiv.org/html/2610.00118#bib.bib10)confirms that theGBM\+Platt model dominates both trivial policies across the entire operationally relevant threshold range\[0\.05,0\.6\]\[0\.05,0\.6\]\.

### 4\.5SHAP Trust\-Anchor and Over\-Attribution

We compute TreeSHAP[Lundberg et al\. \(2020\)](https://arxiv.org/html/2610.00118#bib.bib19)values on the canonicalP\-Clean\+CwGBMmodel, seed = 42, and report the top ten features by mean absolute SHAP in Table[8](https://arxiv.org/html/2610.00118#S4.T8)\.

Table 8:Top ten features by mean absolute SHAP value on the canonicalP\-Clean\+Cw\+Gbmmodel, seed = 42\. “Direction” indicates whether high feature values push the prediction toward churn or away from it\.RankFeatureMean\|\|SHAP\|\|Direction1tenure0\.5733high→\\tono churn2Contract\_Two year0\.4475true→\\tono churn3InternetService\_Fiber optic0\.3952true→\\tochurn4PaymentMethod\_Electronic check0\.2136true→\\tochurn5Contract\_One year0\.1992true→\\tono churn6PaperlessBilling\_Yes0\.1327true→\\tochurn7MonthlyCharges0\.1198high→\\tochurn8OnlineSecurity\_Yes0\.1147true→\\tono churn9TotalCharges*0\.1064**\(redundant; see text\)*10TechSupport\_Yes0\.1030true→\\tono churnAll eight a priori expected churn risk factors are represented in the top ten \(8\-of\-8 alignment\), providing a positive trust anchor\. However,TotalChargesranks ninth despite contributing essentially nothing predictive abovetenureandMonthlyCharges\. The mechanism is well known in the explainability literature[Slack et al\. \(2020\)](https://arxiv.org/html/2610.00118#bib.bib20);[Aas et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib21): SHAP allocates contributions among algebraically related features in a way that depends on the conditional sampling distribution, and on a feature pair with near\-perfect algebraic dependency, attribution can be split without diminishing either\. A high SHAP rank does not by itself license the inference that a feature is predictive\.

### 4\.6Cross\-Benchmark Replication on Iranian Telecom Churn

We replicate the factorial design on the Iranian Telecom Churn dataset[Jafari\-Marandi et al\. \(2020\)](https://arxiv.org/html/2610.00118#bib.bib16)\. Pre\-splitSmoteinflates churn\-classF1F\_\{1\}by\+12\.69\+12\.69pp; pre\-split class weighting inflates by\+0\.05\+0\.05pp\. The Telco result was\+13\.11\+13\.11pp underSmoteand effectively zero under class weighting; the Iranian result reproduces both numbers to within 0\.4 pp\. The asymmetry is not a Telco artefact\. Running the F2 diagnostic on the 13 numeric Iranian features yields three columns withR2\>0\.95R^\{2\}\>0\.95:Customer Value\(R2=0\.9821R^\{2\}=0\.9821\),Frequency of SMS\(R2=0\.9770R^\{2\}=0\.9770\), andSeconds of Use\(R2=0\.9561R^\{2\}=0\.9561\)\.

### 4\.7Cross\-Domain Replication on Bank Customer Churn

We additionally replicate on the Bank Customer Churn Modelling benchmark[Kaggle contributors \(2018\)](https://arxiv.org/html/2610.00118#bib.bib22)\(n=10,000n=10\{,\}000, churn rate 20\.4%, imbalance ratio1:3\.911:3\.91\)\. Pre\-splitSmoteinflates churn\-classF1F\_\{1\}by\+30\.5\+30\.5pp; pre\-split class weighting inflates by\+0\.02\+0\.02pp\. The F2 diagnostic yields a maximumR2R^\{2\}of 0\.094 on Bank features, below theR2\>0\.95R^\{2\}\>0\.95threshold—an honest negative result: the Bank dataset contains no operator\-supplied derived aggregate analogous toTotalCharges\.

Summary\.F1 generalises across both benchmarks with magnitudes that scale with the imbalance ratio\. F2 generalises within the telecom domain but not to banking\.

## 5Discussion

### 5\.1WhySmoteLeaks but Class Weighting Does Not

Smotegenerates synthetic minority points by linear interpolation:x~=xi\+λ⁡\(xj−xi\)\\tilde\{x\}=x\_\{i\}\+\\lambda\(x\_\{j\}\-x\_\{i\}\)forλ∈\[0,1\]\\lambda\\in\[0,1\]\. When applied before the train–test split, every synthetic point is a convex combination of two real points, either of which can land in the test set after the split\. We measure this directly: on the Telco benchmark,36\.6±2\.5%36\.6\\pm 2\.5\\%of synthetic\-train points have at least one parent in the test fold\. Class weighting[Chawla et al\. \(2002\)](https://arxiv.org/html/2610.00118#bib.bib5)reweights the loss function without creating synthetic samples, so no geometric contamination can occur regardless of pipeline ordering\. The asymmetry betweenSmoteand class weighting under the same leaky ordering is therefore a structural property of the resampler, not an artefact of the particular estimator or seed\. Audits should distinguish between leakage modes that produce geometric contamination \(Smote, ADASYN, BorderlineSMOTE, SMOTE\-ENN, SMOTE\-Tomek\) and those that do not \(class weighting, threshold tuning, focal loss\)\.

### 5\.2Calibration on Class\-Weighted Tree Ensembles

Isotonic regression’s piecewise\-constant fit absorbs the heterogeneous miscalibration shapes induced by class weighting\. Temperature scaling fits a single scalarTTapplied to the logits:p^=σ⁡\(s/T\)\\hat\{p\}=\\sigma\(s/T\)\. BecauseTTacts uniformly on the entire score distribution, it cannot redistribute probability mass between regions of the\[0,1\]\[0,1\]interval\. Class weighting on a tree ensemble produces a bimodal predicted\-probability distribution: predictions cluster near00\(leaves dominated by majority votes\) and near11\(leaves dominated by reweighted minority votes\), with relatively few intermediate predictions\. A single temperature scalar cannot correct this\. The recommendation of temperature scaling for multi\-class softmax outputs in the deep\-learning calibration literature[Guo et al\. \(2017\)](https://arxiv.org/html/2610.00118#bib.bib9)should not be transferred without testing to binary class\-weighted tree ensembles\.

### 5\.3Feature Redundancy and the Limits of SHAP

Feature redundancy does not change deployment performance—theP\-CleanandP\-Dropmodels have indistinguishable test accuracy\. What it corrupts is the interpretation: a feature that contributes nothing predictive nonetheless appears in the top ten of every importance ranking\. TreeSHAP’s path\-dependent algorithm allocates contributions among features that co\-occur on tree paths, and on a feature pair\(X,f⁡\(X\)\)\(X,f\(X\)\)whereffis approximately linear, the algorithm splits attribution between the redundant feature and its source\(s\) without any systematic preference for the source[Aas et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib21)\. The fix is a pre\-modelling redundancy audit: fitTotalChargesas a linear function of the other numeric features and check theR2R^\{2\}\. Any practitioner could have run this in fifteen seconds before fitting any model\.

### 5\.4Limitations

\(1\) F3 and F4 are not replicated on Iranian or Bank; replication on software\-as\-a\-service, streaming, or non\-churn tabular decision\-support benchmarks remains future work\. \(2\) No hyperparameter tuning is performed; this is deliberate to preserve the pipeline\-ordering signal\. \(3\) Our cost framework uses a fixed $50 retention\-offer cost and 24\-month CLV proxy; the full Expected Maximum Profit framework[Verbraken et al\. \(2013\)](https://arxiv.org/html/2610.00118#bib.bib12)is a more principled successor\. \(4\) The unconstrained cost\-optimal threshold flags75\.1±5\.9%75\.1\\pm 5\.9\\%of customers, which is operationally implausible; capacity\-constrained top\-K%K\\%alternatives recover most savings within feasible budgets\. \(5\) The SHAP analysis is restricted to TreeSHAP on the canonical GBM model; we expect KernelSHAP and permutation importance to over\-attribute toTotalChargesfor the same algebraic reason[Aas et al\. \(2021\)](https://arxiv.org/html/2610.00118#bib.bib21)\.

## 6Conclusion: A Reporting Checklist for Trustworthy Churn DSS

We audited the IBM Telco Customer Churn benchmark and identified four trustworthiness failures invisible to standard reporting: SMOTE\-specific pre\-split leakage \(F1;\+13\.1\+13\.1ppF1F\_\{1\}aggregated,≈36%\\approx 36\\%of synthetic\-train points are interpolated copies of test\-set neighbourhoods\); algebraic feature redundancy \(F2;R2=0\.999R^\{2\}=0\.999\) that corrupts SHAP interpretation without harming accuracy; isotonic regression as the strongest calibration default and a regime\-specific failure of temperature scaling on bimodal class\-weighted output \(F3\); and a dollar\-cost optimal threshold approximately five times lower than theF1F\_\{1\}\-optimal threshold, saving $77,646 per 1,000 customers \(F4\)\. F1 generalises across two cross\-benchmark replications \(\+12\.7\+12\.7pp on Iranian,\+30\.5\+30\.5pp on Bank\); F2 generalises within the telecom\-churn family\.

### 6\.1The Four\-Component Reporting Checklist

1. 1\.Pipeline\-ordering disclosure\.Report the exact sequence in which the train–test split, feature scaling, and any imbalance correction were applied\. IfSmoteor any synthetic oversampler was used, state explicitly whether it was fit on the full dataset or only on the training fold\. Submitted code should make this verifiable\.
2. 2\.Feature\-redundancy diagnostic\.For each numeric feature, report the maximumR2R^\{2\}from regressing it on a low\-dimensional combination of the other features\. Flag any feature withR2\>0\.95R^\{2\}\>0\.95as a redundancy candidate\. For flagged features, reportΔdrop\\Delta\_\{\\text\{drop\}\}accuracy and confirmΔdrop\>σseed\\Delta\_\{\\text\{drop\}\}\>\\sigma\_\{\\text\{seed\}\}if the feature is retained\.
3. 3\.Calibration audit\.Report Expected Calibration Error and Brier score for the primary model under at least two calibration methods, averaged over multiple seeds\. On binary class\-weighted tree ensembles, isotonic regression is the recommended default; Platt scaling is a strong fallback\. Avoid temperature scaling on models with bimodal raw predicted\-probability distributions\. Include a reliability diagram in the supplementary material\.
4. 4\.Cost\-sensitive threshold\.Report performance at the cost\-optimal threshold under an explicit cost matrix, alongside performance at the default0\.50\.5andF1F\_\{1\}\-optimal thresholds\. State the cost ratioCFN/CFPC\_\{\\text\{FN\}\}/C\_\{\\text\{FP\}\}used and provide a Decision Curve Analysis or equivalent against “treat all” and “treat none” baselines\. The Expected Maximum Profit framework[Verbraken et al\. \(2013\)](https://arxiv.org/html/2610.00118#bib.bib12)is a principled alternative where joint cost–response distributions are available\.

### 6\.2Closing

Decision support is, by definition, the discipline that connects predictions to costed actions\. A churn model that is 99% accurate on a leaky pipeline, miscalibrated, attributes importance to algebraically derivable features, and is operated at a threshold that ignores the cost matrix is not a decision support tool – it is an artefact of the benchmark culture that produced it\. The contribution of this paper is not a better churn classifier but a clearer audit, a checklist by which future studies can avoid the failure modes we have measured, and a cross\-benchmark validation that the audit itself generalises beyond a single dataset within the telecom\-churn family\.

All code, intermediate tables, and figures necessary to reproduce every number in the paper are released at[https://github\.com/Sam422002/telco\-churn\-trustworthy\-ml](https://github.com/Sam422002/telco-churn-trustworthy-ml), including the cross\-benchmark replication script that produced the Iranian results in Section[4\.6](https://arxiv.org/html/2610.00118#S4.SS6)\. We invite further replication on additional churn benchmarks – KKBox, IBM Watson, banking\-sector churn – and on non\-churn tabular decision\-support benchmarks\.

#### Acknowledgements

The author thanks the reviewers for their careful reading\. Computational experiments were run on a hosted notebook environment with free GPU access \(Kaggle Notebooks\), which made the full 1,200\-fit factorial benchmark and the 900\-fit cross\-benchmark replication tractable\.

## Declarations

- •Funding:No external funding was received for this research\.
- •Conflict of interest:The author declares no competing financial or personal interests that could have appeared to influence the work reported in this paper\.
- •Ethics approval and consent to participate:Not applicable\.
- •Consent for publication:Not applicable\.
- •
- •Materials availability:Not applicable\.
- •
- •Author contribution:Sole\-author contribution\.

## References

- Reichheld \(1996\)Reichheld, F\.F\.: The Loyalty Effect: The Hidden Force Behind Growth, Profits, and Lasting Value\. Harvard Business School Press, Boston, MA \(1996\)
- Gupta et al\. \(2004\)Gupta, S\., Lehmann, D\.R\., Stuart, J\.A\.: Valuing customers\. Journal of Marketing Research41\(1\), 7–18 \(2004\)
- IBM \(2018\)IBM: Telco Customer Churn dataset\. IBM Cognos Analytics sample data; Kaggle mirror at[https://www\.kaggle\.com/datasets/blastchar/telco\-customer\-churn](https://www.kaggle.com/datasets/blastchar/telco-customer-churn)\(2018\)
- Nguyen Nhu et al\. \(2022\)Nguyen Nhu, Y\., Tran Van, L\., Dao Vu Truong, S\.: Churn prediction in telecommunication industry using kernel Support Vector Machines\. PLOS ONE17\(5\), 0267935 \(2022\)[https://doi\.org/10\.1371/journal\.pone\.0267935](https://doi.org/10.1371/journal.pone.0267935)
- Chawla et al\. \(2002\)Chawla, N\.V\., Bowyer, K\.W\., Hall, L\.O\., Kegelmeyer, W\.P\.: SMOTE: synthetic minority over\-sampling technique\. Journal of Artificial Intelligence Research16, 321–357 \(2002\)
- Vandewiele et al\. \(2021\)Vandewiele, G\., Dehaene, I\., Kovacs, G\., Sterckx, L\., Janssens, O\., Ongenae, F\., De Backere, F\., De Turck, F\., Roelens, K\., Decruyenaere, J\., Van Hoecke, S\., Demeester, T\.: Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over\-sampling\. Artificial Intelligence in Medicine111, 101987 \(2021\)
- Kapoor and Narayanan \(2023\)Kapoor, S\., Narayanan, A\.: Leakage and the reproducibility crisis in machine\-learning\-based science\. Patterns4\(9\), 100804 \(2023\)
- Kabane \(2024\)Kabane, S\.: Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection \(2024\)
- Guo et al\. \(2017\)Guo, C\., Pleiss, G\., Sun, Y\., Weinberger, K\.Q\.: On calibration of modern neural networks\. In: Proceedings of the 34th International Conference on Machine Learning \(ICML\), pp\. 1321–1330 \(2017\)
- Vickers and Elkin \(2006\)Vickers, A\.J\., Elkin, E\.B\.: Decision curve analysis: a novel method for evaluating prediction models\. Medical Decision Making26\(6\), 565–574 \(2006\)
- Verbeke et al\. \(2012\)Verbeke, W\., Dejaeger, K\., Martens, D\., Hur, J\., Baesens, B\.: New insights into churn prediction in the telecommunication sector: a profit driven data mining approach\. European Journal of Operational Research218\(1\), 211–229 \(2012\)
- Verbraken et al\. \(2013\)Verbraken, T\., Verbeke, W\., Baesens, B\.: A novel profit maximizing metric for measuring classification performance of customer churn prediction models\. IEEE Transactions on Knowledge and Data Engineering25\(5\), 961–973 \(2013\)
- Platt \(1999\)Platt, J\.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\. In: Advances in Large\-Margin Classifiers, pp\. 61–74\. MIT Press, Cambridge, MA \(1999\)
- Zadrozny and Elkan \(2002\)Zadrozny, B\., Elkan, C\.: Transforming classifier scores into accurate multiclass probability estimates\. In: Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp\. 694–699 \(2002\)
- Elkan \(2001\)Elkan, C\.: The foundations of cost\-sensitive learning\. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence \(IJCAI\), pp\. 973–978 \(2001\)
- Jafari\-Marandi et al\. \(2020\)Jafari\-Marandi, R\., Denton, J\., Idris, A\., Smith, B\.K\., Keramati, A\.: Iranian Churn dataset\. UCI Machine Learning Repository \(ID 563\); Kaggle mirror at[https://www\.kaggle\.com/datasets/royjafari/customer\-churn](https://www.kaggle.com/datasets/royjafari/customer-churn)\(2020\)
- Branco et al\. \(2016\)Branco, P\., Torgo, L\., Ribeiro, R\.P\.: A survey of predictive modeling on imbalanced domains\. ACM Computing Surveys49\(2\), 1–50 \(2016\)
- Brier \(1950\)Brier, G\.W\.: Verification of forecasts expressed in terms of probability\. Monthly Weather Review78\(1\), 1–3 \(1950\)
- Lundberg et al\. \(2020\)Lundberg, S\.M\., Erion, G\., Chen, H\., DeGrave, A\., Prutkin, J\.M\., Nair, B\., Katz, R\., Himmelfarb, J\., Bansal, N\., Lee, S\.\-I\.: From local explanations to global understanding with explainable AI for trees\. Nature Machine Intelligence2\(1\), 56–67 \(2020\)
- Slack et al\. \(2020\)Slack, D\., Hilgard, S\., Jia, E\., Singh, S\., Lakkaraju, H\.: Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods\. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp\. 180–186 \(2020\)
- Aas et al\. \(2021\)Aas, K\., Jullum, M\., Løland, A\.: Explaining individual predictions when features are dependent: more accurate approximations to Shapley values\. Artificial Intelligence298, 103502 \(2021\)
- Kaggle contributors \(2018\)Kaggle contributors: Bank Customer Churn Modelling dataset\. Available at[https://www\.kaggle\.com/datasets/shrutimechlearn/churn\-modelling](https://www.kaggle.com/datasets/shrutimechlearn/churn-modelling)\(2018\)