A Comparative Investigation of Class Imbalance Mitigation Strategies in Clinical Machine Learning: Evaluating Data Resampling, Synthetic Sample Generation, Cost-Aware Algorithms, and Decision Threshold Optimization Under Five Levels of Class Imbalance

Abstract

Background. Clinical outcomes of interest are usually rare, and applying a class-imbalance correction most often SMOTE has become near-automatic in medical machine learning. Whether these corrections improve anything a clinician would value has been questioned but is rarely tested across imbalance regimes. Objective. To quantify the effect of ten imbalance strategies on discrimination, classification and probability calibration across systematically varied imbalance ratios. Methods. Three real clinical datasets (Pima diabetes, Haberman survival, Wisconsin Diagnostic Breast Cancer) were subsampled to induce majority-to-minority ratios of 1:2, 1:5, 1:10, 1:20 and 1:50. Ten strategies — no correction, random over- and under sampling, SMOTE, Borderline-SMOTE, ADASYN, SMOTE+Tomek, class weighting, decision-threshold tuning and balanced random forests were combined with three classifiers under five-fold stratified cross-validation, giving 392 evaluated configurations. Every correction was applied inside the cross-validation loop to training folds only. Results. No resampling method improved ranking performance. Mean change in area under the precision–recall curve relative to no correction was −0.008 for SMOTE (winning 47.6% of 42 paired comparisons; paired t-test p = 0.035 in favour of no correction), 0.011 for random oversampling and −0.025 for random under sampling. Corrections did improve classification: F1 rose from 0.414 to 0.499 under SMOTE and G-mean from 0.451 to 0.641. However, simple threshold tuning achieved statistically indistinguishable F1 (0.494 vs 0.499, Wilcoxon p = 0.229) while leaving probabilities untouched. Calibration damage was severe and systematic: mean predicted risk rose from a true prevalence of 0.121 to 0.214 under SMOTE (+9.3 percentage points) and 0.343 under random under sampling (+22.2 points), with Brier scores 56% and 160% worse than baseline, respectively. Separately, AUC-ROC proved almost blind to imbalance, falling only from 0.906 to 0.819 as prevalence dropped from 33% to 3.6%, while AUC-PR halved from 0.841 to 0.457. Conclusions. Imbalance corrections buy a sensitivity shift that threshold tuning provides for free, and they pay for it with systematically overestimated risk. For clinical models whose output is a probability, the correction is not neutral it is harmful.