Data Enablement

Class Imbalance in Machine Learning: What to Do When the Cases You Need Are Rare

A cluster of overlapping translucent purple circles with one small amber circle set apart on the right

A fraud team reviews a model with impressive accuracy and discovers that it misses most of the suspicious transactions investigators needed to find. The score looked reassuring because ordinary transactions dominated the evaluation.

Class imbalance in machine learning means that some outcomes appear far less often than others in the data used for a classification task. The practical question is whether the available examples and evaluation design let a model find the cases that matter at a cost the business can accept.

Key takeaways

  • Preserve a representative held-out evaluation set before changing the training data.
  • Inspect which rare cases are missing, underrepresented, mislabeled, or unavailable at the time of prediction.
  • Compare data collection, sampling, class weights, and threshold choices against the same task and error costs.
  • Read detected cases and false alarms together, and repeat promising comparisons before acting on them.

What is class imbalance, and why can accuracy mislead?

Class imbalance occurs when a classification dataset contains substantially fewer examples of some outcomes than others, so accuracy can conceal poor performance on a rare outcome.

In a fraud task, the positive class might be confirmed fraud and the negative class legitimate activity. A model that favors the common outcome can classify many records correctly while failing to identify the transactions the investigation team needs. The CUBIG glossary entry on imbalanced datasets describes the distribution problem; the task determines whether the resulting errors matter.

The confusion matrix makes those errors visible. It separates cases correctly found from missed positives, false alarms, and correctly rejected negatives. Counts give the operations team a concrete view of the work behind a score: a missed fraud case may go uninvestigated, while a false alarm can consume investigator time.

CUBIG’s UCI Wilt experiment, published in October 2026, evaluated 868 image segments containing 15 diseased-tree segments. In that exploratory run on one internal split, logistic regression found 1 of the 15 diseased-tree segments before preparation and all 15 afterward, while false positives rose from 2 to 9. The result shows why both detection and the resulting workload belong in the discussion.

An imbalanced class distribution does not automatically make a model unsuitable. Some rare outcomes may be identifiable with the available features, while a more balanced dataset can still have weak labels or miss important situations. The question is whether the model meets the decision criteria on cases that resemble its intended use.

CUBIG Shorts

Pipeline Up 60%. So We Hired.

A short take on the same point: a metric that looks good can hide the cases a team actually needed to find.

▶  Watch on YouTubeSubscribe

How can you tell whether class imbalance is hurting your model?

Class imbalance is hurting the task when the model misses too many relevant cases, creates an unacceptable volume of false alarms, or performs poorly on important subsets.

Begin by defining the positive outcome and when the decision is made. A churn label, for example, needs a clear definition of leaving and a prediction period; a failure label needs an agreed account of what constitutes failure. Check whether features were available when the prediction would have been made, since information recorded after the outcome can make evaluation misleading.

Compare recall, the share of actual positives found, with precision, the share of flagged cases that are positive. F1 combines precision and recall, but its balance may not reflect the cost of a missed case or an unnecessary intervention. Review the underlying counts alongside the scores and agree on the operational capacity available to act on alerts.

A ranking metric adds another view when the threshold may change. Saito and Rehmsmeier’s 2015 study explains why precision-recall plots can be more informative than ROC plots for imbalanced binary classification. CUBIG’s precision-recall curve guide explains how precision and recall change across thresholds.

Look beyond the overall positive count. Break down known cases by relevant failure mode, customer segment, site, time period, or other task-specific conditions, and ask which combinations are absent or thinly represented. A model can appear useful overall while missing a subgroup where the cost of failure is high; small subgroup counts also make conclusions uncertain.

How should you evaluate an imbalanced dataset before changing it?

Set aside evaluation records that reflect the intended use, establish a baseline, and keep those records out of preparation and model selection.

Start with the class mix expected when the model will be used, as far as the available evidence allows. If the historical sample was collected through a selective investigation process, its proportions may not represent future traffic. Document that uncertainty and seek an evaluation sample closer to the operating population when possible.

Stratified sampling can preserve approximate class proportions across a random split or cross-validation folds. The scikit-learn cross-validation guide, accessed in November 2026, explains that stratified splitters keep class frequencies approximately the same in each fold, which helps prevent folds with no rare-class examples. Stratification does not solve dependence between records: repeated customers, machines, sites, or time periods may require grouped or time-aware splitting.

Keep a validation set or cross-validation procedure for choosing preparation, model settings, and the decision threshold. Reserve a final evaluation set for the resulting comparison, and record the positive and negative counts as well as metrics. When positives are scarce, a small change in the number found can move a rate substantially, so repeat the evaluation where the data permits.

The imbalanced-learn documentation, accessed in November 2026, warns that resampling before splitting can leak information and alter the evaluation distribution. Fit resampling and other learned transformations within each training fold, then apply the fitted transformations to validation records without resampling them. Keep generated records out of the held-out set.

Which methods handle class imbalance, and when should you use them?

Choose a method according to the missing information, the cost of each error, and what can be evaluated without contaminating the held-out records.

Methods for handling class imbalance in machine learning, and when each fits
Method What it changes When it fits Watch out for
Collect more real cases Adds observed examples and potentially missing conditions A rare subgroup or operating condition is poorly represented and more labeled observations are feasible Labels, collection criteria, and timing must match the intended task
Random oversampling or undersampling Repeats minority training records or removes majority training records A straightforward sampling baseline is needed Repetition can overfit; removal can discard useful majority cases
SMOTE-style synthetic oversampling Creates training examples between minority examples and nearby minority neighbors Minority examples have meaningful neighborhoods in the chosen feature space Generated combinations may be implausible or blur class boundaries
Class weights Changes how training penalizes errors for each class The model supports weighting and the team wants a training baseline without changing row counts Weights still need validation against the task’s error costs
Threshold tuning Changes the score at which a prediction becomes an alert Scores are useful for ranking and the team can choose an operating point A lower threshold can increase workload; select it on validation data

Chawla and colleagues’ SMOTE paper, published in 2002, describes creating synthetic minority examples using a minority example and its nearest minority neighbors. The method can provide a useful comparison, yet plausible feature values depend on the domain: interpolation may produce combinations that a fraud analyst, clinician, or engineer would reject. The CUBIG SMOTE glossary covers the method in more detail.

For supported models, class weighting offers another baseline. The scikit-learn LogisticRegression documentation, accessed in November 2026, describes a balanced setting with weights inversely proportional to class frequencies. Those weights reflect frequency; the business cost of an error may differ, so compare them with weights chosen for the task.

Threshold tuning changes the alert decision without adding training cases. Select a threshold using validation predictions and a criterion such as investigation capacity or the relative cost of errors, then evaluate the chosen operating point on held-out records. If scores are used as probabilities in a business calculation, check their calibration as well as their ranking.

Collecting real cases can fill gaps that duplication cannot, especially when missing cases differ in cause or context. Synthetic or sampled records can change the training distribution, but they cannot establish that an unobserved failure mode has been captured. Compare each candidate against the original-data baseline and inspect whether any gain holds for the cases the team actually needs.

What do CUBIG’s experiments show about the trade-off?

CUBIG’s experiments show that preparation can change both the number of rare cases found and the number of false alarms on a particular evaluation split.

In CUBIG’s October 2026 Iranian Churn run, the evaluation set contained 99 churners among 630 customers. In that October 2026 CUBIG run, with logistic regression settings and the 0.5 threshold held fixed, the prepared-data model found 78 churners instead of 42, while false positives rose from 8 to 41.

In the same CUBIG October 2026 run, logistic regression’s average precision rose from 0.7563 to 0.8144, while precision at the fixed threshold fell from 0.8400 to 0.6555. The ranking result and the longer contact list answer different questions: a retention team still needs to assess the cost and value of acting on the additional flags.

CUBIG’s November 2026 predictive maintenance experiment used the synthetic UCI AI4I dataset and evaluated LightGBM on the same 2,000 held-out records, including 68 labeled failures. In CUBIG’s November 2026 run, LightGBM found 56 failures after preparation instead of 49 before, while false alarms rose from 3 to 4. The dataset lacks timestamps and a forecast window, so the result describes classifying a failure from the same record’s operating conditions; advance warning was outside the test.

The Wilt, churn, and maintenance results each describe a single split, and the relevant cases and preparation differed. In the October 2026 Wilt run, logistic regression F1 rose from 0.1111 to 0.7692, but no McNemar p-value for the reported models was below 0.05; the result is exploratory, with no significance claim. CUBIG’s checks of the exported files also did not independently establish that every upstream preparation step was fitted using training data only.

Several preparation steps were applied together in the runs, so the comparisons do not isolate each step’s contribution; CUBIG’s churn evaluation article notes that threshold tuning and class weighting on the original data were not compared, and the Wilt write-up names the same missing baseline. Repeated splits, independent records, and those simpler baselines are useful next checks before drawing broader conclusions.

What should you check before you rebalance a training set?

Check the task, the missing cases, and the evaluation design before changing the distribution of training records.

  • Outcome and timing: Is the positive label defined consistently, and were the input features available when a decision would be made?
  • Coverage: Which rare cases, segments, sites, or failure modes are missing or underrepresented, and why?
  • Evaluation: Do held-out records reflect the intended use, with a split that respects time, groups, and class proportions where appropriate?
  • Leakage: Are sampling, generation, and learned transformations fitted only on training data within each validation fold?
  • Decision costs: How many missed cases and false alarms can the workflow tolerate, and how will the threshold be selected?
  • Comparison: Can the team compare the original and prepared data on the same evaluation records, inspect the counts, and repeat the result?

A checklist cannot replace domain review of the labels and features. An equipment specialist may recognize an implausible operating state, while an investigator may know that certain fraud labels are observed only after a selective review. Those details help distinguish a class-frequency problem from a gap in how cases are collected or recorded.

Where does CUBIG fit in handling class imbalance?

CUBIG helps teams prepare data for a specific AI task and examine what changes under the same evaluation conditions.

Syntitan, CUBIG’s AI-ready data platform, works toward AI-ready data: data prepared for a specific task, with its results checked, whose relevant parts can be rechecked when data or conditions change. The goal includes preserving the information and exceptions the task needs; readiness is not a one-time certification or a single score.

DTS, Data Enablement for AI, fills missing cases, adjusts skewed distributions, and prepares data that is difficult to use as it stands for a specific task. Inside Syntitan, DTS works as the data preparation and supplementation engine. The choice of outcome, acceptable error trade-off, and follow-up evaluation on the same records remains grounded in the team’s use case.

Syntitan keeps the original Dataset unchanged, works on a copy, and retains preparation and validation results as Versions. As of November 2026, Ready for Model compares built-in models before and after its own preparation on the same evaluation records for yes-or-no prediction on datasets of 100 to 10,000 rows, according to CUBIG’s predictive maintenance article. A team can use the comparison, and the exported Proof Kit, to examine changes in missed cases and false alarms alongside scores.

A favorable comparison on a held-out set is a starting point, especially when rare examples are few or the evaluation does not reflect future conditions. Check the preparation process, compare relevant baselines, and test again on independent or later data before using the result to support an operational decision.

Try it on your data in Syntitan: compare model results before and after preparation on the same evaluation records.

Explore Syntitan, CUBIG's AI-Ready Data Platform

FAQ

What is class imbalance in machine learning?

Class imbalance occurs when some outcomes appear much less often than others in a classification dataset, making task-specific evaluation and coverage of rare cases important.

Why can accuracy be misleading for imbalanced data?

A model can classify the common class correctly while missing many of the rare cases that prompted the task.

Should I balance data before splitting it?

Keep the evaluation records apart before resampling, and apply resampling within each training fold to avoid leakage and preserve a realistic evaluation mix.

Which method should I use for an imbalanced dataset?

Start with the missing cases and the cost of errors, then compare approaches such as collecting real examples, resampling, class weighting, and threshold tuning under the same evaluation conditions.