What is Stratified Sampling?

A fraud team is preparing a model for review. Its evaluation split contains so few confirmed fraud cases that a missed case could noticeably change recall.

The scikit-learn cross-validation guide describes StratifiedKFold as preserving approximately the same class proportions in each fold as in the full dataset. The guide also notes that rare classes can leave a fold with no cases of that class, and that StratifiedKFold and StratifiedShuffleSplit use stratified sampling to keep relative class frequencies approximately the same in each fold.

In CUBIG’s 2026 UCI Wilt evaluation, the evaluation set contained 15 positive cases among 868 image segments. In that single split from CUBIG’s 2026 run, a change of one correctly identified positive would change recall by about 6.67 percentage points.

A team working with an imbalanced dataset can use stratification when planning a train-test split. Class proportions are useful to inspect alongside the actual number of positive cases, because proportional representation alone cannot create more evidence.

Use the following checks when designing a split:

  • Define the label and the population the evaluation should represent.
  • Inspect class counts in training and evaluation data.
  • Keep the evaluation records fixed when comparing preparation approaches.
  • Report the positive-case count alongside recall and other task metrics.

How is Stratified Sampling different from simple random sampling?

  • Stratified sampling selects records within defined groups, such as positive and negative cases.
  • Simple random sampling selects records without requiring each group to be represented in a particular proportion.
  • Cluster sampling selects groups of records, such as sites or machines, as sampling units. Cluster membership and class strata answer different design questions.

Where does Stratified Sampling break in AI work?

Label-only stratification can place records from the same customer or machine in both training and evaluation sets. Shared context may make the evaluation less representative of performance on new customers or machines.

Randomly distributing records can also conflict with a task that predicts future events. Choose group-aware or time-aware splits when the intended use requires them, then inspect class representation within those constraints.

Frequently asked questions

Does stratification guarantee a reliable recall estimate?

No. A small number of positive cases can still make recall sensitive to individual predictions.

Can stratification be used in cross-validation?

Yes. Stratified folds can preserve class proportions across the folds used for [cross-validation](https://cubig.ai/glossary/cross-validation/).

Should evaluation data always have the same class proportions as training data?

The evaluation distribution should reflect the intended decision and population. Document any deliberate difference when interpreting results.