Glossary

What is Overfitting?

Overfitting shows up as a gap. A credit model reaches near-perfect accuracy on the loans it was trained on and noticeably lower accuracy on loans it has not seen. The model has memorized details of the training records, such as one branch’s habits or a run of unusual months, that do not carry over to new applicants.

Measuring the gap takes records kept out of training. In CUBIG’s UCI Polish Companies Bankruptcy experiment (October 2026), the baseline split set 1,942 evaluation records apart from 7,768 training records, and every before-and-after comparison was scored on those same 1,942 records.

Common causes:

  • a model with more capacity than the data can support, such as a very deep tree trained on a few thousand rows
  • too few examples of the cases that matter, so the model learns them from a handful of records
  • wrong labels the model learns as if they were the pattern, as described under label noise
  • repeated tuning against the same validation data until the choices fit that data

A gap between training scores and scores on unseen data is the usual sign. How large a gap matters depends on the task and on how the unseen data was chosen.

How is overfitting different from data leakage?

  • Overfitting means the model learned the training data too closely. Training scores are high and scores on unseen data are lower.
  • Data leakage means information the model should not have reached it, often through a feature or a preparation step. Training and test scores can both look high, because the test is no longer a fair measure.
  • Overfitting is addressed with simpler models, regularization, more varied data or corrected labels. Leakage is addressed by fixing the features, the split and the order of preparation steps.

Where does overfitting break in AI work?

A team adds records for a rare case to its training set, then tunes the model against the same validation set for several weeks. The validation score keeps rising. On a fresh holdout set, the gain is much smaller, because part of it came from fitting the validation records rather than learning the case.

Catching overfitting takes data the model and its tuning never touched: a holdout set or cross-validation during development, and a fixed evaluation set for the final comparison. In Syntitan, Ready for Model compares results before and after data preparation under the same evaluation conditions, so a gain from preparation is measured on fixed evaluation records rather than on a re-drawn test.

Frequently asked questions

How do you know if a model is overfitting?

Compare its scores on the training data with its scores on data it has never seen. The usual signs are a large gap, or a validation score that keeps rising while the score on a fresh holdout set does not.

How do you prevent overfitting?

Common steps include using a simpler model or regularization, adding varied examples of the important cases, correcting label errors, and keeping a final test set that no tuning decision has touched.

What is the difference between overfitting and underfitting?

Underfitting is a model too simple to capture the real pattern, so it scores poorly on training and unseen data alike. Overfitting captures too much of the training data, including noise, so it scores well on training data and worse elsewhere.