A train-test split is often the first decision in a model evaluation, and the split decides what the test score means. If the split is random when the data has a time order, the test contains records from before some of the training records. If one customer appears on both sides, the model is tested partly on people it has already seen.
CUBIG’s UCI Polish Companies Bankruptcy experiment (October 2026) used a baseline split of 7,768 training and 1,942 evaluation records and kept the same evaluation records for every before-and-after comparison.
Questions to settle before splitting:
- does the data have a time order, and should the test come after the training period
- do several records belong to one customer, patient, site or device, and should they stay together
- is the important outcome rare, and does the test portion contain enough of it
- which preparation steps learn from the data, and are they fitted on the training portion only
How is a train-test split different from a train-validation-test split?
- A train-test split has two parts: one to train on and one to measure the result.
- A train-validation-test split adds a validation portion for tuning settings and choosing between versions, so the test portion stays untouched until the final measurement. The evaluation set entry explains how the terms are used.
- Cross-validation replaces the single validation portion with several rotating ones.
Where does a train-test split break in AI work?
A team splits claims records at random, prepares features, and reports a strong test score. The test portion includes claims filed after many of the training claims, and several fields were scaled using statistics from the full dataset. Both choices let information from the test reach training, so the score likely overstates how the model will do on next month’s claims.
A split stays useful only if it reflects how the model will be used and if every learned step is fitted on the training portion alone, the same discipline described under data leakage. The UCI write-up states this limit for its own split: the results come from a single split with 103 positive evaluation records, are not a time-based validation, and are not evidence of performance on current companies.