A credit team’s new default model beats the old one on a single test split. Before the new model replaces the old one, someone outside the build team must decide whether the gap is real and relevant to the lending decision.
In banking, SR 26-2 guidance on model risk management, issued by the Federal Reserve, FDIC and OCC in April 2026 to replace SR 11-7, says model validation evaluates whether models perform as expected and assesses their reliability and limitations. The guidance also calls for effective challenge by objective experts.
Small evaluation sets limit what validation can show. In CUBIG’s UCI Wilt experiment, published October 2026, one internal split held only 15 positive cases among 868 segments, so one case moved recall by about 6.67 percentage points, and no McNemar p-value fell below 0.05.
Core elements of validation include:
- Conceptual soundness: Whether the design and data fit the intended use
- Outcomes analysis: Performance on held-out data, compared with a baseline under the same conditions
- Ongoing monitoring: Checks that continue after the model is in use
- Independent review: Challenge from people with enough independence to assess the model objectively
How is model validation different from model evaluation?
- Model evaluation measures performance with chosen metrics on an evaluation set.
- Model validation asks whether those measurements support the intended business use, including the data, assumptions and limits.
- An evaluation can pass while validation fails, for example when the evaluation data does not reflect the population the model will score.
Where does model validation break in AI work?
Validation breaks when the data used to tune or select a model is reused to validate it. The reported score then reflects choices made with that data, and cross-validation or a separate holdout set is needed for an honest estimate.
Validation also breaks when it ignores the decision the model supports. A model can rank cases well and still flag more cases than a team can review, so the review should include the confusion matrix at the threshold the business will use.