What is Model Validation?

A credit team’s new default model beats the old one on a single test split. Before the new model replaces the old one, someone outside the build team must decide whether the gap is real and relevant to the lending decision.

In banking, SR 26-2 guidance on model risk management, issued by the Federal Reserve, FDIC and OCC in April 2026 to replace SR 11-7, says model validation evaluates whether models perform as expected and assesses their reliability and limitations. The guidance also calls for effective challenge by objective experts.

Small evaluation sets limit what validation can show. In CUBIG’s UCI Wilt experiment, published October 2026, one internal split held only 15 positive cases among 868 segments, so one case moved recall by about 6.67 percentage points, and no McNemar p-value fell below 0.05.

Core elements of validation include:

  • Conceptual soundness: Whether the design and data fit the intended use
  • Outcomes analysis: Performance on held-out data, compared with a baseline under the same conditions
  • Ongoing monitoring: Checks that continue after the model is in use
  • Independent review: Challenge from people with enough independence to assess the model objectively

How is model validation different from model evaluation?

  • Model evaluation measures performance with chosen metrics on an evaluation set.
  • Model validation asks whether those measurements support the intended business use, including the data, assumptions and limits.
  • An evaluation can pass while validation fails, for example when the evaluation data does not reflect the population the model will score.

Where does model validation break in AI work?

Validation breaks when the data used to tune or select a model is reused to validate it. The reported score then reflects choices made with that data, and cross-validation or a separate holdout set is needed for an honest estimate.

Validation also breaks when it ignores the decision the model supports. A model can rank cases well and still flag more cases than a team can review, so the review should include the confusion matrix at the threshold the business will use.

Frequently asked questions

What are the main steps of model validation?

The April 2026 SR 26-2 guidance describes conceptual soundness, outcomes analysis and ongoing monitoring as components of validation, supported by effective challenge from objective experts.

Is model validation the same as cross-validation?

No. Cross-validation is one technique for estimating performance, while model validation is the broader review of whether the model fits its intended use.

How many test cases does model validation need?

There is no single number, but with few positive cases each one moves rates such as recall substantially. Teams should report counts and repeat comparisons where data allows.