A team swaps in a new model and the dashboard shows accuracy up four points. Nobody can say whether the test records, the labels or the data preparation changed along with it.
AI model evaluation is the process of measuring how well a model or agent performs a specific task on data it did not learn from, using metrics and success criteria agreed in advance, so that versions of the model or the data can be compared under the same conditions and the result can be repeated.
By Ho Bae, Founder & CEO, CUBIG Corp. · Updated November 2026.
In CUBIG’s UCI Polish Companies Bankruptcy experiment (October 2026), four models were each retrained on their own training data before and after additional preparation, which differed by model. All were scored on the same 1,942 evaluation records from one split. Random Forest F1 rose from 0.4091 to 0.6098, while ROC-AUC slipped from 0.9269 to 0.9253, average precision rose from 0.6437 to 0.6728 and false positives went from 2 to 11. Read through more than one metric, the same evaluation told a more careful story than the headline number.
Key takeaways
- An evaluation result is comparable only when the evaluation records, metrics, threshold and every setting other than the change being tested stay fixed between versions.
- A single score can hide the trade-off that matters. In CUBIG’s UCI bankruptcy run, Random Forest F1 rose while false positives increased and its ranking metrics were mixed: ROC-AUC slipped slightly as average precision rose.
- Several common causes of misleading results sit on the data side: leakage, unrepresentative samples, a test set that was redrawn, or wrong labels.
What is AI model evaluation?
AI model evaluation answers one question for one task: does this model, on this data, meet the bar the business set? The answer is a measurement taken on records the model did not see during training, against metrics chosen for the mistakes the task can least afford.
An evaluation is rarely a one-time step. A model is evaluated when it is chosen, again when the data or its preparation changes, and again when the task changes. Each evaluation is most useful when it can be compared with the last one, so the records, metrics and settings behind a score need to be written down. NIST’s AI Risk Management Framework (NIST AI 100-1, January 2023) includes this in its MEASURE function: test sets, metrics and details about the tools used for testing, evaluation, verification and validation are documented.
How is AI model evaluation different from model validation and testing?
The three terms overlap in everyday use. The table separates them by what each is for and which data it uses.
| Term | Used for | Data it uses | When it happens |
|---|---|---|---|
| Validation | Tuning settings and choosing between versions | A validation set or cross-validation folds | During development |
| Testing | Checking that the system behaves as specified, including software and integration behavior | Test cases and test data | Before release and after changes |
| Evaluation | Measuring task performance against agreed criteria and comparing versions | A fixed evaluation set (often called the test set) kept apart from training and tuning | At selection and after each data or model change |
Some tools call the validation set an eval set, and many ML texts call the evaluation set the test set, so the label on a score says little on its own. What matters is which records a reported score came from, and whether those records influenced any choice made before the score was taken. The evaluation set and cross-validation entries cover the distinction in more detail.
How does an AI model evaluation work?
A sound evaluation can be built in six steps. The order matters, because each decision is harder to undo once results have been seen.
- Define the task and the bar. Write down what the model decides and what result the business would accept, before any scores exist.
- Fix the evaluation data. Set aside records that reflect the cases the task will meet, including exceptions, with a split that respects time order and grouping.
- Choose metrics for the cost of errors. Pick metrics that reflect which mistakes are expensive for this task, and report more than one.
- Run every version under the same conditions. Same evaluation records, same threshold, and every setting other than the change being tested held fixed, so a difference in results comes from that change.
- Read the result against the bar. An improvement over the last version can still fall short of what the task needs.
- Record what produced the result. Keep the data version, preparation, model settings and evaluation conditions with the score, so the evaluation can be repeated when something changes.
Step six is the easiest to skip. A result that cannot be traced back to its data version cannot be rechecked, which is the problem described in which data version passed the test.
CUBIG Shorts
Same Model. Two Different Results.
A short take on step four: the same model giving a different result when only the state of the data changes.
Which metrics should an AI model evaluation report?
No single metric covers every task. A model that predicts a rare outcome can reach high accuracy by rarely predicting it. In the UCI bankruptcy evaluation set, bankruptcies were about 5.3% of the 1,942 records, so a model that never flagged one would still be right on roughly 95% of them. Before the additional preparation, Random Forest scored 95.98% accuracy while missing 76 of the 103 bankruptcies.
| Metric | What it shows | What it can hide |
|---|---|---|
| Accuracy | Share of all predictions that are correct | Poor performance on rare but important cases |
| Precision | Of the cases flagged, how many were right | How many real cases were missed |
| Recall | Of the real cases, how many were found | How many false alarms it took to find them |
| F1 | A balance of precision and recall at one threshold | Changes in ranking quality across other thresholds |
| ROC-AUC, average precision | How well the model ranks cases across all thresholds | Performance at the threshold actually used |
Reporting metrics that pull in different directions is often more useful than one headline. In the UCI run, Random Forest recall rose from 26.21% to 48.54% while precision fell from 93.10% to 81.97%, and the write-up reads the F1 gain as a shift in the operating point rather than an across-the-board ranking improvement. The write-up also notes that threshold tuning or class weighting on the baseline data was not compared.
Why do AI model evaluations give misleading results?
Several common causes of misleading evaluations sit in the data rather than the model. Leakage alone is widespread: in a 2023 paper in Patterns, Sayash Kapoor and Arvind Narayanan surveyed earlier reviews and found leakage affecting at least 294 papers across 17 fields of machine-learning-based science.
Data leakage
Information from the evaluation data, or from after the outcome, reaches training. See data leakage.
Unrepresentative samples
The evaluation set misses the channels, periods or exceptions the task will meet. See sampling bias.
A test that moved
The evaluation records were redrawn between versions, so the test changed along with the model.
Tuning on the test
Repeated choices made by looking at the same results fit the model to those records. See overfitting.
Each of these problems can make a model look better or worse than it is, and none of them shows up in the score itself. Catching them takes the records behind the score: which data was used, how it was split and prepared, and what changed since the last evaluation. Data quality for AI covers the data checks that come before an evaluation.
How does CUBIG help with the data side of AI model evaluation?
Syntitan keeps the original Dataset as it is and starts each use case from a copy in its own Project, so an evaluation is tied to one task. Ready for Raw Data checks the state of that copy along six axes, including reproducibility and traceability, and records the prepared result as a project Version.
Ready for Model runs the task on the data before and after preparation under the same evaluation conditions and shows the two results side by side. Ready for Model also exports the files needed to rerun model training and scoring, which is how the UCI comparison above can be rerun from its published proof kit; the kit does not rerun the upstream preparation steps.
Ready for Agent builds Tasks and Cases for agent workflows, so the situations an agent must handle can be written as Cases with expected behavior and judgment criteria; connecting agent runs to those Cases is in development.
An evaluation narrows the uncertainty about a model; the decision to use the model still rests on the bar the business set. Try it on your data.
