AI-Ready Data

Data Versioning: Three Release Questions to Ask of AI Data

Stacked translucent purple panels with a blue panel in the corner, representing data versions and the region that changed

Data versioning preserves earlier states of a dataset so a team can see what changed and return to an earlier state. You can compare the rows that changed, but the diff alone cannot tell you how the change affects model performance. That effect has to be measured in the results.

Say you are told the training data was fixed before the next model run. The change list looks reasonable: several derived variables added, underrepresented cases filled in, input variables cleaned up. The list alone does not tell you whether the result improved.

Software teams check every release with three questions: what changed, how was it verified, and can we roll it back. For AI data, each one needs new wording. Under the same conditions, how much did the results change before and after? What did we agree counts as a pass, and who set that criterion? When conditions change, can we run the same comparison again?

Adapted from Data Has No Diff by Daesung Kang, CTO of CUBIG, on the CUBIG Tech Blog (November 2026).

Key takeaways

  • A data diff shows which rows or values changed. To see how the change affects a model, train on each version with the same model architecture and training settings, then compare on the same evaluation data and criteria.
  • Data quality checks work like unit tests. A column can pass every check and still be unusable for the task, as call duration is for deciding whom to call.
  • A score without agreed acceptance criteria is neither a pass nor a fail. The same result can pass for one team and fail for another.
  • An earlier data version is less a safe place to return to than a baseline for measuring again. Its good result held under the model and evaluation conditions used at the time.

Why isn’t a data diff enough to review a data change?

Because a diff shows what changed in the data, not what the change does to the model. A data diff does not work the way a code diff does. In code review, reading a changed conditional lets you roughly infer which inputs will now behave differently. You cannot know every effect, but there is a path from the change to its effect. A newly added row of data can look perfectly fine on its own, and a model learns from the whole distribution rather than from any single row.

Engineers at Google called this CACE, Changing Anything Changes Everything, in Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015). When the distribution of one input changes, the way the model uses every other input can change too. The same paper has a section titled Data Dependencies Cost More than Code Dependencies.

So the effect of a data change is read in the results. For a change to training data, train a model separately on each version with the same model architecture and training settings, then compare both on the same evaluation data with the same criteria. In software terms, that is closer to running the same test suite twice as a regression test than to reading code in review.

Why can data pass quality checks and still fail the task?

Because most data quality checks resemble unit tests. They look for missing values, correct formats and values within range. They do not check whether the product does what the user needs, which is the job of an acceptance test.

In CUBIG’s published UCI Bank Marketing experiment, the bank’s telemarketing data includes a call duration variable. It is numeric, never empty and plausible, so it passes every one of those checks. But the value exists only after the call ends. For a task that decides in advance whom to call, it cannot be used, and the UCI dataset description notes that it should be discarded if the intention is a realistic predictive model. You see the problem only once you know when the prediction will be used.

The same experiment also shows how the comparison was run. A Random Forest model with the same settings was trained separately on the original and the prepared training data, and both models were evaluated on the same 824 original records with the prediction rule unchanged. The preparation both reduced the input variables and added subscription examples, so the comparison shows their combined effect, not the effect of either step alone.

F1 rose after preparation, while false alarms on those 824 records rose from 16 to 39. This comparison used historical records that included call duration. It illustrates a trade-off in that evaluation, not how well the model would select customers before a call.

Even inside that evaluation, the score does not settle the question. Finding more actual subscribers can count as a pass for a team that cares most about missing fewer of them, and a fail for a team without enough people to review what gets flagged. In software, acceptance criteria are written before anything is built. In AI projects, scores are often reported while those criteria are still blank.

What do you need to investigate a changed AI result?

You need the state at the time the result was produced. An incident postmortem in software starts there: which commit was deployed, what the configuration was, what changed in between. With those records you can narrow down the cause. Without them, all that is left is guesswork.

When an AI result changes, there is more to look for. Beyond the model, the training data, search index, reference documents and evaluation conditions can all move together. When the model and the data change at the same time, you cannot tell which change produced the different answer.

Data problems also surface late. At CHI 2021, Sambasivan and colleagues at Google Research named these data cascades: data problems that compound and surface late. In the cases they studied, cascades were hard to diagnose and lacked clear indicators, and those triggered by drift were among the slowest to surface.

Logs may show what happened, but not under which data and conditions it happened. A postmortem is easier to investigate when each result is linked to the data version, settings and evaluation conditions that produced it. Picture an incident postmortem without commit hashes, and you get a sense of the gap.

Can data versioning give you somewhere to roll back to?

Only if the earlier state is kept instead of overwritten. Software teams do not delete previous releases. When something goes wrong, they find the tag and roll back, and because that option exists, they can take the risk of a new deployment.

Data often gets overwritten. The cleaned file takes the place of the original, and what was dropped at which step lives only in the memory of whoever did the work. Go back to the change list from the opening. If the cleaned training data replaced the original, there is nothing left to retrain on for a before-and-after comparison, and no way to tell later whether a drop in results came from that cleanup or from something else. There is nowhere to roll back to and no baseline to compare against.

There is a second difference. A previous software release is usually a state that worked. A good result for a previous data version was confirmed under the model and evaluation conditions used at the time. Once the data, the model or the criteria change, that result no longer vouches for the new conditions; it becomes the baseline for a new comparison. A data version is less a safe place to return to than a yardstick for measuring again. Deciding which earlier checks still apply after a change is covered in which version of the data passed the test.

Where does the software analogy break?

Software habits carry over reasonably well, but in three places the practice has to be redesigned for data.

  1. A green light means something different. A code test gives the same answer for the same input; a data comparison is statistical. In a single internal split from CUBIG’s UCI Wilt experiment, the evaluation set has only 15 positives out of 868 image segments, so missing a single one changes recall by about 6.67 percentage points. Whether a result from one split holds on other splits has to be checked separately.
  2. Acceptance criteria need input from both the business and engineering teams. How many false alarms are acceptable depends on the business team that reviews them, and engineering has to show what each option costs. The CTO’s job is not to set those criteria alone but to refuse reports that have none.
  3. The right preparation cannot be decided in advance. Whether adding certain variables or filling in certain cases was the right choice can only be known by comparing results on that task. Comparison under the same conditions, rather than any particular preparation method, has to be the standard procedure. Five UCI experiments run this way show the results and trade-offs.

The paper by Sambasivan and colleagues is titled Everyone wants to do the model work, not the data work. Making data work verifiable, so that it gets the credit it deserves, is a job for engineering leadership.

What are the three release questions for AI data?

Put side by side, the three questions read like this.

The three software release questions, rewritten for AI data
What we ask of softwareWhat we ask of data
What changed?With each data version trained under the same model architecture and settings, how much did results on the same evaluation data change?
How was it verified?What did we agree counts as a pass, and who set that criterion?
Can we roll it back?Is each result kept with the data version, settings and evaluation conditions that produced it, so the same comparison can be run again when conditions change?

All three can be asked regardless of model architecture. If a team cannot answer them, it may be missing the data operations records it needs to judge results and trace causes.

Where does CUBIG fit in comparing data versions?

CUBIG prepares and checks enterprise data for specific AI tasks, so teams can compare results before and after a data change under the same conditions. Syntitan, CUBIG’s AI-ready data platform, keeps the original Dataset intact and creates a new Version as data is prepared. In supported Ready for Model workflows, teams can compare results before and after preparation under the same evaluation conditions.

Whether an earlier Version’s results can serve as the baseline for a later comparison depends on the supported workflow, so confirm it for your setup.

Whether a prepared Version fits the work, including how many false alarms the team can review, remains the team’s call. Try it on your data in Syntitan: compare the data behind one AI task before and after preparation, under the same conditions.

Explore Syntitan, CUBIG’s AI-Ready Data Platform

FAQ

Why is a data diff not enough to review an ML data change?

A data diff shows which records or values changed. To understand how those changes affect a model, teams also need to compare results under consistent evaluation conditions.

Why is passing data quality checks not enough?

Checks for missing values, formats and ranges work like unit tests. A column such as call duration can pass them all and still be unavailable at the moment the prediction is made.

Is data versioning enough to roll back an AI training dataset?

Keeping the original and each prepared version, instead of overwriting them, gives you somewhere to go back to. Even then, an earlier version is a baseline for a new comparison. Its good result was confirmed under the model and evaluation conditions used at the time, not under new ones.