A claims team at an insurer uses an AI assistant to draft claim summaries from policy documents and adjuster notes. An email arrives from their model provider: the model behind the assistant retires in eight weeks. The team runs the replacement on its evaluation set of closed claims, with the same questions and the same pass criteria. Scores drop on questions about coverage limits. Meanwhile, the knowledge team has rebuilt the policy search index, adding next year’s policy wording and changing how documents are split into passages. Did the drop come from the new model, the new index, or both? (Illustration)
A model migration test compares two models only when everything else the task reads stays the same. Freezing the evaluation set fixes the questions. It does not fix the documents, tables, and index the task retrieves while answering them. Pin those to a data version too, and run the current model on that version before its retirement date. Once access ends, you cannot establish a new baseline using that model.
Key takeaways
- A frozen evaluation set fixes the questions, not the retrieval index, reference tables, and preprocessing the task uses at run time. Pin those to a named data version for the length of the test.
- Run the current model on the pinned version before its retirement date. Once access ends, you cannot establish a new baseline using that model.
- Separate the effects with three runs: the current model on the pinned data, the replacement on the same data, then the same replacement setup on the updated data.
- Record the model version, data version, evaluation set version, pass criteria, and date with every run.
- 01Retirement notice
The provider sets a retirement date eight weeks out.
- 02Index rebuilt
The knowledge team adds next year’s policy wording and changes passage splits.
- 03Replacement tested
Same evaluation set, same criteria, new index underneath.
- 04Cause unclear
Scores drop on coverage-limit questions. The model, the index, or both?
Record the baseline while you still have access
Hosted models come with retirement dates, and providers publish them with notice. One major provider gives at least six months’ notice for generally available models, according to its deprecations page. That provider shuts down a large batch of older models on October 23, 2026, and on October 14 it announced another round for January 2027. Another provider commits to at least 60 days for publicly released models and advises testing applications with newer models well before the retirement date. For models launched on Amazon Bedrock on or after September 7, 2026, a legacy period of six months or 45 days precedes end of life, and migration does not happen automatically.
The notice period is also the window for measuring the model you have now. A comparison needs a baseline, and the baseline has to come from the current model. A September 2026 preprint that studied model migrations in open-source applications found that many migration commits landed after the shutdown, that model identifiers were usually hard-coded, and that few commit messages mentioned evaluation or regression testing. Without a recorded baseline, a team can measure the replacement but cannot compare it against the old model on the same data.
Why a frozen evaluation set is not a frozen test
Migration guides already ask teams to hold the test still. Microsoft’s model migration guide tells teams to freeze the inputs, ground truths, and success criteria for the whole migration, because changing any of them breaks the comparison between source and target. It also advises replaying the current configuration unchanged on the new model before tuning prompts, to separate what the model changed from what the team changed.
In practice, a team may freeze the evaluation questions while leaving the retrieval index and reference tables free to change. For a retrieval or agent workload, the answers also depend on what the task reads while answering: the passages the index returns, the reference tables it looks up, and the preprocessing that shapes them. Other teams often own those and change them on their own schedule. The engineers behind Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) warned that an input owned by a separate team can be updated at any time. In the claims example, the questions stayed fixed while the index underneath them changed.
Three runs that separate the model effect from the data effect
Change one thing per run, and keep a record of the data version each run used. DoorDash’s engineering team described the same approach for ML model migrations: isolate the risky components and move them one at a time, so the team can measure each effect on its own. The table below applies that idea to a model change that overlaps with a data change. It is CUBIG’s practical guide, and the version numbers are illustrations.
| Run | Model | Data version | What the comparison shows |
|---|---|---|---|
| Run Abaseline | Current model, before its retirement date | Pinned version (v12) | The reference point. Record it while the current model still runs. |
| Run Bmodel change | Replacement model, prompts unchanged | Same pinned version (v12) | Run A to Run B shows the effect of the model change. Record prompt adjustments as separate runs. |
| Run Cdata change | Same replacement model, prompt, and parameters as Run B | Updated version (v13) | Run B to Run C shows the effect of the data change on the new model. |
Each run here is an evaluation configuration, not a single call. Repeat it as often as you need to see how much results vary between runs. These comparisons measure each change under the tested conditions. They do not establish a data effect that holds for every model.
For the claims team, Run A needs the pre-rebuild index. If the knowledge team kept that version, the team can still test both configurations on it, and the old index must stay available until Run B finishes. If the index was rebuilt in place and the old version cannot be recovered, the team can no longer make that comparison. That is why step 2 below asks for named versions. If B performs similarly to A on the same cases but C drops, investigate the index rebuild first by checking which passages it now returns for coverage-limit questions. If B already drops, the model change likely accounts for part of it, and prompt adjustments can be tested as separate runs against the same version.
When a model change forces a data change
Sometimes the data has to change with the model. If the retiring model is the embedding model behind retrieval, the index must be rebuilt with the replacement, so Run B and Run C collapse into one. A replacement with a different context length or output format can also call for new passage sizes or clearer field descriptions in reference tables.
In these cases, keep what you can fixed: the questions, the pass criteria, and the source documents. Before judging the answers, compare what the task retrieved for the same questions under the old and new index. If the evidence cannot separate the causes, record the result as inconclusive rather than assigning the change to the model or the data.
How to test a model migration without mixing in data changes
- Find every place the model is called. List the model identifiers in code and configuration, with each one’s retirement date and the tasks that use it.
- Pin the data version for the test window. Ask the teams that own the index, reference tables, and preprocessing to release changes as a new named version instead of updating in place.
- Record the baseline now. Run the current model on the pinned version and the frozen golden dataset before the retirement date.
- Swap only the model first. Run the replacement unchanged on the same version, then record each prompt or parameter change as its own run.
- Test the updated data separately. Keep the replacement model, prompt, parameters, and evaluation criteria fixed. Change only the data version, then compare the results case by case.
- Version the golden dataset when you add to it. New production cases make the next migration test stronger. Save them as a new evaluation set version so earlier results stay comparable.
Done by hand, this means keeping a ledger that ties every run to the exact model version, data version, and evaluation conditions. The ledger gets hard to maintain when several teams update the index and tables on their own schedules and model changes arrive several times a year. Keeping that ledger is the part of the work CUBIG built into Syntitan.
Where Syntitan fits
The team chooses the replacement model, adjusts prompts, and decides when to roll out. Syntitan, CUBIG’s AI-ready data platform, covers the data side of the comparison. In Ready for Model, it holds the model task and the evaluation conditions fixed and compares results before and after preparation against the criteria the team agreed on. Each successful run is saved as a new Version, so the team can tell which data a result came from. Checking what can be reused after a model or data change is planned for December. Approval for production remains the team’s decision.
Knowing which data version a result came from is part of preparing AI-ready data. Pick one AI task whose model has a published retirement date. Check whether you can name the data version used in its last evaluation, and whether you still have access to the current model long enough to record a baseline on it. Explore Syntitan to see how data versions, evaluation conditions, and results stay linked.
Related reading
- Which version of the data passed the test?: what to revalidate after the data changes.
- Model versioning is not enough: why reproducing an AI result needs the data state too.
- LLM benchmark comparison: when two scores are comparable.
- Golden dataset: the glossary definition.
