Yesterday’s AI run produced an acceptable result. Today’s run did not. The prompt looks the same, the workflow appears unchanged, and the output offers few clues about where the difference began.
AI reproducibility starts with evidence, not a rerun alone. Rerunning the workflow may confirm that the difference persists, but it does not automatically explain it. A useful investigation starts by asking whether the team retained enough evidence to identify both runs, recover their input states, compare their execution context, and document what happened during the next attempt.
This guide provides an evidence checklist for that work. It complements the AI Data Failure Review Map, which helps teams name and locate possible review conditions. The map identifies where to look. This checklist focuses on what to collect and compare before claiming that a result can or cannot be reproduced.
A changed output is not yet an explanation
Two outputs can differ for many reasons. The input data may have changed. A schema or preprocessing rule may have drifted. A runtime parameter, model version, dependency, permission, or external service may be different. The first run may also be missing the records needed for a valid comparison.
The output alone rarely separates these possibilities. It shows the consequence, not necessarily the state and sequence that produced it.
That distinction matters because three statements require different evidence:
- The outputs differ. A comparison can establish an observed difference.
- The run cannot be reproduced. A controlled attempt can establish that the recorded state and context did not produce the expected result.
- A specific change caused the difference. A causal conclusion requires evidence that rules out credible alternatives.
Treating the first statement as proof of the third creates a premature diagnosis. The investigation should instead build an evidence bundle that makes the comparison and reproduction attempt inspectable.
Define the evidence bundle before rerunning anything
An evidence bundle is the set of records needed to connect an output to the data state, execution context, and review actions behind it. The exact fields depend on the workflow, but the bundle should answer four practical questions:
- Which run produced this output?
- Which exact data state did that run use?
- Which code, configuration, model, tools, and dependencies shaped execution?
- What changed between the reference run and the run under review?
The following checklist is a starting point, not a universal standard.
| Evidence group | Minimum record | Question it answers | Common gap |
|---|---|---|---|
| Run identity | Unique run ID, start and end time, owner or service, status | Which execution are we reviewing? | A screenshot or output exists without an addressable run. |
| Input data state | Dataset name, source, digest or version, schema, data window | What exact state entered the workflow? | The dataset name is known, but the version or time window is not. |
| Run-data relationship | Explicit binding between the run and its input state | Can we prove which data state the run used? | The run and dataset are recorded separately with no durable relationship. |
| Execution context | Parameters, flags, environment, permissions, model and tool versions | What conditions shaped execution? | Defaults or runtime overrides were not captured. |
| Transformation trail | Ordered jobs, queries, code revision, preprocessing steps, upstream sources | How did the input become the material used by the run? | The final table is retained, but intermediate transformations are not traceable. |
| Output record | Output identifier, evaluation result, timestamp, destination | Which result belongs to this run? | The output was copied or exported without its run identity. |
| Comparison record | Reference state, reviewed state, selected fields, observed differences | What changed, and what remained constant? | Teams compare outputs without comparing inputs and execution context. |
| Reproduction record | Hypothesis, controlled inputs, procedure, result, unresolved differences | What did the rerun test? | A rerun is labeled a reproduction without a stated reference state or controlled scope. |
Current evidence systems preserve parts of this bundle in different ways. MLflow Dataset Tracking documents dataset properties such as name, digest, source, schema, and profile, along with the context in which a dataset was used. OpenLineage provides structures for run metadata, including a dataset version facet and an execution parameters facet. These fields make evidence representable. They do not prove that a particular implementation captured complete or correct records.
Identify the exact run and its decision context
Begin with the run, not the output file. Record a unique run identifier, its timing, execution owner, status, and the workflow or decision it supported. If the result influenced a downstream approval, report, model evaluation, or customer-facing action, record that context too.
This step prevents a common failure: comparing two outputs that look related but came from different jobs, environments, or decision windows. It also gives the review a stable object. Team members can discuss one addressable run instead of referring to “yesterday’s version” or “the result in the screenshot.”
If no unique run record exists, state that limitation before proceeding. Reconstruction may still be possible, but confidence in the comparison should be lower.
Bind the run to the exact input data state
A dataset name is not an input state. The same name may point to changing records, a different data window, a revised schema, or a newly processed representation.
Record the source, version or digest, schema, relevant profile, selection window, and any split or sample used by the run. Then preserve an explicit relationship between that state and the run. This relationship addresses Missing run-data binding: the team may have a run record and a data record, yet still lack evidence that they belong together.
The record must also be usable. A version label has limited value if the underlying state cannot be retrieved, inspected, or reconstructed under the applicable retention and access rules.
Preserve execution configuration, code, and dependencies
The same input state can produce a different outcome when execution context changes. Capture the parameters supplied before and during the run, including defaults and overrides. Record the relevant code revision, preprocessing logic, model and tool versions, dependency lock, environment, permissions, and external services.
Do not assume that the configured value was the effective value. If the platform supports it, preserve the runtime-resolved configuration rather than only the intended configuration. OpenLineage’s execution parameters facet, for example, is designed to attach supplied parameters and configurations to a job run. Whether a specific deployment captures everything necessary remains an implementation question.
Permissions deserve separate attention. A workflow can follow the same code path but read a different object, omit a protected field, or fail to execute a transformation when its effective access changes.
Compare states before comparing explanations
Once both runs have addressable evidence, compare the reference and reviewed states. Start with the dimensions most likely to affect the result:
- input digest, version, schema, profile, and data window;
- code, preprocessing, parameters, environment, and permissions;
- model, tool, and dependency versions;
- upstream sources and ordered transformations;
- output and evaluation records.
Keep the comparison bounded. A large undirected diff creates noise and can encourage reviewers to select the first visible change as the explanation. Document which fields were compared, why they were selected, what changed, and what remained constant.
Lineage can help define that scope. Within its documented requirements, Databricks Unity Catalog lineage can show upstream sources and transformations for supported workflows. Its documentation also records coverage, permission, and retention limitations. A missing node may reflect a capture boundary rather than proof that no dependency existed.
Reproduce with a stated hypothesis and controlled scope
A reproduction attempt should test a specific claim. Examples include:
- the reference data state and recorded configuration produce the reference output;
- the reviewed data state produces the changed output under the same execution context;
- one identified configuration change accounts for the observed difference when other relevant conditions remain controlled.
Before executing, record the reference state, the hypothesis, the fields held constant, the field deliberately changed, the procedure, and the expected observation. After execution, retain the new run identity, output, comparison, and any unexpected deviation.
A successful rerun can show that a recorded state and procedure produced a consistent observation in that attempt. It does not establish that every future run will behave identically. A failed rerun can expose missing or incorrect evidence, but it does not identify root cause by itself.
Record what the attempt proves and what it does not
Finish the review with an evidence statement, not a verdict that exceeds the test.
For example:
Using the recorded input state, code revision, and execution parameters, the review run did not reproduce the reference output. The comparison found a schema difference and a missing runtime configuration record. Further review is required before attributing the output difference to either condition.
This format separates observation, known evidence gaps, and the next review action. It also gives another reviewer enough context to challenge or continue the investigation.
If the evidence bundle remains incomplete, name the missing record. “Not reproducible” is weaker and less useful than “the prior run cannot be reconstructed because its input version and effective execution configuration were not retained.”
Use the review map to choose the next investigation area
The evidence checklist may reveal several possible review conditions. A schema difference can point to Schema drift. A run with no addressable input relationship can point to Missing run-data binding. An unrecorded environment or flag can point to missing execution configuration.
The AI Data Failure Review Map organizes the approved 36-condition reference by workflow stage, evidence action, and Condition type. Use it to name the next area to inspect. Do not treat the map as a diagnosis or assume that adjacent conditions form a causal sequence.
Where Syntitan fits and where it does not
For CUBIG, the point is not to retain metadata for its own sake. AI-ready data should come with evidence that teams can use to judge whether a data state is fit for a specific AI task. That work follows an operating flow from diagnosis and improvement to verification and history.
The current Syntitan product page describes Release State, Run Binding, Diff, and Reproduce. Together, these concepts align with the evidence path in this checklist: preserve an addressable data state, connect it to a run, compare relevant states, and retain enough context to attempt reproduction.
That alignment has limits. The checklist is not a Syntitan feature inventory, and it does not support a claim that Syntitan automatically diagnoses every changed AI result. Teams still need to define the relevant evidence, verify capture coverage, interpret the comparison, and decide what a reproduction attempt can establish.
Make the next run easier to review
Reproducibility work begins before a result looks wrong. A team that records the run only after an incident may discover that the input state, configuration, or transformation trail has already been lost.
The practical goal is not to collect every possible field. It is to preserve enough evidence to connect a result to one addressable data state and execution context, compare that state with a reference, and describe the outcome of a controlled reproduction attempt.
Build that evidence path before the next investigation. See how Syntitan supports verifiable data states.
