The rerun should have been routine. The team loaded the same model, set the temperature to zero, reused the saved seed, and ran the same evaluation command. The score still moved.
Nothing obvious explains the gap. The code commit matches. The configuration looks complete. Yet the team can no longer tell whether the earlier number will hold in a release, an audit, or even another run next week.
The problem is not necessarily a forgotten setting. A seed controls a source of randomness that the software exposes to it. It does not identify the data version, reconstruct preprocessing, pin every dependency, define the evaluation protocol, or control how numerical operations are scheduled across hardware.
LLM reproducibility depends on the full set of conditions that produced a result. The seed is one of those conditions, not an identifier for the run.
A seed controls a random process, not the whole run
Teams use seeds for a good reason. When a framework routes random operations through a seeded generator, the same seed can help reproduce sampling, parameter initialization, data shuffling, or another stochastic step. The exact scope depends on the framework and workflow.
Two executions can share a seed while reading different records, applying different preprocessing, loading different library builds, using different evaluation splits, or running on a different serving configuration.
Reviews of machine-learning reproducibility trace the problem across experiment design, method, code, data, implementation, and reporting. The implication for an operating team is direct: “same seed” answers only one question. It does not establish that the two executions are comparable.
Before comparing results, the team needs to know what the seed actually controlled and which other conditions were held constant.
Run state disappears in more than one place
Some conditions vanish because nobody wrote them down precisely. Hyperparameters, prompts, preprocessing logic, data splits, metric definitions, stopping criteria, and the exact data state may live partly in a configuration file and partly in a teammate's memory. The file helps only when it resolves the versions and artifacts the execution actually used.
Other conditions hide in the environment. Framework versions, tokenizer builds, drivers, operating system libraries, container base images, and compiler settings are easy to treat as background. The application code can remain unchanged while one dependency changes what reaches the model or how the model executes.
The least visible conditions sit in the physical and numerical execution path. GPU type and count, batch construction, kernel choice, parallel reduction order, and numerical precision may shape the calculation. These details are not always exposed in the application configuration. With a hosted service, some may never be under the team's control.
What the team should do depends on which layer is missing. Record the known experiment conditions, package the software environment where possible, and either control the numerical execution path or state its limits.
Greedy decoding can diverge without random sampling
Temperature zero makes this limitation easier to see. With greedy decoding, there may be no sampled token for a seed to stabilize, yet outputs can still diverge.
Floating-point arithmetic is sensitive to operation order because intermediate results are rounded to a limited precision. Parallel hardware can group reductions differently as batch size, GPU count, or kernel execution changes. The underlying model and input can stay the same while a small numerical difference appears in a token score.
Greedy decoding then turns a continuous score difference into a discrete choice. If two tokens have nearly equal scores, a small numerical change can switch which token ranks first. The selected token becomes part of the next input, so an early difference can send the continuation down another path.
Yuan and colleagues demonstrated this mechanism in a controlled study of LLM inference. For DeepSeek-R1-Distill-Qwen-7B on AIME’24, BF16 precision and greedy decoding produced an accuracy standard deviation of 9.15 percentage points across 12 runtime configurations. The same experiment reported an average output-length standard deviation of 9,189.53 tokens. These results apply to the tested model, benchmark, precision, and configurations; they are not a universal variance rate for LLMs.
The study matters because it shows why a seed cannot be the universal remedy. The relevant difference may come from numerical execution rather than a random draw. System-specific work on deterministic inference therefore moves the control lower in the stack, toward batch-invariant operations and kernels, with engineering tradeoffs of its own.
- Same prompt
- Same seed
- Greedy decoding
Nearly tied scores
A small numerical change can reverse the order.
Token A selected
The next step includes Token A.
Token B selected
The next step includes Token B.
Consequence: the selected token becomes part of the next input, so the continuations can diverge.
Build an evidence envelope around the result
A reproducibility record should tell a second operator which conditions must match, which artifacts can be restored, and which conditions remain outside the team's control.
Consider a hypothetical support-ticket routing evaluation that runs through a hosted LLM API. The dataset hash, prompt version, seed, and scoring script all match the prior run, but the aggregate score slips. If the provider does not expose the serving batch or hardware path, the team cannot honestly call the executions identical. The team can state that the application-controlled conditions matched while the infrastructure boundary remains unresolved. That conclusion is narrower, but another reviewer can evaluate it.
Teams do not need to freeze every variable forever. They do need enough evidence to state what a result proves.
If the data, code, environment, execution path, and evaluation all match closely enough for the decision, the rerun is a meaningful reproducibility test. If one layer cannot be matched, the result may still be useful, but the comparison must state that limitation. The evidence should say what changed rather than hiding the difference behind a shared seed.
Reproduction does not always mean byte-identical output
The required level of sameness depends on the decision.
A debugging team may need the same failure to reappear. A model-selection team may need rankings and metrics to remain within a defined tolerance. A regulated review may need a traceable explanation of the inputs and conditions even when the serving provider cannot expose every hardware detail.
These are different acceptance rules. Declaring a rerun “reproduced” without naming the rule creates another ambiguity.
The team should define the result identity and tolerance before rerunning. It should also separate three outcomes:
- The result matches under the stated conditions and acceptance rule.
- The result differs, and the evidence identifies a changed condition that can be tested.
- The result differs across a condition the team cannot reconstruct or control, so the original claim must remain qualified.
This makes “reproduced” a testable claim rather than a blanket promise.
Syntitan covers the data-state layer
Once the run is treated as a set of linked conditions, responsibility across the stack becomes clearer. Model registries, code repositories, experiment trackers, and serving systems hold different parts of the execution record. The data state needs equally precise versioning.
Syntitan addresses that data-state layer. A Release State fixes the approved data state for an AI task. Run Binding connects an execution to that state. Diff identifies changes between released states, and Reproduce restores a recorded state for investigation.
This lets the team identify the data state used by the run and isolate changes on the data side. It does not freeze the model, application code, prompt, dependencies, GPU topology, serving batch, or evaluation system. Those records must come from the systems that own them.
This boundary is useful during an investigation. If a team restores the same released data state and holds the other material conditions constant, it can test whether a data change contributed to the result. If the output still moves while the data state remains fixed, the evidence directs attention toward the remaining model, software, infrastructure, or evaluation conditions.
Neither outcome proves root cause automatically. It gives the team a way to stop treating the run as one opaque configuration and test one evidence layer at a time.
The handoff test for an AI result
Take one result that matters to a release, customer decision, or internal approval. Give its run record to someone who did not produce it.
Can that person identify the target output, restore the data state, resolve the model and code, rebuild the environment, understand what the seed controlled, match or qualify the numerical execution, and apply the same evaluation rule?
The first layer that another operator cannot reconstruct defines both the next task and the limit of the original claim. A result becomes defensible when another operator can reconstruct it within a declared boundary, not when one familiar setting happens to match.
Keep the seed in the record. Just do not ask it to carry the whole run.
Make the data state behind each run part of the reproducibility record. See how Syntitan connects an approved data state to an AI run.
