This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or verified determinism on it.
It is specified around a constraint none of the other nine share: the same input must produce the same output, on demand, years later.
That sounds like it should be free. It is not, and the reason is worth understanding before you specify anything.
The same machine can give you two different answers
Set temperature to zero. Fix the seed. Same model file, same prompt, same GPU, same afternoon. Run it twice.
You can still get different text. This surprises almost everyone, including people who have worked with these systems for years, and it is not a bug in any particular tool.
Why
Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bits. That is a property of the number format, not of any library.
A GPU computes a sum across thousands of threads by combining partial results, and the order in which those partials arrive is not fixed. Atomic accumulation, different reduction trees, different kernel selections — any of these can change the order between runs. The differences are tiny, around the last bit of a float.
Then the model amplifies them. Greedy decoding picks the highest-scoring token, and when two candidates are nearly tied, a last-bit difference decides which one wins. After that the continuations diverge completely and you have two different answers to the same question.
So temperature zero removes sampling randomness. It does nothing about numerical randomness, and that is the part that bites.
Two requirements that get confused
| You need | When | What it costs |
|---|---|---|
| Auditability — a record of what produced an output | Most of the time | Cheap; a manifest and disciplined logging |
| Reproducibility — the same output, recomputed | Regulated research, published results, forensic and clinical work | Expensive; this whole architecture |
If somebody needs to know what happened, log the model hash, the parameters, the prompt and the output, and you are finished. The air-gapped machine covers the manifest discipline that makes that hold up.
If somebody needs to recompute it and get the same thing — because a regulator, a reviewer or a court will ask — the rest of this page applies.
Hardware for a result you must repeat
-
Best Overall
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
ECC eliminates the one nondeterminism source no software switch can address, and the professional line stays available longer — which matters when the specification is a particular card rather than a class of card.
-
Best Value
NVIDIA GeForce RTX 5090
Fast enough to absorb the throughput cost of deterministic kernels and batch size one. No ECC, and consumer cards are discontinued sooner, so buy the identical spare at the same time or accept the risk.
-
Best Budget
Samsung 990 PRO NVMe SSD
Unglamorous and load-bearing: model weights stored by hash and verified, mirrored, so the file that produced a result is provably the file you still have. Reproducibility fails quietly if the weights drift.
Where the nondeterminism actually comes from
Ranked by how much trouble each causes, because the fixes have very different costs.
1. Sampling
The obvious one, and the only one most people address. Greedy decoding or a fixed seed removes it.
Necessary and nowhere near sufficient, which is why people believe they have solved the problem and then discover they have not.
2. Reduction order in GPU kernels
The mechanism described above. Frameworks expose switches for it: PyTorch has torch.use_deterministic_algorithms(True), cuBLAS needs CUBLAS_WORKSPACE_CONFIG set, and cuDNN needs its autotuner turned off.
Two consequences worth knowing before you commit:
- Not every operation has a deterministic implementation. PyTorch will raise an error rather than silently give you a nondeterministic one, which is the correct behaviour and occasionally a wall.
- Deterministic kernels are slower, sometimes substantially. You are buying repeatability with throughput.
3. Autotuning
cuDNN benchmark mode times several algorithms on first use and keeps the fastest. Which one wins depends on what else the machine was doing at that moment, so the same code can select different kernels on different runs. Turn it off; accept the first-run cost.
4. Reduced-precision defaults
On Ampere and later, TF32 can be used for matrix multiplication in place of full FP32 — faster, and fewer mantissa bits. Whether it is on by default has varied across framework versions, so check rather than assume, and pin the setting explicitly if the result must hold.
5. Batch composition — the one that catches serving
This is the subtle one, and it is why this architecture and the multi-user inference server cannot be the same machine.
In a batched server, your request is computed alongside whatever else happened to arrive at the same moment. Batch size changes tensor shapes, which changes kernel selection and reduction order, which changes the last bits. Your output can therefore depend on what a stranger asked for at the same time.
Nothing is broken. It is the direct consequence of the batching that makes serving efficient — and it means:
You cannot have continuous batching and reproducibility on the same machine at the same time. Batch size one, or accept that outputs are not reproducible.
That is a genuine, expensive trade. The multi-user server exists to raise throughput 10–20× by batching; this machine gives all of that up on purpose.
6. Hardware
Different GPU models produce different results — different core counts mean different reduction trees, and different architectures mean different kernels. A result reproducible on one card is not guaranteed on another, even from the same vendor and generation.
Silent memory errors are the last source, and the rarest. A flipped bit changes an answer with no warning at all, which is precisely the failure this architecture cannot tolerate.
What that means for the hardware
Four consequences, and they are unlike every other build on this site.
Pin the exact SKU, not the model class. “An RTX PRO 6000” is not a specification here; a specific card is. If reproducibility must survive a hardware failure, the replacement has to be the same product, which means buying the spare now rather than sourcing one in three years when it is discontinued.
Do not mix GPUs. A second card of a different model in the same machine is a second set of results. Even matched cards are safer used one workload at a time.
Specify ECC. The fine-tuning workstation argues for it on the grounds that a flipped bit ruins a twelve-hour run. Here the argument is stronger: a flipped bit produces an answer you cannot reproduce and cannot explain, in a context where being unable to explain it is the whole problem.
Prefer boring, available, long-lived parts. As with the air-gapped machine, the exotic component that benchmarked slightly better is a poor trade against one you can still buy a replacement for.
The component specification
| Component | Specification | Why |
|---|---|---|
| Accelerator | One card, exact SKU recorded, ECC | The result depends on it |
| Spare accelerator | Identical SKU, bought at the same time | Cannot be sourced later |
| CPU and memory | ECC where the platform supports it | Silent errors are unexplainable answers |
| Storage | Mirrored, holding model files by hash | Weights must not drift |
| Software | Container image pinned by digest | Version numbers are labels; digests are artefacts |
| Serving stack | Batch size one, no continuous batching | See above |
| Configuration | Determinism flags set explicitly, in version control | Defaults change between releases |
| Records | Manifest of every version and hash, per result | The deliverable |
Read that as a specification rather than a shopping list. The requirement — one pinned hardware configuration, one pinned software stack, no batching, and a record tying results to both — holds; a named product is one example.
What you give up
Stated plainly, because this architecture is expensive and the costs are the reason to check you actually need it:
- Throughput, twice over: deterministic kernels are slower, and batch size one gives up the largest efficiency win available in serving.
- Free upgrades. A driver update, a framework release or a new card can change results. Every update becomes a revalidation.
- The newest tooling, often, since determinism support lags features.
- Flexibility. The pinned configuration is the product. Changing it means re-establishing that the results still hold.
Before you commit: verification checklist
Almost all of this is empirical, and the first item is the one that decides whether the rest is worth doing.
- Establish that you need reproducibility rather than auditability. Ask who will ask, and what they will ask for. If the answer is “a record of what we ran”, stop here and build a manifest instead.
- Run the same prompt one hundred times with sampling disabled and compare outputs byte for byte. This is the baseline measurement, it takes minutes, and it tells you how bad the problem is for your specific stack.
- Enable the determinism settings and repeat. Record which operations refuse to run deterministically.
- Measure the throughput cost of the deterministic path against the default, so the trade is a number rather than a feeling.
- Verify batch size one is enforced, not merely typical under light load.
- Test on the spare card and confirm the results match the primary. If they do not, you have discovered that early rather than during an audit.
- Pin everything by digest — container image, model files, driver version — and record the pins with the results, not separately.
- Re-verify after any change, and write down that this is the policy before the first update tempts you.
When not to build this
- When you need auditability. Covered at the top, and it is the common case. A manifest is a fraction of the cost and answers the question most people are actually being asked.
- When you serve more than one person at a time. Batching and reproducibility are mutually exclusive; the multi-user inference server is the right machine and it is not this one.
- When the model is not the thing under scrutiny. If the output is reviewed by a person before it matters, you need a record of the review rather than a rerunnable computation.
- When you cannot commit to freezing the stack. An architecture whose defining property is that nothing changes is not compatible with wanting the newest model each quarter. Choose one.
Frequently asked questions
Why does the same prompt give different answers at temperature zero?
Because temperature zero removes sampling randomness, not numerical randomness. Floating-point addition is not associative, and a GPU sums thousands of partial results in an order that is not fixed between runs, so the last bits of a score can differ. Greedy decoding then picks the highest-scoring token — and when two candidates are nearly tied, a last-bit difference decides the winner and the continuations diverge from there.
Is this a bug in Ollama, vLLM or PyTorch?
No. It is a consequence of how floating-point arithmetic and parallel reduction interact, and it appears in every framework because it is below all of them. The frameworks do offer switches — deterministic algorithm modes, cuBLAS workspace configuration, disabling cuDNN autotuning — but those are opt-in because they cost performance, and some operations have no deterministic implementation at all.
Do I need this, or do I just need good records?
Almost certainly records. Ask who will ask and what they will ask for. If the answer is “what did you run” — model, version, parameters, prompt, output — a manifest and disciplined logging answers it for a tiny fraction of the cost. Only build this if somebody will genuinely recompute the result and expect the same thing, which is the case in regulated research, published science, and forensic or clinical work.
Can a batched server be reproducible?
No, and this is the trade that separates this architecture from the multi-user one. In a batched server your request is computed alongside whatever else arrived at the same moment, and batch size changes tensor shapes, kernel selection and reduction order. Your output can therefore depend on what a stranger asked at the same time. Batch size one, or accept that outputs are not reproducible — you cannot have both.
Will the same results run on a different GPU?
Not guaranteed, even from the same vendor and generation. Different core counts produce different reduction trees and different architectures select different kernels, so results can differ across cards. That is why this is the one architecture on the site that specifies an exact SKU rather than a class, and why the spare should be bought at the same time as the primary rather than sourced years later when it has been discontinued.
How much performance does determinism cost?
Enough to measure rather than estimate, which is why the checklist asks you to. Deterministic kernels are slower than their default counterparts, sometimes substantially, and giving up continuous batching forfeits the largest efficiency win in serving — the multi-user architecture exists because batching often raises aggregate throughput 10–20×. Run your own workload both ways and make the trade a number.
Does ECC memory actually matter for this?
More than for any other build here. A silent bit flip changes an answer with no warning and no explanation, which is precisely the failure this architecture cannot tolerate — you would be unable to reproduce the result and unable to say why. The fine-tuning workstation wants ECC because an error ruins a long run; here it wants ECC because an error produces an unexplainable answer in a context where explanation is the entire point.
What happens when I want to update the model or driver?
You revalidate. Any change to the model file, framework, driver or card can change results, so the pinned configuration is effectively the product and updating it means re-establishing that your results still hold. Decide that policy before the first tempting update rather than after — an architecture whose defining property is that nothing changes is not compatible with wanting the newest model each quarter.
Have you built this machine and verified determinism on it?
No. It is a reference architecture: components are specified from published documentation and are compatible on paper, and the mechanisms described here are properties of floating-point arithmetic and GPU execution rather than measurements taken on this hardware. How bad the problem is for your specific model, framework and card is empirical, which is why the checklist opens by asking you to run the same prompt a hundred times and count the differences.