The Air-Gapped AI Machine: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or operated it under an audit.

The fine-tuning workstation names three reasons to own hardware rather than rent it, and one of them is that the data cannot leave the building. This is the machine for that case, specified around a constraint none of the other five share: nothing reaches the internet, and you must be able to prove what is on the machine.

Everything difficult here follows from that. The hardware is the easy part and takes up about a fifth of this page.

Most people need “no egress”, not an air gap

These are different properties with wildly different costs, and they are routinely confused.

No data egress means the machine cannot send your data out. It is achieved with a VLAN, a default-deny egress firewall, and a narrow audited allowlist for the handful of things that genuinely need to reach out. The machine can still be patched, still pull a model, still resolve a package. Most organisations that say “air-gapped” want this, and it is comparatively cheap.

An air gap means no network path exists at all. Everything that arrives, arrives on physical media that somebody carried. Updates become a project. This is genuinely expensive to operate, and the expense is ongoing rather than one-off.

Build the air-gapped version when a rule, a contract or a regulator actually requires physical isolation — not because it sounds more secure. If nobody can point at the clause, specify no-egress instead and spend the difference on a better machine.

The rest of this assumes the clause exists.

The supply chain now runs through you

On a connected machine, ollama pull and pip install are the supply chain, and somebody else operates it. Here, you are it — and that is the actual work.

Bringing in model weights

Three steps, and the order matters.

1. Obtain the weights on a connected staging machine and record the publisher’s checksums.
2. Carry the checksums separately from the weights — different medium, different trip, ideally a printed sheet or a signed manifest. A hash that travels alongside the file it validates proves very little, because whatever could alter one could alter the other.
3. Verify on the isolated side before loading anything, and record the result in a manifest you keep.

Prefer data-only formats, and understand why

This is the single most important technical point on the page, and it is not about performance.

PyTorch’s traditional .bin and .pt checkpoints are Python pickle files. Deserialising a pickle can execute arbitrary code by design — it is a feature of the format, not a flaw in any particular file. Loading an untrusted checkpoint is therefore equivalent to running an untrusted program, on a machine you have deliberately made hard to patch and hard to observe.

safetensors and GGUF are data-only container formats. They hold tensors and metadata, and there is no code path from parsing them to executing something. Recent PyTorch offers weights_only=True, which helps, but choosing the format is a stronger guarantee than remembering the flag.

Specify safetensors or GGUF, and treat a pickle-format checkpoint as something to convert on the staging machine rather than carry across. An air-gapped machine cannot be casually reinstalled, which makes it exactly the wrong place to take that risk.

Bringing in software

Two workable patterns, and the choice determines your operational life:

• A full container image, exported with docker save and carried as a tarball. Everything is pinned inside a digest you can record. This is the easier pattern to audit and the one to prefer.
• An offline package mirror — a local repository populated on the staging side. More flexible, considerably more work, and easier to let drift.

Either way, record digests, not version numbers. “v0.6.2” is a label somebody can move; a digest is the artefact.

Updates are the real cost of an air gap

A connected machine takes security updates automatically, which is what the always-on server specifies. This one cannot, and pretending otherwise is how air-gapped machines end up years behind on known vulnerabilities while everyone assumes isolation is doing the work.

Isolation reduces exposure. It does not remove the vulnerability, and the threat model that justified the air gap usually includes an insider or a carried-in file — precisely the paths isolation does not close.

So specify a process, not an intention:

• A named cadence. Monthly is defensible; quarterly is defensible with a written argument. “When we get to it” is how machines rot.
• A staging machine on the connected side that assembles and verifies each update bundle.
• A rehearsed rollback. Snapshot before applying, and confirm at least once that restoring actually works. An update that breaks the driver stack on a machine you cannot search the internet from is a genuinely bad afternoon.
• A written record of what was applied and when. This is the artefact an auditor asks for, and it is trivial to keep and painful to reconstruct.

Removable media is the attack surface you just created

Air-gapping does not remove risk; it relocates it onto whatever crosses the gap. The canonical example is Stuxnet, which reached isolated industrial systems on USB media, and the lesson generalises: the gap moves the attack surface to the courier.

Reasonable controls, in rough order of cost:

• Dedicated media that lives on one side. Sticks that shuttle between a general-purpose laptop and the isolated machine are the problem.
• Scan and verify on the connected side, before the media crosses, and verify checksums again after.
• Write-once media for anything inbound where the volume allows it.
• A data diode if the requirement genuinely justifies hardware-enforced one-way transfer. This is a real product category rather than a metaphor, and it is priced accordingly.

The point is not that any one of these is essential. It is that “we air-gapped it” is not a control, and the media procedure is where the actual security lives.

Licensing and telemetry will bite you

Anything that phones home to activate, validate a licence, or report usage will fail on this machine — sometimes loudly at install, sometimes quietly at renewal, months later, in the middle of something important.

• NVIDIA’s Linux drivers and CUDA install and run offline. No activation step, which is one fewer thing to worry about.
• Commercial software with a licence server is the hazard. Establish before purchase whether it supports a node-locked offline licence, and what happens when that licence expires with no way to reach the vendor.
• Telemetry is worth auditing rather than assuming. Some tools attempt update checks on startup. Offline they fail harmlessly; the reason to know anyway is that an unexplained outbound attempt in a log is exactly the sort of thing that costs a day of investigation during an audit.

Check this before the purchase order, not after the machine is inside the boundary.

The hardware is deliberately boring

Two consequences of isolation that change what “good hardware” means:

Buy components you can buy twice. The exotic part that was slightly faster is a poor trade against one you can source again in eighteen months. Keep a cold spare of anything whose failure stops work — a drive, a PSU, ideally the board.

Drives are consumables with a disposal procedure, not returnable parts. A drive that has held regulated data cannot be sent back for warranty replacement; it is destroyed. Price that in, specify a mirror so a single failure does not stop work, and write the destruction procedure down before you need it.

On full-disk encryption and unattended restart

Encrypt at rest — LUKS on Linux — because the threat model for an isolated machine genuinely includes somebody removing the drive, which is not a serious concern for a machine in a rack you already control.

There is a real trade-off here and it deserves stating rather than glossing:

• A passphrase at boot means the strongest protection and no unattended restart. After a power cut, the machine sits at a prompt until a person attends it.
• A TPM-sealed key allows automatic boot and weakens exactly the protection you encrypted for, because a stolen machine boots itself.

Neither is wrong. Choose deliberately based on whether the risk is a stolen drive or an unattended outage, and record which you chose and why. The colocated machine resolves the same trade-off the other way, because there a passphrase prompt is unreachable. The failure mode is choosing by default and discovering it during an incident.

Proving what is on the machine

An air-gapped machine that nobody can describe is not meaningfully more trustworthy than a connected one. The audit story is half the value, and it is cheap if you build it in and expensive if you retrofit it.

Keep a manifest: model files with checksums and provenance, container digests, package versions, driver version, and the date each arrived. Version it. This is the document that answers “what is running and where did it come from”, which is the question that actually gets asked.

Make provisioning reproducible — scripted, in version control, so the machine can be rebuilt from a known state rather than remembered. This is worth doing on any machine; here it is the difference between a rebuild being an afternoon and a rebuild being an archaeology project.

Plan for logs that stay local. Nothing ships them anywhere, so rotation and retention are decisions rather than defaults, and a full disk on a machine nobody can reach remotely is an avoidable outage.

What you are giving up

Stated plainly, because these are the frictions that erode discipline over time and produce the machine that is technically isolated and practically unmaintained:

• No pulling a model because a better one came out this morning. Every model is a scheduled transfer.
• No installing a package to try an idea. Every dependency is planned.
• No searching the internet from the machine. Documentation is a thing you carried in.
• No automatic security updates, which is the one that matters and the one people quietly stop doing.

Every one of those is survivable with a process and corrosive without one.

Before you deploy: verification checklist

Mostly procedural, because on this machine procedure is the architecture.

1. Identify the rule that requires physical isolation. If nobody can point at it, build the no-egress version instead. This check saves the most money and is the one most often skipped.
2. Confirm every piece of software runs without network access, including at first launch and at licence renewal. Test it disconnected on the staging side before the purchase order.
3. Verify checksums on the isolated side and record them in the manifest. Confirm the checksums travelled separately from the files.
4. Confirm every model is safetensors or GGUF, and that no pickle-format checkpoint is loaded on the isolated machine.
5. Write the update cadence down, with a named owner and a date for the first one.
6. Rehearse a rollback from snapshot at least once, before you need it.
7. Decide the encryption trade-off — passphrase or TPM — and record the reasoning.
8. Write the media procedure, including what happens to a stick that has touched both sides. It has become an isolated-side stick permanently.
9. Write the drive destruction procedure, and confirm it satisfies whichever standard applies to you.
10. Confirm the cold spares exist and are the same revision as the parts in service.

When not to build this

• When no rule requires it. Covered at the top, and it is the common case. Egress control gets most of the benefit for a fraction of the operational cost.
• When nobody owns the update process. An unpatched isolated machine is a worse outcome than a patched connected one, and the honest question at specification time is who is doing the monthly bundle in a year.
• When the workload is bursty and the data is not actually sensitive. That is the fine-tuning workstation's rent-instead argument, and it applies here too.
• When it needs to serve a team and nobody has scoped that. Concurrency is its own constraint — the multi-user server covers it, and air-gapping does not remove the need to do that arithmetic.