The Colocated AI Machine: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has racked this machine or recovered one at three in the morning from another country.

The six architectures before it are all machines you can walk over to. Some of them lean on that without saying so: the air-gapped machine suggests a boot passphrase, and the always-on server tells you to test that it POSTs headless before it goes in the cupboard — advice that assumes the cupboard is down the corridor.

This one is specified around the constraint that removes that assumption: nobody will ever physically touch it. Not you, not today, not in an emergency. Everything below is downstream of that.

The machine may be in a colocation facility, another office, or a relative’s basement. The distance does not matter. What matters is that when it stops responding, your hands are not an option.

The recovery ladder

The single most useful way to specify this machine is to write down what you can still do at each stage of a bad day, and then buy your way as far down the ladder as the budget allows.

The specification question is which rung you fall off. A machine with SSH and nothing else falls off at the second row, which means a bad kernel update is a week of downtime and a courier. That is the failure this architecture exists to prevent, and it costs surprisingly little to prevent.

Out-of-band management is not an accessory

A BMC — baseboard management controller, exposed as IPMI or Redfish — is a small independent computer on the motherboard with its own network port, its own firmware, and its own power domain. It works when the main system is off, hung, or has no operating system at all. Server boards from Supermicro and ASRock Rack have one; consumer motherboards do not, and that single omission disqualifies them here more decisively than any performance consideration.

If the machine is built on a consumer platform, an IP-KVM supplies the same capability externally: HDMI capture plus USB keyboard and mouse emulation over the network, working from power-on. The KVM guide covers the products; the point here is that it is not optional equipment for this machine, and that it needs an ATX header cable to power-cycle rather than only to watch.

The management interface is a computer, and it is a liability

This deserves stating plainly rather than in a footnote, because it is the most commonly mishandled part of the whole architecture.

A BMC runs its own firmware, is updated rarely, has a long public history of serious vulnerabilities, and is reachable when the machine it manages is switched off. An exposed IPMI interface is an attacker’s ideal target: full remote control, below the operating system, invisible to anything running above it.

• Never place a management interface on a public address. Not behind a strong password, not on a non-standard port.
• Put it on a dedicated management VLAN reachable only over a VPN.
• Change the default credentials before the machine leaves your desk, because some vendors still ship with well-known ones.
• Update its firmware on the same cadence as everything else, and accept that this occasionally means a reboot.

The convenience that makes out-of-band management worth buying is exactly what makes exposing it dangerous.

Full-disk encryption, decided the other way

The air-gapped machine frames the boot key as a trade-off between a passphrase — strongest protection, no unattended restart — and a TPM-sealed key, which boots on its own and weakens exactly the protection you encrypted for.

In a colocation facility that trade-off resolves differently, because a passphrase prompt is unreachable. A machine that reboots at 4 a.m. and sits waiting for someone to type at a keyboard nobody is standing at is simply down.

Two workable answers, and the second is better:

• Remote unlock over SSH. A minimal SSH daemon in the initramfs accepts the passphrase over the network at boot. You keep the passphrase; you need to be awake.
• Network-bound unlock. The disk key is escrowed with a key server on your own network — Clevis and Tang is the common implementation — and the machine unlocks automatically only if it can reach that server. A drive pulled from the rack and taken elsewhere stays encrypted, because the network it needs is not there.

The second gives you unattended reboots and keeps the property you actually wanted, which is that hardware leaving the facility is useless. A TPM-sealed key does not, because the key travels with the machine.

The rack is a physical specification

Everything below is a constraint a tower in a study does not have.

Airflow is front-to-back, and it is not negotiable. Racks are built around cold and hot aisles. A tower cooler blowing sideways or an AIO radiator mounted for a desktop case fights that pattern and heats its neighbours. Specify a chassis designed for front-to-back flow and cards that suit it — blower-style GPUs exhaust out of the bracket, which is why they exist and why they are worth the noise here.

Noise does not matter at all, which is genuinely freeing. The 40 mm fans that would be intolerable under a desk are correct in a rack, and they solve thermal problems that quiet cooling cannot.

Depth and height are hard limits. Modern GPUs are long, and a 4U chassis that fits one on paper may not fit it behind a cable arm. Confirm the rack’s usable depth, not the chassis specification alone.

You are billed for space and for power, separately. A circuit allocation is a real ceiling: two 600 W cards plus the rest of the machine may exceed the amperage you are paying for, and the facility will not care that the rack had space. Establish the allocation before specifying the accelerators, and re-read the dual-GPU machine's point about continuous-load derating — it applies to a colo circuit exactly as it applies to a domestic one.

Watchdogs, because nobody is there to press the button

A hardware watchdog timer resets the machine if the operating system stops petting it. On a desk machine that is a curiosity. Here it converts a class of failure — a kernel hang, a driver deadlock after an update — from a support ticket into a thirty-second outage nobody notices.

Specify it, enable it, and test it deliberately by hanging the machine on purpose while you still have physical access. An untested watchdog is a belief, not a control.

Pair it with firmware set to power on after AC loss. The always-on server makes this point for a cupboard; in a facility where a PDU may be reseated without your involvement, it is the difference between a blip and an outage that lasts until someone reads your ticket.

The component specification

Read that as a specification rather than a shopping list. The requirements — out-of-band access, remote power control, airflow direction, unattended unlock — hold; a named product is one example that satisfies them.

Monitoring has to come from outside

A machine that monitors itself and reports its own health is useless at the moment it matters, because a machine that has stopped also stops reporting. Silence is the alert you will miss.

Specify an external check — a service or a small machine elsewhere — that alerts on absence rather than on a bad reading. Watch the things that predict a call-out rather than the things that look impressive on a dashboard: reachability, temperatures, disk health, and whether the model server is answering rather than merely running.

And monitor the management interface separately from the host. Discovering that the BMC died three weeks ago, at the moment you need it, is the specific bad day this architecture is meant to prevent.

When not to build this

• When you can walk to it. This is the common case and the whole justification collapses without it. Every rung of the recovery ladder is free when the machine is in the next room.
• When demand is bursty. Rented GPU-hours have no rack fee, no remote-hands contract and no shipping. That is the fine-tuning workstation's argument, and colocation strengthens it rather than weakening it — you are now paying monthly for idle hardware and for the space it idles in.
• When the data has residency conditions. A third-party facility is a location, and it is somebody else’s. If a contract names where processing may happen, that is the air-gapped machine's territory rather than this one.
• When you have not priced the whole thing. Rack unit, power allocation, cross-connect, remote hands, and the spares you keep on site. The hardware is frequently the smaller number, and the recurring cost never stops.

Before you rack it: verification checklist

Every check here must be done while the machine is still within arm’s reach, because that is the last moment any of them is cheap.

1. Reach the BMC or IP-KVM from a different network, over the VPN, and confirm it is not reachable from anywhere else. Test from a phone on mobile data.
2. Change every default credential on the management interface, and record where those credentials live.
3. Power-cycle the machine entirely from the management interface, including a hard reset from a hung state.
4. Mount virtual media remotely and boot from it. If you cannot reinstall the operating system without hands, you do not have out-of-band management, you have a remote screen.
5. Reboot with the disks encrypted and confirm it comes back unattended.
6. Hang the machine deliberately and confirm the watchdog resets it.
7. Pull the power and confirm it restores on its own.
8. Confirm the external monitor alerts on silence by switching the machine off and waiting for the alert to arrive.
9. Confirm airflow direction on every component, including the GPU.
10. Measure sustained draw at the wall and check it against your circuit allocation with headroom, not against the label.
11. Write the remote-hands runbook: which cable, which port, which LED, what to do and what never to touch. Somebody unfamiliar with your machine will read it under pressure.
12. Label everything physically. Ports, drives, cables. The person following the runbook cannot ask you which one you meant.