The Always-On AI Server: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or run it for a year.

The other two architectures on this site describe machines that sit under a desk and work when you do. The $2,000 machine optimises for capability per dollar; the dual-GPU 70B machine optimises for capability per anything.

This one is different in a way that inverts several of their recommendations, because it is specified around a fact neither of them has to think about: it is switched on when you are asleep.

The number that decides this build

A machine that runs continuously spends the overwhelming majority of its life doing nothing. Peak performance is what you notice; idle draw is what you pay for.

Multiply the right-hand column by your own electricity tariff — we do not publish prices, and household rates vary by more than a factor of three between countries.

Two things follow, and both of them push against the advice in the other two architectures.

Heat is continuous, not bursty. A machine drawing 100 W puts 100 W of heat into wherever it lives, permanently. In a cupboard or a small room, that is noticeable within an hour and cumulative over a summer.

Peak throughput is worth less than it looks. If the machine answers a few dozen queries a day, the difference between 20 and 40 tokens per second costs you seconds of waiting and saves you nothing else. The difference between 30 W and 100 W idle costs you 613 kWh a year, every year.

That is the whole thesis. Buy the machine that is cheap to leave on, not the machine that is fast when used.

The decision unique to always-on machines

Here is a question the desk machines never have to answer: do you keep the model loaded?

Ollama unloads a model from VRAM five minutes after the last request by default (OLLAMA_KEEP_ALIVE=5m). Set it to -1 and the model stays resident indefinitely; set it to 0 and it unloads the moment a request finishes.

Both settings cost something, and which one is right depends on hardware you have not bought yet.

Keeping it resident means the first token arrives immediately, at any hour, without warning the machine. It also means the memory holding it stays clocked up. On a discrete card, allocated VRAM keeps the memory subsystem awake — a card that idles near 20 W with nothing loaded can sit at 100 W or more with weights resident and no work being done. That is the single largest controllable number in this architecture.

Letting it unload drops idle draw to the floor and pays for it in cold-start latency, which is a storage problem:

So the keep-alive setting, the storage layout and the network specification are one decision, not three. A machine with local NVMe can unload aggressively and still feel instant. A machine loading from a NAS over 2.5 GbE cannot, and must either keep models resident or accept that every conversation starts with a wait.

This architecture specifies local NVMe for the active models and aggressive unloading, because that combination gives the lowest idle draw with a cold start most people will not notice.

Which is why unified memory wins here

On the desk machines, a discrete NVIDIA card is the obvious answer and the only real question is which one. Here it is genuinely arguable, and the argument mostly goes the other way.

A Strix Halo machine allocates up to 96 GB of its 128 GB pool to the GPU and reads it at 256 GB/s. That bandwidth is a quarter of a 3090’s, so it generates tokens perhaps three to four times more slowly — and it does so while drawing about 120 W at full tilt, idling in single-digit-to-low-double-digit watts, and making almost no noise.

For a desk machine that trade is poor. For a machine answering occasional queries at three in the morning, it is close to ideal: it holds a 70B model that the dual-GPU build needs 700 W of graphics cards to reach, in a 4.5-litre box you can put in a cupboard.

The honest limits, because they are real:

• No CUDA. ROCm and Vulkan back-ends work well for llama.cpp-family inference and less well for everything else. Image generation and fine-tuning are markedly worse served.
• 256 GB/s is not fast. It is enough to make 70B usable and not enough to make it quick. Read the VRAM explainer on why bandwidth divided by model size predicts your token rate.
• Soldered memory. What you buy is what you have, forever.

Specify a discrete card instead if the server also does image generation, or if you need CUDA for anything. In that case a 16 GB card at 180 W is the sensible ceiling — and configure it with nvidia-smi -pl rather than accepting stock limits.

The component specification

Read that as a specification rather than a shopping list. The requirements — capacity, bandwidth, idle draw, network topology — hold; a named model is one example that satisfies them.

Networking: buy it for the weights, not the tokens

This is where homelab instinct misleads people, so it is worth doing the arithmetic.

A model generating 30 tokens per second emits roughly 120 bytes per second of actual content. Wrap it in server-sent events, HTTP framing and TLS and you are still comfortably under 10 KB/s. A 1 GbE link is roughly ten thousand times more capacity than a conversation needs.

Nothing about interactive inference is bandwidth-limited. If your chat feels slow, the bottleneck is the GPU’s memory bandwidth, not your switch.

What genuinely wants a fast network:

• Pulling models. A 42 GB download saturates whatever you have.
• Loading models from a NAS, per the table above — which is the case where the network becomes a latency problem you feel at the start of every conversation.
• Datasets and backups, which is the ordinary homelab reason.

So the guidance in 2.5GbE versus 10GbE applies unchanged, with one addition specific to this machine: if the model library lives on a NAS, the network moves onto the critical path and 10 GbE stops being a luxury. If models live on local NVMe, 2.5 GbE is ample and 1 GbE is survivable.

Running it headless

A machine with no monitor needs three things confirmed, all in firmware, and all easier to check before the machine is in a cupboard.

Restore on AC power loss. Default on most boards is to stay off after an outage. Find the setting — it is usually under a power-management heading — and set it to power on. Otherwise the first brief outage silently ends your uptime until you next walk past.

POST without a display. Some boards halt on boot with no display attached. Most modern ones do not. This is worth testing deliberately rather than discovering remotely.

Wake-on-LAN, if you intend to suspend rather than run continuously. Which, incidentally, is worth considering: a machine that suspends to RAM at 3 W and wakes on a packet has different economics from one that idles at 30 W, at the cost of a few seconds of wake latency.

Headless has one straightforward benefit on a discrete-GPU build: with no desktop session, the compositor is not holding several hundred megabytes of VRAM, and all of it is available to the model.

Exposure: the section that matters most

Ollama has no authentication. None. It binds to 127.0.0.1:11434 by default, and that default is the only thing protecting it.

Setting OLLAMA_HOST=0.0.0.0 to reach the server from your laptop exposes an unauthenticated API to every device on the network — one that will accept any prompt, pull down new models, and delete existing ones on request. Forward that port through your router and you have published a machine that anyone who scans for it can use as their own inference server, at your electricity cost, with no log of who did what. Internet-wide scans routinely turn up thousands of instances in exactly that state.

Three ways to do this correctly, in order of preference:

1. A WireGuard or Tailscale tunnel. The server keeps listening on localhost or a private interface; your other devices join the same private network. This is the right answer for almost everyone and takes about fifteen minutes.
2. A reverse proxy that terminates TLS and requires authentication in front of the API, bound to the LAN only. More moving parts, useful if several people share the machine.
3. Bind to the LAN interface with a firewall rule restricting source addresses. Workable, and the easiest of the three to get subtly wrong.

Never option four, which is a forwarded port and a hope.

Two more things follow from a machine being on continuously rather than intermittently: it is a persistent target, so key-based SSH only, no password authentication, and unattended security updates enabled. A desktop that is off two-thirds of the day forgives a lot of this. A server does not.

What the finished machine runs

For the specified Strix Halo configuration, at Q4_K_M:

The capacity is genuinely unusual for the power envelope — 96 GB addressable by the GPU is double what two 3090s provide. What you are trading away is the speed at which it can be read, and on a machine used a few dozen times a day that is a trade many people should take.

Before you deploy: verification checklist

The other two architectures’ checklists are about parts fitting together. This one is mostly about behaviour over time, and almost all of it can be verified in the first hour.

1. Measure idle draw at the wall, with a plug meter, in the state you intend to leave the machine. This is the one number that decides the economics of the whole build, it takes five minutes, and no specification sheet will tell it to you.
2. Measure it again with a model resident, then decide the keep-alive setting from the difference rather than from a guess.
3. Time a cold start from your actual storage layout. If it is longer than you will tolerate at 8 a.m., either move models to local NVMe or keep them loaded.
4. Confirm restore-on-AC-power-loss in firmware, then verify it by pulling the plug.
5. Confirm it POSTs headless, before it goes in the cupboard.
6. Check ambient temperature where it will actually live, after it has been running an hour with the door shut. Cupboards recirculate.
7. Verify the API is not reachable from outside your network. From a phone on mobile data, off Wi-Fi, try to reach it. Nothing should answer.
8. Confirm unattended upgrades are enabled and SSH accepts keys only.
9. Confirm the model library is backed up. Re-downloading 400 GB of weights is a slow afternoon.

When not to build this

• If it is your only machine. This architecture deliberately sacrifices peak speed for idle draw. As a desk machine it will feel sluggish, and the $2,000 build is a better use of the same money.
• If you need CUDA. Image generation, fine-tuning and most research tooling assume it. The unified-memory route is weakest exactly where the ecosystem is strongest.
• If it will serve several people at once. Concurrency wants batching, batching wants compute and bandwidth, and this machine is specified to have neither in quantity — the multi-user inference server is specified for that instead.
• If you already have the dual-GPU machine. Running it headless with a hard power limit is a worse always-on server than this one and a much better everything else, and one machine that does both is usually the right answer for a household.