The Multi-User Inference Server: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or run a team’s workload through it.

The always-on server ends by saying that a machine serving several people at once wants batching, that batching wants compute and bandwidth it does not have, and that this is therefore a different machine. This is that machine.

It is also the point where the arithmetic this site uses everywhere else stops being the arithmetic that matters. On a serving machine, the model is not what fills the memory.

Concurrency is not a tax. It is the point.

Start with the fact that makes serving different, because almost everything else follows from it.

When one person generates one token, the GPU reads every active weight in the model to produce it. That is one arithmetic operation per weight read — appallingly low arithmetic intensity, and the reason single-user generation is memory-bandwidth-bound while the tensor cores sit almost idle.

Now serve thirty-two people at once with continuous batching. The weights are read once and used to produce a token for every sequence in the batch simultaneously. The expensive part — the read — is amortised across all of them.

Adding users is close to free until it is not. That is the opposite of the intuition most people bring, and it is why a serving machine is a genuinely good use of hardware in a way that a single-user machine is not — a desk machine wastes most of the silicon it paid for.

The corollary matters just as much: you cannot get this from Ollama. It is built for single-user convenience and does not do continuous batching in any serious way. Serving software means vLLM, TensorRT-LLM, SGLang or equivalent. That is a specification decision, not a preference, and it is made before any hardware is bought.

What actually fills the memory

Here is where the site’s usual model-size arithmetic stops being the binding constraint.

Every sequence in flight holds a KV cache: the keys and values for every token it has seen, at every layer, kept so that generating the next token does not require reprocessing the whole conversation. Its size per token is:

2 × layers × KV heads × head dimension × bytes per element

The weights are a fixed cost paid once. The cache scales with concurrency multiplied by context length, and on a busy server it dwarfs the model.

Those figures assume grouped-query attention, which every current model of consequence uses. It is worth knowing why: older multi-head attention gave every attention head its own K and V, so a model with 64 heads rather than 8 KV heads would produce a cache eight times larger. GQA is a large part of why serving these models is tractable at all.

The concurrency arithmetic

Take 96 GB of VRAM and work out what it holds. Weights first, then divide what remains by the per-user cache:

These are arithmetic ceilings, not measurements. Real capacity lands below them because activations, fragmentation, CUDA graph buffers and scheduler headroom all take a share. Treat them as the number to design against and expect to reach perhaps 70–80% of it.

Two levers move these numbers materially:

• KV cache quantisation to fp8 roughly halves the cache and therefore roughly doubles the users, at a quality cost most deployments find acceptable. Test it against your own evaluations rather than taking anyone’s word.
• Prefix caching computes a shared system prompt once and reuses it across every request that starts with it. For a team where everyone hits the same assistant with the same two thousand tokens of instructions, this is a large and easily overlooked win.

This is the whole specification question, and it is worth stating as a formula: how many concurrent users, at what context length, at what acceptable per-token latency. Answer that first. The hardware follows from it, and no amount of hardware fixes having never asked it.

Prefill and decode are two different machines

A request has two phases with opposite characteristics, and confusing them is the most common reason a server that benchmarks well feels bad to use.

Prefill processes the whole prompt at once. Every token is available, so the work parallelises across all of them — it is compute-bound, and it is what determines time to first token.

Decode produces one token at a time, each depending on the last. It is memory-bandwidth-bound, and it determines how fast the reply streams.

The scheduling problem is that a naive server runs them in the same queue. Someone pastes a twenty-thousand-token document; the server spends several seconds on that prefill; every other user’s stream stalls for the duration. That is the “why does it stutter when a colleague uploads something” complaint, and it is a scheduler symptom rather than a hardware one.

Chunked prefill is the fix: break long prefills into pieces and interleave them with the decode steps of everyone else. Slightly slower for the person who pasted the document, dramatically better for everyone else. Specify a server that does it, and turn it on.

Specify against latency targets, not a single number

Three metrics matter, they trade against each other, and you cannot maximise all three:

• TTFT — time to first token. Dominated by prefill, so by compute and queue depth. This is what feels like responsiveness.
• TPOT — time per output token. Dominated by bandwidth and by how large the batch has grown. This is what feels like reading speed.
• Throughput — total tokens per second across everyone. This is what you are actually buying capacity for.

Raising the batch size raises throughput and worsens TPOT. There is a point past which the server is doing more work in total while every individual user feels it getting slower, and a server tuned purely for throughput lands there.

Pick targets in advance — something like TTFT under a second at the 95th percentile, TPOT under 50 ms — and tune the maximum batch size until you meet them at your expected concurrency. Measure percentiles, not averages. The mean is dominated by the easy requests, and the complaints come from the tail.

The component specification

Read that as a specification rather than a shopping list. The requirements — VRAM capacity against your concurrency arithmetic, a serving stack that batches, an auth layer — hold; a named model is one example that satisfies them.

On the network, again

The always-on server makes the point that a conversation streaming at 30 tokens per second moves well under 10 KB/s. Multiply that by fifty users and it is still under half a megabyte per second.

Serving many people does not make the network the bottleneck. 2.5 GbE is generous. Buy fast networking here for the same reasons as any other server — moving weights, backups, shared storage — and not because the inference traffic needs it.

On paged attention

Naively, each sequence’s KV cache is a contiguous block sized for the longest reply it might produce. Most replies are shorter, so most of that reservation is wasted, and the waste is what limits how many sequences fit.

PagedAttention allocates the cache in small fixed-size blocks with an indirection table, exactly like virtual memory pages. Sequences grow a block at a time, fragmentation largely disappears, and a much larger share of the cache budget does useful work. It is the reason the concurrency figures above are reachable at all, and it is a property of the serving stack rather than the hardware — another reason the software decision comes first.

The licence question, stated plainly

NVIDIA’s GeForce driver licence has historically included terms restricting datacenter deployment, and the professional and datacenter cards are the products sold for that use.

For a machine in an office serving a handful of colleagues, whether that constitutes datacenter deployment is genuinely unclear, and we are not in a position to tell you it does not. What we can say is the practical shape of the risk: for anything commercial, at any scale you would be unhappy to explain, the professional cards are the defensible choice, and part of what their price buys is not having this conversation.

If that matters to your situation, read the current licence rather than a summary of it — including this one — and take advice. We are not lawyers and this is not advice.

Operating a machine other people depend on

Every architecture before this one serves its owner. This one serves colleagues, which changes what “working” means.

Authentication is not optional, and here you have a real option. Unlike Ollama, the mainstream serving stacks expose an OpenAI-compatible API that supports API keys. Use them, and put an authenticating gateway in front regardless, so that keys can be issued and revoked per person without restarting the server.

Rate limit per user. One person’s runaway script should degrade their own experience, not everyone’s. This is the multi-user equivalent of a fuse.

Watch queue depth and TTFT percentiles, not GPU utilisation. A saturated server looks busy either way; queue depth tells you whether people are waiting, and the 95th percentile tells you whether the complaints are justified. High utilisation with a short queue is exactly what you want and looks identical, on a utilisation graph, to being overloaded.

Decide what happens when it is full, before it is full. Queue, shed load with a clear error, or degrade to a smaller model — all three are defensible, and defaulting into an unbounded queue means requests time out somewhere the user cannot see.

Plan for restarts. Loading a 42 GB model takes tens of seconds even from fast local storage, and during that window the service is down for everyone. Model swaps are not free and should not be casual.

When not to build this

• Fewer than about four concurrent users. The batching advantage barely engages, and the always-on server does the job for a fraction of the money and power.
• Bursty demand. If real usage is a spike on Monday mornings, capacity sized for the spike idles all week. That is the argument the fine-tuning workstation makes about renting, and it applies here with equal force.
• When the users do not trust each other. Everything here assumes colleagues, and that assumption carries more weight than it looks — batching and prefix caching both put several parties’ data in one place. The multi-tenant machine is specified for mutually untrusted tenants instead.
• When outputs must be reproducible. Batching means a request is computed alongside whatever else arrived at the same moment, which changes kernel selection and reduction order — so the same prompt can return different text. The reproducible machine gives up batching entirely for that reason.
• A product with real service-level commitments. Everything here is a single machine with a single accelerator. No redundancy, no failover, no second site. If people outside your organisation depend on it, this architecture is a prototype rather than a deployment.
• You mainly want the biggest model you can run. Concurrency and model size compete for the same memory. Serving 70B to twenty people needs the same card as fine-tuning it, and the dual-GPU machine reaches larger models for one person more cheaply.

Before you deploy: verification checklist

Mostly measurement, like the always-on server, but the things being measured are behaviours under load rather than draw at idle.

1. Do the concurrency arithmetic for your actual model, context length and user count. If it exceeds the card you were about to buy, nothing else on this list matters.
2. Confirm the serving stack supports your model architecture, at the quantisation you intend, in the version you will actually run. Support arrives at different times for different models.
3. Set explicit TTFT and TPOT targets before benchmarking, so the numbers have something to fail against.
4. Load-test at your expected concurrency and at twice it, and record percentiles rather than averages.
5. Test the long-prompt case deliberately — someone pasting 20k tokens while others are mid-stream. If everyone stalls, chunked prefill is off.
6. Verify per-user authentication and rate limiting by exhausting one key and confirming the others are unaffected.
7. Measure a cold start, and decide whether that outage window is acceptable during a working day.
8. Confirm KV cache quantisation quality against your own evaluation set before enabling it, not after.
9. Confirm the API is not reachable from outside your network, from a device on mobile data. The always-on server makes this point at length and it applies with more force to a machine holding other people’s conversations.