The Multi-Tenant GPU Machine: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or attempted to break isolation on it.

The multi-user inference server serves several people from one machine and assumes they are colleagues. That assumption does most of the work in it, quietly. Remove it and almost every recommendation needs re-examining.

So this architecture is specified around a constraint none of the other ten share: the people sharing this machine must be isolated from each other.

Different departments with confidentiality obligations. A university group where students should not read each other’s research. A consultancy serving clients whose data must not mingle. The adversary here is not outside the building — it is a legitimate user of the same box, which is exactly the case the other architectures do not cover.

Concurrency and isolation are different problems

Worth separating cleanly, because they are routinely conflated and they fail in opposite directions.

That last row is the one with a purchase order attached, so it comes first.

Consumer cards cannot do this

The sharpest hardware constraint on this site, and it is worth stating before anything else: GeForce cards cannot partition. There is no hardware isolation mechanism on them at all. If several mutually untrusted tenants must share one card, a consumer card is not a cheaper option — it is not an option.

What the alternatives actually give you:

MIG — Multi-Instance GPU partitions the hardware itself. Each instance gets its own memory, its own paths through cache, and its own slice of compute. A fault in one instance does not take down the others, and one tenant exhausting its memory does not exhaust anyone else’s. This is the only mechanism on the list that provides genuine hardware isolation, and it is available on datacenter parts and on the professional line — the RTX PRO 6000 Blackwell lists MIG partitioning among its features, which is the specific reason it appears in this architecture rather than a faster consumer card.

Time-slicing is the default when several processes share a GPU. The scheduler switches between contexts. It shares one memory space, provides no fault isolation, and a single tenant’s out-of-memory condition can take down work that has nothing to do with them. Fine for cooperating processes; not an isolation boundary.

MPS — Multi-Process Service lets kernels from several processes run concurrently rather than in turn, which improves throughput considerably. It also shares a memory space and provides no isolation. It is a performance feature that is frequently mistaken for a security one.

vGPU virtualises the card for separate virtual machines and does provide strong separation — at the cost of a software licence separate from the hardware, which is a recurring expense people discover after the purchase order rather than before.

Separate cards, one per tenant is the crude answer and often the correct one. See the last section.

Containers do not isolate GPUs the way you think

This is the assumption most likely to be wrong in an existing deployment.

Containers isolate CPU, memory and filesystems well. A container with GPU access has comparatively direct access to the hardware, and the boundary is far weaker than the one around the rest of the container. Two containers sharing a card without MIG are sharing hardware with no meaningful partition between them, regardless of how carefully the rest of the stack is separated.

Two consequences worth designing around:

• Do not treat “each tenant gets a container” as isolation when those containers share an unpartitioned GPU. It is isolation of everything except the part holding the data.
• GPU memory is not guaranteed to be scrubbed between one process releasing it and another acquiring it. Reading residue from a previous tenant’s allocation has been demonstrated in research and is not something to assume away. Where the requirement is serious, partition the hardware rather than relying on allocator behaviour.

Prefix caching becomes a leak

The clearest inversion between this machine and its sibling, and a good illustration of how much the trust assumption was carrying.

The multi-user server recommends prefix caching without reservation: compute a shared system prompt once, reuse it for every request that begins with it, and take a large and easily overlooked win. Among colleagues sharing one assistant, that is straightforwardly correct.

Across a trust boundary it is a data leak, in two distinct ways:

• Direct, if cached content can be served to a tenant who did not supply it. This is a configuration failure, and it is the obvious one.
• By timing, which is subtler and survives correct configuration. A cache hit returns faster than a miss. A tenant who can measure response times can therefore test whether a particular prefix has been seen before — which is a way of asking whether somebody else submitted it.

So: partition the cache per tenant, or disable it across boundaries. You are giving up a real efficiency gain, and that is the cost of the constraint rather than an oversight.

The same logic applies to batching. Batching within one tenant’s requests is fine. Batching across tenants puts several parties’ data in one kernel launch, which is a boundary you have to be able to defend.

Quotas are the other half of the job

Isolation is not only about reading. A tenant who consumes everything has denied the service to everyone else, and on shared hardware that is a routine accident rather than an attack.

Specify limits on memory, on request rate, and on the number of concurrent requests, and enforce them per tenant rather than globally. A global cap is satisfied by one tenant using all of it.

MIG helps here in a way software quotas cannot, because the memory limit is a property of the partition rather than a rule somebody’s code has to honour. That is the difference between a tenant being unable to exceed their share and being asked not to.

The component specification

Read that as a specification rather than a shopping list. The requirement — hardware partitioning, no shared caches, per-tenant quotas and per-tenant logs — holds; a named product is one example.

Sizing the partitions before you buy

The awkward part of MIG is that the partition sizes are not arbitrary, and the memory each tenant gets is a fraction of the card rather than all of it. A card holding a 70B model comfortably for one user may hold nothing useful once divided four ways.

So the arithmetic runs backwards from the tenants: decide the largest model any single tenant needs, add their KV cache budget using the multi-user server's method, and multiply by the number of tenants who must be served simultaneously. That total, not the model size, is the card you need.

This is where many multi-tenant plans fail — sized for the model, then divided, and suddenly nobody can run anything.

When not to build this

• When the users trust each other. This is the common case, and the multi-user server is faster, simpler and cheaper. Colleagues sharing an assistant do not need hardware partitioning, and buying it is money spent on a threat that does not exist.
• When you have few tenants. Two or three separate machines, one per tenant, provide isolation no partitioning scheme can match, cost less than one card capable of MIG, and fail independently. Partitioning wins on density, and density only matters past a certain number of tenants.
• When tenants need the whole card. If each one runs a 70B model, they cannot share. The arithmetic above will tell you this before you buy, which is the only good time to find out.
• When the requirement is really about the outside world. If the concern is data leaving the building rather than crossing between users, the air-gapped machine addresses a different threat model and this one does not help.

Before you deploy: verification checklist

1. Establish that tenants are genuinely untrusted, and write down what they must not learn about each other. Colleagues who share a manager are usually the other architecture.
2. Confirm the card supports MIG in the specific SKU, not the product line, before ordering.
3. Do the partition arithmetic: largest per-tenant model, plus KV cache, times simultaneous tenants. Confirm the result fits before purchase.
4. Verify partitions are enforced by trying to exceed a memory limit from inside one and confirming the others are unaffected.
5. Confirm no shared prefix cache across tenants, and test it — submit a distinctive prefix as one tenant and confirm response times as another are unchanged.
6. Confirm batching does not cross tenants. Read the serving configuration rather than assuming the default.
7. Check logs and metrics for cross-tenant visibility. Prompts end up in logs, and a shared dashboard is a disclosure.
8. Test the noisy-neighbour case by saturating one partition and measuring the others.
9. Confirm model weights are separated at rest, with per-tenant keys. A fine-tuned model is tenant data.
10. Write down what happens when a tenant leaves, including how their weights, caches and logs are destroyed.