The Dual-GPU 70B Machine: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine, and on a two-card build that caveat carries more weight than it does on a single-card one — the failure modes here are physical, thermal and electrical, and none of them appear on a spec sheet.

The $2,000 machine ends by saying that reaching 70B-class models at home means two cards or a unified-memory machine, and that both are different architectures rather than upgrades. This is the first of those two, written out.

It is also the point where a home AI machine stops being a slightly unusual PC and starts being an engineering problem. Two cards is not one card twice.

What 70B actually costs in memory

Start with the arithmetic, because it determines everything downstream.

A Q4_K_M quantisation occupies roughly 0.6 GB per billion parameters, so a 70B model needs about 42 GB for weights alone, before any KV cache. The VRAM explainer works through why that ratio holds.

That table contains the whole reason this architecture exists. 32 GB is an awkward number — it is more than enough for 32B-class models and not nearly enough for 70B, so the single fastest consumer card on the market cannot run this tier at all. The next rung up is not a faster card. It is more cards.

And 48 GB clears 70B at Q4 by about six gigabytes, which is enough for a working context window and not enough to be relaxed about it. This is a tier you fit into deliberately, not one you grow into.

The four routes to 70B, honestly compared

Two used 24 GB cards is the cheapest route by a wide margin and the one this architecture specifies. It is also the one that brings every problem in this article.

One RTX PRO 6000 is the same job with none of the engineering: 96 GB on one card, 1,792 GB/s, one power connector, one slot decision, and enough headroom to run 70B at Q8 rather than Q4. It costs several times what two used cards do. If that is affordable, stop reading and buy it — nothing below is a better answer, it is only a cheaper one.

A Mac Studio M3 Ultra trades peak speed for capacity, silence and about a quarter of the power draw. 819 GB/s of unified memory is slower than a pair of tensor-parallel 3090s but faster than most people expect, and the large configurations run models that no dual-GPU build reaches at any price. No CUDA, which for some workflows ends the conversation.

A DGX Spark holds 128 GB at 273 GB/s. That is a capacity machine, not a speed machine — it exists so you can develop against a 70B model on the same CUDA stack you deploy to, not so you can chat with one quickly. Read the bandwidth number before buying on the memory number.

The rest of this specifies the first route.

The thing most dual-GPU guides get wrong

Two cards give you the sum of their memory. Whether they give you the sum of their bandwidth depends entirely on which inference engine you run, and the difference is roughly a factor of two in tokens per second.

Layer splitting: capacity adds, speed does not

This is what llama.cpp does, and therefore what Ollama and LM Studio do by default. The model’s layers are divided between the cards — say layers 0–39 on the first, 40–79 on the second. Generating a token walks the layers in order: the first card computes its half and hands a small activation vector to the second, which computes the rest.

Only one card is working at a time. The other is idle, waiting its turn. You have doubled capacity and gained almost nothing in speed — the ceiling is roughly one card’s bandwidth divided by the portion of the model it holds, which works out close to single-card performance.

The upside is that it barely touches the link between the cards. The data passed at each handoff is a single activation vector, measured in kilobytes. A second card sitting in a PCIe 4.0 ×4 slot off the chipset performs about as well as one at ×16.

Tensor parallelism: both cards work at once

This is what vLLM, ExLlamaV2 and TensorRT-LLM do. Every layer’s weight matrices are split across both cards, both compute their slice of every layer simultaneously, and the partial results are combined before moving on.

Now the bandwidth genuinely adds. Each card reads only its own half of the weights, so a 42 GB model becomes two 21 GB reads happening in parallel.

The cost is that combining partial results requires an all-reduce at every layer — eighty times per token for a 70B model. Each transfer is small, but they are frequent and latency-sensitive, and this is where the interconnect starts to matter.

What that means in tokens per second

Using the site’s usual estimate — generation speed is roughly memory bandwidth divided by the size of the weights being read — for a 42 GB model on two 936 GB/s cards:

Those ceilings are arithmetic, not measurements, and real throughput lands below them because attention, sampling and the KV cache all consume bandwidth the formula ignores.

The practical consequence is a decision, not a detail. If you intend to run Ollama and nothing else, you are building a capacity machine and can be relaxed about slot allocation. If you intend to run vLLM, the interconnect belongs in the specification — which is the next section.

Why the RTX 3090 specifically

Not because it is the fastest 24 GB card. The 4090 has higher bandwidth (1,008 vs 936 GB/s), better efficiency and a newer feature set, and if you have two of them, use them.

The 3090 is specified because it is the last GeForce card with NVLink. A three- or four-slot bridge gives the pair a direct 112.5 GB/s link that bypasses PCIe entirely, and that is precisely the interconnect tensor parallelism wants. The 4090 dropped it; every card since has gone without.

So the pairing is unusually coherent: the cheapest route to 48 GB of CUDA memory is also the only consumer route that keeps the fast interconnect. If you are building for vLLM, this is the reason to accept a five-year-old card rather than a newer one.

If you are building for Ollama, NVLink is close to irrelevant and the choice reverts to whichever 24 GB card is cheaper.

The component specification

The motherboard is the constraint

An AM5 CPU provides 24 PCIe 5.0 lanes: sixteen to the primary graphics slot, four to a CPU-attached M.2, four to the chipset link. There are no more.

Running two cards therefore means one of two things.

Bifurcation splits the sixteen CPU lanes into ×8/×8. Both cards sit on CPU lanes with equal, low-latency access. PCIe 5.0 ×8 is about 63 GB/s each, which is ample. The board must explicitly support this — many do not, and the ones that do say so in the manual under a heading like “PCIe bifurcation” or in the slot-allocation table. Do not infer it from the presence of two full-length slots.

Chipset attachment puts the second card in a slot wired through the chipset, typically PCIe 4.0 ×4, sharing a single ×4 uplink with everything else on the board. For layer splitting this is genuinely fine. For tensor parallelism it is the bottleneck, and it is why the second full-length slot on a cheap board is not the same thing as a second graphics slot.

The honest alternative is a Threadripper or Xeon workstation platform, which provides enough lanes to run both cards at full ×16 and never think about it again. It roughly doubles the platform cost for a benefit only tensor-parallel workloads see, which is why this architecture specifies AM5 and tells you to read the manual.

Slot spacing matters as much as lane count. Two triple-slot cards need their slots four positions apart, and the NVLink bridge you buy must match that spacing. Board layouts that place the second ×16 slot three positions down will physically fit two cards and suffocate the top one.

Power: the section people skip

Two 3090s at 350 W board power each, plus a 170 W CPU under load, plus drives, fans and board overhead, is roughly 900 W sustained — and Ampere’s transient spikes sit well above that for microseconds at a time.

Specify 1,300 W minimum, 1,600 W comfortable, ATX 3.1, with enough native connectors for both cards. A 3090 typically wants three 8-pin PCIe connectors, so two cards need six; check what your specific models use, and prefer native cables to daisy-chained ones.

The part that surprises North American builders

A standard 15 A / 120 V household circuit supplies 1,800 W, and electrical code derates continuous loads to 80% of that — 1,440 W. A machine drawing 900 W at the components pulls roughly 1,000 W from the wall after supply losses, which is fine. A machine that transiently peaks toward its 1,300 W supply rating, sharing a circuit with a monitor, a space heater or a kettle, is not.

On 230 V mains this is a non-issue. On 120 V it is a real constraint, and the failure is a tripped breaker mid-generation rather than anything graceful.

Power limiting costs less than you would expect

nvidia-smi -pl 280 caps each card at 280 W, saving 140 W across the pair.

For token generation the performance cost is small — often 5% or less — and the reason follows directly from what this site says everywhere else: decode is memory-bandwidth-bound, not compute-bound. Limiting the power budget mostly limits the shaders, and the shaders were not the constraint. Memory clocks are largely unaffected.

Prefill, fine-tuning and image generation are compute-bound and lose more. But for a machine whose job is conversational inference, power limiting is close to free, and it lowers the thermal problem in the next section at the same time.

Thermals: two cards is not two independent problems

Stack two triple-slot cards in a tower and the top one ingests the bottom one’s exhaust. That card will run 15–20 °C hotter than it would alone, and on a 3090 the number that matters is not the core temperature but the GDDR6X memory junction temperature, which throttles around 110 °C.

The 3090’s memory modules sit on the back of the PCB, cooled through the backplate, directly facing the next card up. It is close to a worst case, and it is the specific reason sandwiched 3090s throttle in ways that sandwiched cards of other generations do not.

Four things help, in descending order of effect:

• Space between the cards. Four slots rather than three, which is a board-layout decision made before you buy anything.
• Power limiting, as above. Less power in is less heat out, at almost no cost to generation speed.
• Replacing the thermal pads. On a five-year-old card the pads have hardened. This is routine maintenance on a used 3090, not a repair.
• Front-to-back airflow with fans actually fitted. Two intake fans directly on the cards, and a case that exhausts rather than recirculating.

The alternative some people reach is a vertical mount and riser for the second card, or an open-frame chassis. Both work. If you use a riser, confirm it is rated for the PCIe generation you intend to run — an underspecified riser will silently link-train down a generation, or produce intermittent errors that look like driver faults.

What the finished machine runs

The KV cache is what consumes the six gigabytes of headroom, and it grows with context length and batch size. A long-context session on a 70B model at Q4 will use all of it. This tier is comfortable at ordinary context lengths and tight at very long ones — the same relationship the 24 GB machine has with 32B models, one rung up.

Before you order: verification checklist

Every item on the single-card architecture’s checklist still applies. These are the additional ones, and they are the ones that make a two-card build fail.

1. Bifurcation, confirmed in the manual. Find the board’s PCIe slot-allocation table and confirm ×8/×8 is supported with two cards installed. If the manual does not say so, assume it does not.
2. Slot spacing, counted. Physically count the positions between the two ×16 slots. Two triple-slot cards need four. Three positions means the top card is suffocating.
3. NVLink bridge spacing, matched. Bridges are sold in 3-slot and 4-slot variants and are not interchangeable. Buy after you have chosen the board, not before.
4. Case expansion slots, counted. Eight, not seven. Many full towers stop at seven, which will not take two triple-slot cards with the required spacing.
5. PSU connector count and type. Six native PCIe 8-pin, or the 12V-2×6 equivalents for your specific cards. Count the cables in the box, not the wattage on the label.
6. Circuit capacity, if you are on 120 V mains. Know what else is on the breaker.
7. Riser cable generation rating, if you are vertically mounting. PCIe 4.0-rated at minimum.
8. Both used cards inspected. Photographs of the PCB and power connectors, fans spinning freely, and a plan to replace the thermal pads on both. Buying two used cards doubles the odds of getting one bad one.
9. Physical weight and sag. Two cards of 2 kg each hanging off the board is a real mechanical load. Support brackets are cheap.

If any of these fails, the failure is in this specification’s fit to your parts, not in your reading of it.

When not to build this

This architecture is the cheapest route to 70B and it is not the easiest one. Three situations where something else is the better answer:

• You want a quiet machine in a room you work in. Two 3090s under sustained load are loud, hot, and produce roughly a small heater’s worth of waste heat. A unified-memory machine does the same job in near silence — the always-on AI server specifies one.
• You only ever run Ollama or LM Studio. You will get capacity without much speed, which is a legitimate goal, but be clear that is what you are buying — and note that a single 32 GB card running a 32B model will feel considerably faster than this machine running a 70B one.
• You value your time more than the price difference. A single 96 GB card removes bifurcation, spacing, NVLink, connector counts, circuit capacity and thermal stacking from your life in one purchase.

The reason to build it anyway is that nothing else reaches 48 GB of CUDA memory for anywhere near the money, and 48 GB is where the 70B tier begins.