The Fine-Tuning Workstation: A Reference Architecture

This is a reference architecture, not a tested build. Components are specified from published manufacturer documentation and are compatible on paper. Nobody at AI Gear Stack has assembled this machine or run a training job on it.

It is also the first architecture on this site for a workload that is not inference. Every other page here — all thirty-two of them — assumes you are running a model someone else trained. This one assumes you are changing the weights, and almost every rule the rest of the site teaches inverts when you do.

Before any of that, though, the recommendation most readers need.

Most people should rent, not build

Fine-tuning is bursty. You prepare a dataset for a week, train for six hours, evaluate, and then do nothing with the hardware for a fortnight. That is close to the worst possible utilisation pattern for a depreciating asset that also costs money to keep switched on.

Rented compute inverts every one of those properties. You pay for the six hours. You get current-generation accelerators with memory configurations no consumer card offers. You do not own anything that is worth less next quarter.

Build this machine only if one of these is true:

• You iterate constantly. Several runs a week, indefinitely. At that duty cycle ownership starts to make sense, and the waiting-for-a-machine tax on rented capacity starts to hurt.
• The data cannot leave the building. Patient records, legal discovery, unreleased source, anything under a contract that names where it may be processed. This is the strongest reason and it is not really an economic one — and if the contract requires physical isolation rather than merely local processing, the air-gapped machine specifies what that costs to operate.
• You are learning. A local machine you can break at three in the morning without watching a billing meter is a genuinely better teacher than rented capacity. Buy less than you think you need.

If none of those describe you, rent, and spend the difference on the $2,000 inference machine instead. We would rather say that plainly than sell you a workstation.

Why the memory arithmetic changes completely

The rest of this site works from a single number: a Q4_K_M model needs roughly 0.6 GB per billion parameters, and generation speed is bandwidth divided by that. Training uses a different number, and it is much larger.

Full fine-tuning at mixed precision with AdamW holds, per parameter:

Sixteen bytes per parameter, before a single activation. A 7B model therefore needs about 112 GB to full fine-tune, which is why almost nobody does it on one machine at home — and why the 64 versus 128 GB comparison notes that the offloaded optimiser state alone is around 56 GB for a 7B model.

Note what dominates. The weights are the small part. Optimiser state is eight of those sixteen bytes; in inference it does not exist at all.

The two techniques that make this tractable

LoRA freezes the base model and trains small low-rank adapter matrices alongside it. Frozen weights need no gradients and no optimiser state, so the sixteen bytes collapses to two for almost every parameter, with full optimiser overhead on only the fraction of a percent that is trainable. A 7B LoRA fits in 24 GB with room for activations.

QLoRA goes further and holds the frozen base in 4-bit, dequantising on the fly during the forward pass. The base model shrinks to roughly 0.5 bytes per parameter — a 7B base becomes about 3.5 GB — at the cost of extra compute for dequantisation. This is what makes fine-tuning a 70B model on a home machine possible at all.

Those are working figures with modest sequence lengths, not guarantees. An 8-bit optimiser (as in bitsandbytes) replaces the fp32 momentum and variance with 8-bit equivalents, taking full fine-tuning from roughly 16 bytes per parameter to about 6. It is the difference between 7B full fine-tuning being impossible on 96 GB and being merely tight.

Activations, and the setting that saves you

The tables above ignore activations, which is safe only at short sequence lengths. Activation memory scales with batch size × sequence length × hidden size × layer count, and at 8k or 16k context it can exceed everything else combined.

Gradient checkpointing is the lever. Instead of storing every intermediate activation for the backward pass, it stores a subset and recomputes the rest — roughly square-root memory scaling for about 30% more compute. At long sequence lengths it is not an optimisation, it is the difference between the run starting and not.

If a job dies with an out-of-memory error partway through the first backward pass, activations are almost always the reason, and gradient checkpointing is almost always the fix.

What inverts from the rest of this site

Five things this site says confidently about inference machines stop being true here.

Compute stops being irrelevant

Token generation is memory-bandwidth-bound, which is why we spend the whole budget on VRAM capacity and bandwidth and tell you the CPU and shader count barely matter.

Training is compute-bound. A forward pass, a backward pass and an optimiser step over large batches is dense arithmetic, and it keeps the tensor cores genuinely busy. Shader throughput and native low-precision formats now predict your throughput the way bandwidth predicts generation speed.

That reverses a recommendation. The RTX 5090 review notes that much of its 21,760-core compute advantage goes unused during inference. Here it does not go unused — and its native FP8 and FP4 tensor support is doing real work rather than sitting idle.

System memory stops being surplus

The 64 versus 128 GB comparison concludes that 64 GB is ample for a discrete-GPU inference machine, because weights live on the card.

For training, system RAM does three jobs it never does during inference: it holds datasets during preparation and tokenisation, it buffers the data loader feeding the GPU, and — if you use ZeRO-Offload or an equivalent — it holds optimiser state that will not fit in VRAM. 128 GB is the specification here, and this is the one machine on the site where filling that is not over-buying.

Storage endurance starts to matter

Inference reads weights once per session and never writes. Training writes checkpoints repeatedly, and a checkpoint is not small: a 7B model at fp16 is roughly 14 GB, written every few hundred steps, for the length of the run.

That is a write-endurance workload, and consumer NVMe drives are rated in terabytes written for a reason. The NVMe guide spends most of its length on the SLC cache cliff — the point where a drive’s fast write buffer fills and throughput collapses to native TLC speed — and checkpointing is exactly the pattern that finds it.

Specify separate drives for the dataset, the checkpoints and the operating system. Not for speed, but so the checkpoint drive can be a consumable that gets replaced without disturbing anything else.

ECC stops being a professional-market upsell

A single-bit memory error during inference produces one strange token in one reply, and you never notice.

The same error during hour nine of a twelve-hour training run corrupts a gradient, which corrupts the weights, which propagates to every subsequent step. You find out at evaluation, if you find out at all — and the failure looks like a training problem rather than a hardware one, which is worse.

This is the workload where the RTX PRO 6000’s ECC GDDR7 stops being a line item on a spec sheet aimed at somebody else. It does not make the card faster. It makes multi-day runs trustworthy.

PCIe starts to matter across cards

The dual-GPU architecture explains that layer-split inference passes a few kilobytes between cards per token, so a second card on a chipset-attached ×4 slot performs fine.

Distributed training passes the full gradient tensor in an all-reduce at every optimiser step. That is gigabytes, not kilobytes, and it happens continuously. If you intend to train across two cards, the interconnect belongs in the specification for real — ×8/×8 bifurcation at minimum, and the NVLink argument from that article applies with considerably more force.

The component specification

Read that as a specification rather than a shopping list. The requirements — memory capacity, compute class, write endurance, sustained thermals — hold; a named model is one example that satisfies them.

On the 128 GB memory decision

This is the one architecture where the site’s usual “two DIMMs, not four” advice is worth revisiting. Two 64 GB modules keep DDR5 at full speed and leave the slots free; four 32 GB modules are often cheaper and will typically drop the memory controller below 5,000 MT/s on a consumer board.

For training, take the capacity and accept the frequency loss if that is the cheaper route. Memory bandwidth is not the bottleneck in a workload that is compute-bound on an accelerator, and running out of system memory during dataset preparation costs more than a few percent of throughput ever will.

Thermals over days, not hours

Every other machine on this site is specified for a load that lasts minutes to hours. A training run lasts as long as it lasts, and the relevant question is not peak temperature but steady state after six hours with the case closed.

Three consequences worth designing for:

• Fan curves tuned for sustained load, not for a quiet desktop that ramps under bursts. The machine will sit at its steady state for the whole run.
• A room that can take the heat. A 600 W card plus the rest of the machine is close to a kilowatt of continuous heat. In a small room in summer, ambient rises until the machine is ingesting its own exhaust.
• Checkpoint often enough to survive a thermal shutdown. If the run dies at hour nine, the only question that matters is when it last wrote state. This is a cheap insurance policy people skip.

Before you order: verification checklist

The fitment checks from the $2,000 architecture still apply. These are the additional ones specific to training.

1. Run the memory arithmetic for your actual target before buying anything — model size, technique, sequence length, batch size. If the number exceeds the card you were about to buy, no other decision here matters.
2. Confirm the card’s power connector and PSU headroom. 600 W board power is a different specification from 350 W, and Max-Q variants are configurable to 300 W, which changes the answer.
3. Check the framework supports your card’s numeric formats. FP8 and FP4 need software that emits them; an older PyTorch or a pinned container may not.
4. Verify ECC is enabled if you bought a card that has it. It is a driver-level setting, and it can be off.
5. Confirm three separate drives are physically possible on the board you chose, at the lane allocation you expect. Boards frequently disable a SATA controller or an M.2 slot when another is populated — this is in the manual, in a footnote, in a table.
6. Establish a checkpoint cadence and test a resume before the first long run, not during it.
7. Measure steady-state temperatures under an hour of synthetic load with the case closed and the door shut, before committing to a twelve-hour job.
8. Price a rented equivalent for the runs you actually plan. Do this last, honestly, and be willing to abandon the build. It is the check most likely to save you the most money.

When not to build this

• If your duty cycle is low. Covered at the top, and it is the common case. Rent.
• If you only want to run models, not change them. The $2,000 machine is a better inference machine for less money, and the dual-GPU architecture reaches larger models than this one fine-tunes.
• If you need it quiet or always on. This machine is neither, deliberately. The always-on server is specified for the opposite constraint.
• If you have not yet fine-tuned anything. Do a QLoRA run on a 7B model on rented capacity first. It costs very little and it will tell you more about what you need than any specification can, including this one.