NVIDIA GeForce RTX 5090 Review: The 32GB Threshold

The RTX 5090 is the fastest consumer graphics card you can buy for local AI, and the interesting thing about it is not the speed.

It is the 32 GB. That figure clears a specific threshold that 24 GB cards sit just below, and clearing it changes which models you can run rather than how quickly you run the ones you already could. In a field where capacity behaves like a cliff rather than a slope, that is the whole story.

What it does not do is reach 70B. This review is largely about what falls between those two facts.

What you are actually buying

The architecture, briefly

The card is built on GB202, the largest Blackwell consumer die, running 170 streaming multiprocessors for 21,760 CUDA cores. That is roughly double the RTX 5080’s 84 SMs, and it is a cut-down configuration — the full die has more still, which is what the RTX PRO 6000 uses.

Three things changed from Ada that matter for AI work:

Fifth-generation Tensor Cores with native FP4. Blackwell adds hardware support for 4-bit floating point. In principle this roughly doubles effective capacity against 8-bit for models that use it. In practice, support across local inference runtimes is still partial — real for some stacks, absent in others.

GDDR7 on a 512-bit bus. The memory is the headline. GDDR7 signals using PAM3, a three-level encoding that carries three bits over two cycles, where GDDR6 uses simple two-level signalling. Combined with the widest memory interface NVIDIA has shipped on a consumer card, that produces 1,792 GB/s — about 78% more than an RTX 4090.

PCIe 5.0. Largely irrelevant for single-card inference, where weights are transferred once. It matters when a model is split across two cards, or offloaded to system memory.

The 512-bit bus is also why the card has 32 GB. GDDR7 modules are 2 GB each on a 32-bit channel, so 512 bits addresses sixteen of them. Capacity and bandwidth are one design decision, not two — which is why no 24 GB variant exists or will.

AI performance

Token generation

The useful ceiling for generation is memory bandwidth ÷ model size in memory. Applied to this card:

These are calculated from published specifications, not measured by us — see the basis note at the top of this page. Real figures depend on the runtime, the quantisation and the context length, and typically land at 60–80% of the theoretical ceiling.

The row that justifies the card is the 32B one. Sixty to seventy-five tokens per second on a 32B-class model is comfortably faster than reading speed, which means the model keeps ahead of you rather than making you wait. That is the difference between a tool and a demonstration.

Prompt processing, which the bandwidth figure does not describe

Generation is only half of inference, and the half that gets quoted.

Before a model generates anything it has to read your prompt, and that phase — prefill — works completely differently. Every prompt token is available at once, so the work becomes large matrix-by-matrix multiplication that saturates the arithmetic units. Prefill is compute-bound, not bandwidth-bound.

This is where the 5090’s 21,760 CUDA cores do something the bandwidth number does not capture. Against an RTX 5080 it has roughly twice the compute against 1.87× the bandwidth — so the gap is wider on prompt processing than on generation.

For short conversational prompts, none of this is visible. For the work people actually buy these cards for — pasting a large source file, running retrieval-augmented generation with several thousand tokens of context, analysing a document — prefill is most of the wait, and this is the card’s least-discussed advantage.

Running it as a service

If you host a model for more than one person, spare VRAM converts directly into concurrent sessions, because every conversation needs its own KV cache alongside the shared weights.

A 14B model at Q4 occupies about 8.5 GB. On a 16 GB card that leaves roughly 6 GB for caches; on the 5090 it leaves roughly 22 GB. The weights are a fixed cost both cards pay first, so the difference in usable concurrency is closer to fourfold than twofold.

Batching also improves efficiency: several sequences share each weight read, which amortises the memory-bound bottleneck. A card serving eight concurrent users produces far more total throughput per watt than one serving a single user.

What 32 GB means in practice: budgeting context

The capacity headline is easy. What it buys you at a given context length is more useful, and rarely spelled out.

A 32B-class model with grouped-query attention uses roughly 0.25 MB of KV cache per token. Adding that to a 19 GB Q4 model and about 1.5 GB of runtime overhead:

Two things follow. A 32B model at 32K context — which covers most practical work — fits with room to spare. And quantising the KV cache to 8 bits, which most runtimes support, roughly doubles the context you can hold for a small quality cost. That option is worth knowing before concluding a workload will not fit.

Choosing a quantisation at 32 GB

The other thing capacity buys is the freedom to run a model at higher precision rather than a bigger model badly:

A 32B model at Q5 or Q6 is a genuinely better model than the same one at Q4, and 24 GB cards cannot hold it with usable context. That is a quieter benefit of the extra 8 GB than the headline model-size jump, and for day-to-day output quality it may matter more.

The 70B question

It does not fit, and it is worth being precise about why.

A 70B model at Q4_K_M needs roughly 42 GB for weights before any context. The card has 32 GB. No quantisation closes a 10 GB gap without dropping to Q3 or below, at which point a 32B model at Q5 produces better output for less memory.

You can run 70B by offloading layers to system RAM. It works, and the offloaded portion crosses PCIe 5.0 at about 64 GB/s against 1,792 GB/s on-card — roughly 28 times narrower, in the critical path for every token. Expect five to fifteen times slower. It is a fallback, not a plan.

If 70B-class models are your requirement, this is not your card. Look at unified-memory machines or the RTX PRO 6000.

Image generation

Here the card’s compute matters and its memory largely does not.

Diffusion models are compute-bound rather than bandwidth-bound, and they fit comfortably in 16 GB at typical resolutions. The 5090’s shader count makes it substantially faster per image than any card below it — but a 16 GB RTX 5070 Ti does everything it does, more slowly, for far less money.

Where 32 GB earns its place is large batches, very high resolutions, and ComfyUI workflows that chain several models simultaneously without unloading between stages.

Fine-tuning

32 GB opens up parameter-efficient fine-tuning — LoRA and its relatives — on 7B to 14B models that 16 GB cards struggle with.

Full fine-tuning of anything substantial remains out of reach, and the arithmetic explains why: gradients match the parameter count, and an AdamW optimiser keeps two further 32-bit states per parameter. For a 7B model that is roughly 14 GB of weights, 14 GB of gradients and 56 GB of optimiser state. That is a 96 GB problem, which is what the RTX PRO 6000 exists to solve.

You can push some of that state into system RAM with DeepSpeed ZeRO-Offload or a paged optimiser, which is a legitimate technique and a reason to specify 128 GB of system memory if training is a real part of your plans.

Power, heat and noise

This is the part of the card most easily underestimated, and the reason it is not a straightforward recommendation.

575 W is a sustained load, not a bursty one. Gaming loads fluctuate; inference does not. A model generating for twenty minutes holds the card near its power limit for twenty minutes. The thermal problem is different in kind from the one a gaming benchmark measures.

Power delivery, specifically

The card uses the 12V-2×6 connector — the revised design that followed 12VHPWR, with shortened sense pins so that a connector which is not fully seated fails to negotiate full power rather than delivering it through partial contact. Seat it properly and check the latch; the failure mode this revision addresses was largely a seating problem.

More importantly, the wattage on a power supply is not the whole specification. GPUs draw brief transient excursions well above their rated figure, and a supply that trips its protection on those will shut the machine down under load even though the average draw is within budget. Supplies built to the ATX 3.1 specification are rated to tolerate large short-duration excursions, and that compliance matters more than another 200 W of headroom on a supply that is not.

A 1,000 W ATX 3.1 unit from a reputable manufacturer is the sensible target.

Cooling and physical fit

NVIDIA’s Founders Edition is a genuinely impressive piece of engineering: a two-slot card dissipating 575 W through a dual flow-through design, where both fans push air through the heatsink and out rather than recirculating it in the case.

Board partner cards are usually three slots or more, and cool more quietly at the cost of taking the space. Which is right depends on your case — and on whether you might ever want the slot below.

Either way, verify clearance before ordering. These are long cards, and a 575 W thermal load also needs case airflow that actually moves air through rather than around.

Idle power, which matters for an always-on box

Something worth knowing if the machine will run continuously as an inference server: high-end cards idle at a modest but non-trivial figure, and certain display configurations — multiple monitors, or high refresh rates — can prevent memory from downclocking, holding idle draw considerably higher than it needs to be.

If the machine is headless or single-display, this is a non-issue. If it drives three high-refresh panels and also serves models overnight, it is worth measuring.

The technique nobody mentions: cap the power limit

Blackwell, like Ada before it, ships configured high on its voltage-frequency curve — the last few percent of clock speed costs a disproportionate amount of power.

Reducing the power limit is a single slider in most tuning utilities, and for inference specifically the case is stronger than it is for gaming. Generation is memory-bandwidth-bound, and memory clocks are not what a power cap principally restrains. So the performance you give up should be smaller than the power you save.

For a card that will run sustained loads in a room you work in, capping the power limit is the highest-value adjustment available: less heat, less noise, lower running cost, and a machine that is more pleasant to sit next to.

We have not measured the exact trade-off on this card and will not quote a figure we have not produced. It is a technique worth testing on your own hardware, with your own model, on the first day.

Noise

Removing 575 W of heat makes sound, and in a room you work in all day that is a genuine consideration.

We have not measured this card’s acoustics or sustained thermal behaviour ourselves, and those are precisely the questions specification analysis cannot answer. Cooler designs differ substantially between board partners, so treat published figures with appropriate caution and read measurements from someone who has the card in front of them.

Software

Blackwell is a current architecture, which mostly means everything works and occasionally means something does not.

The card needs a recent CUDA toolkit and driver branch. That is unremarkable on a desktop, and it does bite in one specific place: older container images pinned to earlier CUDA versions will not see the GPU. If your workflow involves pulling a published Docker image for an inference server or a training framework, check the CUDA version it was built against before assuming it will run.

Beyond that, the mainstream local AI stack — Ollama, llama.cpp, LM Studio, vLLM, the diffusion ecosystem — treats it as a first-class target.

Value

The 5090 is not good value in any conventional sense. It costs far more than an RTX 5070 Ti and does not run 70B models, which is the capability people most often want when they reach for a flagship.

Its value case is narrow and real: it is the only consumer card that runs 32B-class models properly, with enough headroom left for long context and higher-precision quantisation. If that is what you need, there is no cheaper way to get it, and the alternatives — two cards, a workstation GPU, unified memory — are each worse on some axis that matters.

If you do not need it, almost everything else on the market is better value. Our Best GPUs for Local AI guide covers the alternatives properly.

There is a longevity argument that quietly favours it. Compute ages quickly; memory capacity ages slowly, because model sizes have only moved in one direction. A 24 GB card from 2020 still runs 32B models today, while faster cards from the same era with 8 GB do not. Whatever you buy, VRAM is the number you will still care about in three years.

How we scored it

Against our nine criteria, on a five-point scale:

Overall: 4.1 out of 5 — the weighted mean of the criteria that apply.

The weights are not decoration. A flat average let thermals, noise and power efficiency count three times over against AI performance’s one, which on a site about AI hardware measured the wrong thing — and this card, which is punishing on all three and exceptional on the first, was among those most distorted by it. The methodology page sets out the full table and why it changed.

What to consider instead

RTX 5070 Ti — 16 GB and 896 GB/s at 300 W. If your models are 14B and below, this does the same job for far less money and heat.

RTX 4090 — 24 GB and 1,008 GB/s. Reaches 32B at Q4 with less context headroom, and on the used market is frequently better capability per dollar.

RTX PRO 6000 Blackwell — 96 GB at the same bandwidth. Several times the price, and the single-card answer to 70B and to real fine-tuning.

Framework Desktop, 128 GB — runs 70B at roughly six tokens per second, at around 120 W. Slow, and it does something the 5090 cannot do at all.

Who should buy it

• You want to run 27B and 32B-class models at interactive speed, at good quantisation, with long context
• You do long-context work — large-file analysis, retrieval-augmented generation, whole-repository prompts
• You intend to serve a model to other people
• Local AI is a primary purpose of the machine, not an occasional use
• You have or will build a system that handles 575 W properly

Who should skip it

• Your models are 14B or smaller — the 5070 Ti is the better purchase
• You need 70B-class models — this card does not reach them
• Image generation is your main workload — you are paying for memory you will not use
• You cannot supply 575 W, or noise in your workspace matters and you would not power-limit it
• Value per dollar is the deciding factor

Continue your research

• RTX 5090 vs RTX 5080 for AI — the comparison most buyers are actually making
• Best GPUs for Local AI — where this card sits in the range
• How Much VRAM Do You Need for Local AI? — whether 32 GB is the right target for you
• Best AI Workstations — building a machine around it