“How much VRAM do I need?” is the first question almost everyone asks when they start running models locally, and it is usually answered with a rule of thumb that falls apart the moment you try to use it.
The honest answer is that it depends on three things — how many parameters the model has, what numeric precision you run it at, and how much context you need — and all three are things you control. Once you can do the arithmetic yourself, hardware buying gets dramatically simpler, because you stop asking “is this GPU good?” and start asking “will my workload fit in this, and how fast will it go?”
This page shows you the arithmetic.
Why memory is the binding constraint
When a language model generates text, the weights have to live somewhere the accelerator can reach. If they fit in GPU memory, generation runs at the speed of that memory. If they do not fit, one of two things happens:
- The runtime refuses to load the model. Clean failure, easy to diagnose.
- The runtime splits the model, keeping some layers in GPU memory and offloading the rest to system RAM across the PCIe bus. This works, and it is slow — typically five to fifteen times slower, because every offloaded layer is now bottlenecked by a link that is an order of magnitude narrower than the GPU’s own memory bus.
That is why memory capacity behaves like a threshold rather than a gradient. A 16 GB card is not “two thirds as good” as a 24 GB card for a workload that needs 18 GB. It is the wrong card.
Step 1: model size at different precisions
A model’s parameter count is the headline number — 7B, 32B, 70B. What that costs in memory depends entirely on how many bits each parameter is stored in.
Full precision is 16 bits (2 bytes) per parameter. Quantisation reduces that, trading a small amount of output quality for a large amount of memory. The common 4-bit formats — Q4_K_M and its relatives — average roughly 4.5 to 5 bits per weight once you account for the metadata each block carries.
Approximate memory for weights alone:
| Precision | Per parameter | 7B | 14B | 32B | 70B | 120B |
|---|---|---|---|---|---|---|
| FP16 (full) | 2 bytes | 14 GB | 28 GB | 64 GB | 140 GB | 240 GB |
| Q8 | ~1 byte | 7 GB | 14 GB | 32 GB | 70 GB | 120 GB |
| Q6_K | ~0.8 GB/B | 5.7 GB | 11 GB | 26 GB | 57 GB | 98 GB |
| Q5_K_M | ~0.7 GB/B | 5 GB | 10 GB | 22 GB | 48 GB | 84 GB |
| Q4_K_M | ~0.6 GB/B | 4.4 GB | 8.5 GB | 19 GB | 42 GB | 72 GB |
| Q3_K_M | ~0.5 GB/B | 3.5 GB | 6.8 GB | 15 GB | 34 GB | 58 GB |
Treat these as planning figures rather than exact values — the precise size varies between model architectures and between quantisation tools. They are close enough to decide what hardware to buy.
Which quantisation should you actually use?
For most people, most of the time: Q4_K_M. It is the point on the curve where memory savings are large and quality loss is small enough that it rarely shows in ordinary use.
- Q8 is worth it if you have memory to spare and are doing something precision-sensitive, such as structured extraction or code generation with strict output formats.
- Q5_K_M is a sensible middle step when a model at Q4 leaves several gigabytes unused.
- Q3 and below degrade noticeably. A 70B model at Q3 is usually worse than a 32B model at Q5, despite using more memory — parameter count is not a free lunch once you quantise hard enough.
The general rule: prefer a smaller model at higher precision over a larger model crushed into 3-bit.
Step 2: add the context window
Weights are only part of the bill. Every token in the conversation is stored in the KV cache — the key and value tensors the attention mechanism reuses instead of recomputing — and that cache grows linearly with context length.
The size per token depends on the model’s architecture:
KV bytes per token ≈ 2 × layers × KV heads × head dimension × bytes per element
For a 70B-class model with 80 layers, 8 grouped-query KV heads and a head dimension of 128, at FP16 that works out to roughly 0.33 MB per token. Which gives:
| Context length | 8B-class model | 70B-class model |
|---|---|---|
| 4,096 tokens | ~0.5 GB | ~1.3 GB |
| 8,192 tokens | ~1 GB | ~2.7 GB |
| 32,768 tokens | ~4.2 GB | ~10.7 GB |
| 128,000 tokens | ~16 GB | ~42 GB |
Two things follow from this table, and both surprise people.
Long context is expensive. Running a 70B model at its full advertised context can cost as much memory again as the weights. “Supports 128K context” is a capability of the model, not a promise your hardware can use it.
Context is adjustable. Most runtimes let you cap it, and many support quantising the KV cache to 8-bit, roughly halving its footprint. If you are 3 GB short, dropping from 32K to 8K context usually solves it with no effect on quality for typical use.
Step 3: add overhead
Budget another 1–2 GB for the CUDA or ROCm context, the runtime’s working buffers, and — if the GPU also drives your monitors — the desktop itself. A 4K display with a browser open can hold half a gigabyte or more.
The working formula
Memory needed ≈ (parameters × bytes per parameter) + KV cache + 1–2 GB overhead
Or, for quick mental arithmetic at 4-bit: take the parameter count in billions, multiply by 0.6, add 20%.
A 32B model at Q4: 32 × 0.6 = 19 GB, plus 20% ≈ 23 GB. Which is exactly why 24 GB cards run 32B models and 16 GB cards do not.
What each VRAM tier actually gets you
| VRAM | Comfortable ceiling (Q4) | Honest assessment |
|---|---|---|
| 8 GB | 7B with short context | Entry level. Fine for learning and small assistants; you will hit the wall quickly. |
| 12 GB | 7B–8B with long context | Workable. Adequate for image generation at typical resolutions. |
| 16 GB | 14B comfortably | The current sensible minimum. Handles most everyday assistant and coding work. |
| 24 GB | 32B at Q4 | The sweet spot. 32B-class models are a real capability step up from 14B. |
| 32 GB | 32B with long context, or Q5 | Meaningful headroom. Still short of 70B. |
| 48 GB | 70B at Q4 | The entry point for 70B-class models — usually two 24 GB cards. |
| 96 GB+ | 70B at Q8, or 120B at Q4 | Workstation and unified-memory territory. |
Note the gap between 32 GB and 48 GB. It is the most awkward step in the whole ladder, because no single consumer graphics card sits in it. Getting to 70B on consumer hardware means either two GPUs or a unified-memory machine.
The other half of the question: bandwidth
Capacity tells you what will fit. It says nothing about speed — and the two are often inversely related in the machines you might actually buy.
Generating each token requires reading the active model weights out of memory. That makes generation almost entirely memory-bandwidth-bound, which gives a useful ceiling:
theoretical tokens/second ≈ memory bandwidth ÷ model size in memory
Real-world results typically land at 60–80% of that figure. Applied to hardware you can buy today:
| Machine | Memory | Bandwidth | 70B Q4 (~42 GB) |
|---|---|---|---|
| RTX 5090 | 32 GB | 1,792 GB/s | Will not fit |
| RTX 4090 | 24 GB | 1,008 GB/s | Will not fit |
| Mac Studio M3 Ultra | 512 GB | 819 GB/s | ~19 t/s ceiling |
| Mac Studio M4 Max | 128 GB | 546 GB/s | ~13 t/s ceiling |
| NVIDIA DGX Spark | 128 GB | 273 GB/s | ~6.5 t/s ceiling |
| Ryzen AI Max+ 395 | 128 GB | 256 GB/s | ~6 t/s ceiling |
This table is the single most useful thing on this page, because it explains the result that confuses everyone: the fastest consumer graphics card in the world cannot run a 70B model at all, while a mini PC drawing a fifth of the power can.
It also shows the cost of that trade. The mini PC runs it at roughly six tokens per second — about as fast as a person reads, and slow enough to be frustrating if you are iterating.
Where unified memory changes the answer
Machines built around unified memory — Apple silicon, AMD’s Strix Halo, NVIDIA’s GB10 — share one memory pool between CPU and GPU. There is no separate VRAM to run out of.
That is transformative for capacity. A 128 GB unified machine can allocate most of that pool to the model, putting 70B-class work within reach at a fraction of the price and power of a multi-GPU build.
The catch is bandwidth. Even the fastest of these (Apple’s M3 Ultra at 819 GB/s) sits below a mid-range discrete card, and the LPDDR5X-based systems sit far below.
The practical guidance:
- If your models fit in 32 GB, buy a discrete GPU. You will get several times the token rate.
- If you need 70B-class models, buy unified memory. Slow is infinitely faster than does-not-run.
- If you need both, that is what a discrete card for daily work plus a unified-memory box for the large models is for.
What about splitting across two GPUs?
Two 24 GB cards give you 48 GB and will run a 70B model at Q4. This is a real and well-trodden option, with real caveats:
- It is not free bandwidth. Layers are split between cards, and activations cross the PCIe bus at each boundary. Expect meaningfully less than double a single card’s throughput.
- Power and heat are cumulative. Two 450 W cards need a 1,200 W-class supply and a case that can actually move the air.
- Not every runtime handles it equally well. llama.cpp and vLLM both support multi-GPU; support elsewhere is patchier.
For a machine that already has one capable card and a free slot, adding a second is often the cheapest route to 48 GB. For a new build, weigh it honestly against a unified-memory machine.
Image generation is a different problem
Everything above concerns language models. Diffusion models behave differently:
- They are far smaller. Most current image models fit comfortably in 8–12 GB.
- They are compute-bound rather than bandwidth-bound, so raw GPU throughput matters more than memory speed.
- VRAM becomes the constraint at high resolutions, with large batches, or when chaining several models in a ComfyUI workflow.
For image generation, 12 GB is a workable floor and 16 GB is comfortable. Beyond that, spend on compute rather than capacity.
Common questions
Can I run a model larger than my VRAM?
Yes, by offloading layers to system RAM — but expect it to run five to fifteen times slower, because offloaded layers are bottlenecked by the PCIe bus rather than the GPU’s own memory. It is fine for occasional use and painful as a daily workflow.
Does system RAM matter if the model fits in VRAM?
Much less than people expect. Once weights are resident in GPU memory, system RAM mostly handles loading and the operating system. 32 GB is comfortable for a discrete-GPU machine. It matters enormously on unified-memory systems, where system RAM is the model memory.
Is more VRAM always better?
Only up to what your workload needs. Capacity you never use contributes nothing, and cards frequently trade bandwidth for capacity at a given price. A 16 GB card on a 128-bit bus will load models it then runs slowly.
How do I check what a model will actually use?
Load it and watch. nvidia-smi on NVIDIA hardware, rocm-smi on AMD, or Activity Monitor on macOS will show real allocation including context and overhead. Measured beats estimated every time.
Do mixture-of-experts models change the maths?
Yes, in a useful direction. MoE models keep all parameters in memory but only activate a fraction per token, so capacity requirements follow the total parameter count while speed follows the much smaller active count. They run faster than their size suggests.
What to read next
Now that you can size a model, the buying decision becomes concrete:
- Best GPUs for Local AI — which cards actually make sense once you know your memory target
- Ollama Hardware Requirements — the same arithmetic applied to the most common runtime
- 64GB vs 128GB RAM for Local AI — when system memory is worth doubling
- Best AI Workstations — complete machines, chosen by what they can run