Best GPUs for Ollama: What Runs Fully on the GPU

Ollama is the easiest way to run language models locally, and that ease is exactly what makes hardware questions about it confusing. It never refuses to run. Give it a model too large for your GPU and it will quietly push the overflow onto the CPU and carry on — slowly.

So the useful question is not “will this GPU work with Ollama?” Almost anything works. The question is which models will run entirely on the GPU, because that is where the performance cliff sits.

What Ollama actually asks of a GPU

Ollama’s model library defaults to 4-bit quantisation. That is why its memory requirements are lower than raw parameter counts imply, and it makes the sizing straightforward:

These figures include context and runtime overhead — they are what you need in practice, not the file size on disk. The reasoning behind them is in How Much VRAM Do You Need for Local AI?

Read down that table and the hardware tiers pick themselves. 16 GB covers everything up to 14B. 24 GB reaches 32B. 48 GB is where 70B begins, and no consumer card gets there alone.

The specifications

The recommendations

Best overall for Ollama

For most people running Ollama, this is the right card.

16 GB covers the models people actually use day to day — the 8B and 14B classes that handle coding assistance, summarisation and general question answering perfectly well. At 896 GB/s they run fast enough that generation keeps ahead of reading speed, which is the threshold where local AI stops feeling like a compromise.

And the setup is nothing: install the NVIDIA driver, install Ollama, run a model. No ROCm packages, no architecture overrides, no checking compatibility matrices.

For larger models

If you want the 32B-class models — and they are a genuine step up in reasoning quality over 14B — this is the consumer card that runs them properly.

32 GB holds a 32B model at 4-bit with real context headroom, so you are not trading window size against fit. The 1,792 GB/s bandwidth means it runs them quickly rather than merely successfully.

It stops short of 70B. Ollama will happily attempt llama3.3 on a 5090 by offloading the overflow to the CPU, and the result will be a few tokens per second. Fitting matters more than speed, every time.

Best budget

The cheapest sensible entry to 16 GB of CUDA memory.

Set expectations correctly: the 128-bit bus gives it 448 GB/s, roughly half the 5070 Ti. Ollama will run the same models on it, at roughly half the tokens per second. For a background service answering occasional queries that is fine. For interactive work it is noticeably slower.

Best value on capacity

24 GB usually costs less here than anywhere else, and Ollama’s ROCm support for text generation is solid.

The caveat is setup rather than capability. Expect to confirm your card is on AMD’s supported list, and on Linux to deal with ROCm packaging. Once running, a 32B model at 4-bit performs well.

If you are exclusively running Ollama, this is defensible value. If you expect to branch into image generation or fine-tuning, read our NVIDIA vs AMD comparison first.

If you need 70B

At some point the answer stops being a graphics card.

A 70B model at 4-bit needs roughly 48 GB. Reaching that with consumer GPUs means two cards, around 900 W, and a build designed around them. A unified-memory machine gets there in a 4.5-litre box drawing about 120 W.

Ollama runs natively on these and addresses the shared memory pool directly. The trade is bandwidth: 256 GB/s means roughly five to six tokens per second on a 70B model — reading pace, not thinking pace. For overnight batch work, for privacy-constrained workloads, or simply for access to models a graphics card cannot load, it is a reasonable exchange.

The one thing to check after buying

Whatever you choose, verify Ollama is actually using it:

ollama ps

The PROCESSOR column is the answer. 100% GPU means fully resident. Anything showing a split — 35%/65% CPU/GPU — means part of the model is on the processor and you are leaving most of your hardware’s performance unused.

If you see a split you did not intend:

• Reduce context. /set parameter num_ctx 4096 frees several gigabytes on a large model.
• Drop a quantisation level. Roughly 20% less memory from Q4 to Q3, though quality suffers.
• Free up VRAM. Browsers with hardware acceleration, video calls and desktop compositors all hold memory.
• Use a smaller model. A 14B at Q6 fully on the GPU beats a 32B half on the CPU, comfortably.

Continue your research

• Ollama Hardware Requirements — the full requirements picture, including CPU, RAM and storage
• Best GPUs for Local AI — the same cards assessed across all AI workloads
• How Much VRAM Do You Need for Local AI? — size your requirement first
• Best Mini PCs for Local LLMs — when a GPU is not the answer