NVIDIA vs AMD for Local AI: What the Gap Actually Costs

This comparison is unusual because the hardware is not really the question.

AMD’s cards are competitive on the specifications that matter for AI — a Radeon RX 7900 XTX has 24 GB and 960 GB/s, which is more memory than an RTX 5080 and the same bandwidth. On paper it is excellent value.

The question is what happens when you try to use it. And the honest answer depends almost entirely on which workload you have, because the gap is not uniform: for one of them AMD is genuinely fine, and for the others it is a recurring tax on your time.

The hardware, side by side

Read that table without knowing the vendors and AMD looks like the obvious value choice. The 7900 XTX offers 50% more memory than an RTX 5080 at the same bandwidth, usually for less money.

The hardware is not where the difference lives.

What CUDA actually is

The usual explanation — “CUDA is more popular” — is true and explains nothing. It is worth understanding what the moat is actually made of, because that determines how quickly it could close.

CUDA is three things stacked on top of each other:

A programming model and compiler. This layer is genuinely replaceable, and AMD has replaced it. HIP is a near-identical API, and AMD ships hipify, a tool that mechanically translates CUDA source into HIP. For straightforward code this works.

Hand-tuned mathematical libraries. cuBLAS and cuDNN have had close to two decades of engineers optimising matrix operations for each new architecture. AMD’s rocBLAS and MIOpen are the equivalents, and they are good — but “good” and “twenty years of tuning against every generation” are different things.

The kernel ecosystem — and this is the real moat. Modern inference does not use generic library calls. It uses thousands of purpose-written GPU kernels: FlashAttention and its successors, quantisation kernels such as Marlin and ExLlama’s, fused operations, custom Triton code from research repositories. Almost all of it was written against CUDA, by people who had NVIDIA hardware in front of them.

That third layer is why porting is harder than it looks. The API translates cleanly; the kernels frequently do not, because they reach past the API into architecture-specific behaviour.

Where translation breaks

Three recurring reasons, worth knowing so the failures are legible when you hit them:

• Inline PTX assembly. Performance-critical CUDA kernels often drop into NVIDIA’s assembly language. There is no mechanical translation for that.
• Warp versus wavefront width. NVIDIA executes threads in groups of 32. AMD’s compute-oriented architectures use 64, and RDNA uses 32. Any kernel that assumes a fixed group size for its data layout needs rewriting, not translating.
• Architecture-specific intrinsics. Tensor core instructions, asynchronous copy operations and memory-hierarchy hints all have vendor-specific forms.

None of this is insurmountable. All of it is work that somebody has to do, per kernel, and the incentive to do it lands wherever the hardware is.

How ROCm behaves in practice

ROCm has come a long way, and the experience is uneven in a way that is predictable once you know where to look.

The supported-GPU list is narrower than the hardware

AMD officially validates ROCm against a specific list of GPUs, and it is shorter than the set of cards people own. A card missing from it may still work perfectly well — the runtime simply refuses to load kernels built for a different architecture string.

The widely used workaround is an environment variable:

HSA_OVERRIDE_GFX_VERSION=11.0.0

That tells the runtime to treat your GPU as a nominated supported architecture, so it loads those kernels. It works when the underlying instruction sets are genuinely compatible — which is often, within a generation. It is a community practice rather than a supported configuration, and it is exactly the kind of thing that makes people describe AMD as “fine, once you get it working”.

Linux packaging is the other friction point

ROCm versions, kernel versions and distribution packages all have to agree. On a supported distribution at a supported release this is uneventful. Off that path it is an afternoon.

The third option nobody mentions: Vulkan

Worth knowing, because it sidesteps most of the above.

llama.cpp has a Vulkan backend that runs on essentially any modern GPU — AMD, NVIDIA, Intel, and integrated graphics — with no vendor runtime to install at all. Performance is generally below a native ROCm or CUDA path, sometimes noticeably. But it works immediately, it works on unsupported cards, and for text generation it is frequently fast enough.

If you own an AMD card and want to run language models today without a ROCm expedition, start here.

Where performance actually lands

Assuming you get past setup, the two vendors diverge differently depending on which phase of inference you are measuring — the same split that separates the RTX 5090 from the 5080.

Token generation is memory-bound. It reads the active weights out of memory to produce each token, so it is governed by bandwidth rather than by how clever the kernels are. Both vendors reach a broadly similar fraction of their theoretical bandwidth here, which means a 960 GB/s Radeon performs much like a 960 GB/s GeForce. This is the case where AMD’s hardware value is real value.

Prompt processing is compute-bound. It hammers the matrix units, and this is precisely where twenty years of kernel tuning shows up. NVIDIA’s advantage is larger here, and it grows with context length.

So the practical shape of it: for short prompts and conversational use, an AMD card of equivalent bandwidth is genuinely competitive. For long-context work — large files, retrieval-augmented generation, whole-repository prompts — the gap widens.

Quantisation formats, which quietly constrain you

An under-discussed practical limit. Not every quantised model format runs on every vendor:

If you live entirely in Ollama or LM Studio — both GGUF — this table never affects you. If you want to pull an arbitrary quantised model from Hugging Face and run it in vLLM, it does, and it does so at the least convenient moment.

Workload by workload

Text generation — AMD is competitive

If you run language models through Ollama, llama.cpp or LM Studio, an AMD card is a reasonable purchase. The back-ends are mature, performance tracks the hardware specifications, and 24 GB for the price is a real advantage.

Expect a slower start: check the supported list, install ROCm, possibly set an override — or take the Vulkan path and skip most of it. Once running, it runs.

Verdict: AMD is genuinely good value here.

Image generation — NVIDIA, clearly

Diffusion pipelines assume CUDA more deeply than text generation does. Custom nodes, optimisation extensions and new sampling techniques land on CUDA and reach ROCm later, if at all.

Compounding it, image generation is compute-bound and fits in 16 GB, so AMD’s memory advantage does not apply. You are giving up ecosystem support in exchange for capacity you will not use.

Verdict: buy NVIDIA.

Fine-tuning — NVIDIA, decisively

The training ecosystem is CUDA-first to a degree that is difficult to overstate. Memory-efficient attention kernels, quantised training, and the optimiser implementations everyone uses all target CUDA. This is the deepest part of the moat.

Verdict: buy NVIDIA.

Serving at scale — it depends where

At single-machine scale the text-generation picture applies. At datacentre scale AMD’s Instinct accelerators are a serious proposition, and the software support at that tier is considerably better than on consumer Radeon cards — a distinction worth keeping in mind when you read that “AMD is fine for AI now”, because the person saying it may be running MI300s.

What the friction actually costs

The most useful way to think about this is time rather than money.

If you are the sort of person who enjoys getting things working — reading GitHub issues, matching ROCm versions, trying overrides — AMD’s price advantage is real and the friction is a hobby rather than a cost.

If you want to install a runtime and get on with your actual work, the NVIDIA premium buys you that, and for most people it is worth paying. The relevant comparison is not card against card; it is the price difference against several evenings.

There is a second cost that is easy to miss: optionality. With an NVIDIA card, anything new that appears will work. With an AMD card, some fraction of it will not — and you will not know which fraction until you try. That uncertainty has a value, and it is why people who have been burned once tend not to go back.

What about Apple and Intel?

Worth naming, because the question is rarely a straight two-way choice.

Apple silicon has no CUDA either, but its ecosystem position is different: llama.cpp and Apple’s own MLX framework are both mature and genuinely fast on Metal, and unified memory gives capacity no consumer GPU matches — 128 GB on an M4 Max, up to 512 GB on an M3 Ultra. For language models specifically, a Mac is often a better non-NVIDIA choice than a Radeon: more capacity, better-supported tooling, far lower power. For image generation and fine-tuning it has the same CUDA-shaped hole. See Best AI Workstations.

Intel Arc is the outsider. The B580 offers 12 GB at a low price, and Intel’s IPEX-LLM stack plus the Vulkan backend make text generation workable. The ecosystem is thinner than AMD’s, so treat it as a budget experiment rather than a considered platform choice.

The winner, by workload

How to decide, in three questions

1. Will you do anything other than text generation? If yes — image models, fine-tuning, research code — buy NVIDIA and stop reading.
2. Are you comfortable on Linux, and does troubleshooting bother you? If troubleshooting bothers you, buy NVIDIA.
3. Is the price difference material to you? If it is, and the first two answers were “no” and “comfortable”, AMD is a defensible and sensible choice.

The verdict

Buy NVIDIA if you do image generation, you fine-tune, you want to try new techniques as they appear, or you would rather spend your time on your work than on your toolchain. This covers most people.

Buy AMD if your workload is text generation through Ollama or llama.cpp, you are comfortable on Linux, and 24 GB for the price is worth an afternoon of setup. This is a real and defensible position, not a consolation prize.

Consider Apple if you need large models, you are not tied to CUDA, and low power and silence matter.

What we would not recommend is buying AMD hoping the ecosystem catches up during your ownership. ROCm is improving steadily and has been improving steadily for years, while CUDA has not stood still. Buy the card for what works today.

What would change this recommendation

Three things worth watching, since none of the above is permanent:

• A vendor-neutral kernel layer reaching maturity. Triton already compiles to both vendors; if the popular attention and quantisation kernels standardise on it, most of the moat evaporates.
• AMD widening official ROCm support to the cards people actually own, so the override trick stops being necessary.
• Consumer cards inheriting Instinct’s software attention. The datacentre stack is markedly healthier than the desktop one; the gap between them is a choice, not a law.

Continue your research

• Best GPUs for Local AI — both vendors, ranked by capability
• Best GPUs for Ollama — the runtime where AMD is most competitive
• Best AI Workstations — including the Apple option
• RTX 5090 vs RTX 5080 for AI — if you have settled on NVIDIA