Best GPUs for Local LLMs in 2026: VRAM & Value Guide

TL;DR: Buy the least expensive GPU that fits your exact model, quantization, context window, and runtime with headroom. For the August 2026 US market, a used RTX 3060 12GB is the low-friction budget CUDA pick, a used RTX 3090 is a broadly supported 24GB CUDA choice, AMD offers stack-specific capacity value, Intel is the more experimental capacity path, and a used RTX A6000 is a practical single-card 48GB CUDA tier. An RTX 5090 is a premium 32GB GeForce purchase, not the value winner.

“Fits” means the workload can load under stated assumptions. “Runs comfortably” means it remains GPU-resident with useful context, runtime workspace, and safety headroom. Those are not interchangeable claims.

Exact B60, B70, and R9700 listings checked August 25, 2026. Other new-price observations and used-sale snapshots are dated per row. GPU prices are unusually volatile, so the verdict can change even when the hardware does not.

GPU specification and price-evidence links are non-affiliate.

Table of Contents

• Quick Picks: Best GPUs for Local LLMs
• How Much VRAM Do You Actually Need?
• What Makes a GPU Fast for LLM Inference?
• GPU Comparison: Specifications, Price, and Model Fit
• Recommendations by Budget
• RTX 3090 vs RTX 4090 vs RTX 5090
• NVIDIA vs AMD vs Intel
• Multi-GPU Reality
• GPU PC vs Mac Mini
• Total Build-Cost and Compatibility Checklist
• Buying Used Without Getting Burned
• Five-Minute Software Smoke Test
• Rent or Buy?
• What I Would Buy
• Frequently Asked Questions
• Sources and Methodology

Quick Picks: Best GPUs for Local LLMs

These picks assume inference, one interactive user unless stated otherwise, and verification of the exact backend before purchase.

If your current card runs a representative 8B model, start there. Real use will reveal whether capacity, decode latency, prompt processing, context, concurrency, or software support is the constraint.

How Much VRAM Do You Actually Need?

The useful sizing equation is:

usable VRAM needed = model weights + KV cache + runtime workspace + multimodal or batching overhead + safety headroom

Physical VRAM is not all usable. The display, driver, kernels, temporary buffers, vision encoders, and concurrent requests consume memory too. A 19.8GB file is therefore not automatically comfortable on a 20GB card.

This guide uses three distinct labels:

• Full GPU residency: model layers and intended cache live on the GPU.
• Partial CPU offload: some layers live in system RAM and cross PCIe, changing latency and throughput.
• Multi-GPU sharding: an engine distributes weights and sometimes cache across cards; they do not become one transparent device.

A practical model-fit matrix

This planning matrix assumes common four-bit GGUF artifacts, one sequence, an f16 KV cache, and runtime reserve. The 8K and 32K labels are total context targets.

Official artifacts anchor the boundaries: Qwen3-8B Q4_K_M is 5.03GB, Qwen3-14B is 9GB, Qwen3-30B-A3B is 18.6GB, and dense Qwen3-32B is 19.8GB. llama.cpp lists Llama 3.1 70B Q4_K_M at 43.1GB. All are weight files, not total VRAM requirements.

Why context can move a card into a different tier

KV cache size depends on architecture, tokens, cache dtype, sequence count, and runtime. An f16 estimate is:

layers × KV heads × head dimension × 2 for key and value × 2 bytes × tokens

For Qwen3-32B—64 layers, 8 KV heads, head dimension 128—the estimate is 0.25MiB per token, roughly 2GiB at 8K and 8GiB at 32K, before buffers. Added to its 19.8GB Q4 file, that makes 24GB an 8K-ish low-headroom tier, not a robust 32K tier.

Qwen3-30B-A3B—48 layers and 4 KV heads—uses about 0.094MiB per token: 0.75GiB at 8K or 3GiB at 32K. Its roughly 3B active parameters reduce compute, but all 30B weights still load.

Ollama’s generic defaults are 4K context below 24GiB VRAM, 32K at 24–48GiB, and 256K at 48GiB+. Its 64K recommendation for agents and coding is operational guidance, not a fit guarantee.

What Makes a GPU Fast for LLM Inference?

VRAM comes first: a faster card cannot accelerate layers it cannot hold without offload. Then consider:

1. Bandwidth and kernels. Bandwidth helps decode, but datatypes, quantization kernels, fused operations, and flash attention decide how well an engine uses it.
2. Runtime compatibility. Unsupported model formats, operating systems, or backends erase theoretical hardware advantages.
3. Workload shape. Prompt processing, decode, and concurrent serving stress hardware differently.
4. Power and cooling. Limits, thermals, noise, cable routing, and airflow affect sustained performance.
5. Complete-system cost. A used card that forces a new PSU, case, or motherboard is not a cheap inference box.

This guide does not print a cross-GPU tokens-per-second ranking because no controlled, like-for-like matrix was available for all recommended cards. Vendor AI TOPS, gaming benchmarks, theoretical bandwidth, and screenshots using different models or engines are not substitutes for that test.

GPU Comparison: Specifications, Price, and Model Fit

Board power is a design limit, not measured inference draw; partner limits vary. Bandwidth does not rank real throughput by itself.

For the RTX 5090 Founders Edition, NVIDIA’s current specification table and R2 user guide specify 575W total graphics power, a 1,000W minimum system PSU, and either four PCIe 8-pin leads through the included adapter or one 600W-or-greater PCIe Gen 5 16-pin cable. Partner-card requirements vary; follow the exact board manual.

Dated US price snapshot

MSRP is historical context. New means observed in-stock street price; used means completed-sale evidence where available. Thin samples are labeled.

Tom’s August 10 values are medians of all Newegg listings it found, with sample counts undisclosed. PC Gamer’s August 18 values are single lowest-price observations. They are separate measures, not market ranges. PCPartValue uses completed US eBay sales from sellers with at least 50 positive reviews and exposes percentiles, but low-volume parts remain volatile.

Other cards considered

• RTX 5070 Ti: 16GB, 896 GB/s, and 300W. Its $749 launch MSRP and $1,099.99 August 10 Newegg-listing median charge substantially more without opening a larger model-fit tier.
• RTX 5080: 16GB, 960 GB/s, and 360W. Its $999 launch MSRP and $1,499.99 August 10 Newegg-listing median make it another higher-spec card that remains in the 16GB fit tier.
• Radeon Pro W7900: 48GB ECC, 864 GB/s, and 295W. The August 21 used snapshot was a $2,910 median with a $2,887–2,933 middle 50%, but only two qualified sales; treat it as a stack-specific professional AMD alternative.
• RTX 6000 Ada: 48GB ECC, 960 GB/s, and 300W. The August 21 used snapshot was a $5,580 median with a $5,521–5,640 middle 50% across three qualified sales; it is a newer professional CUDA alternative, not a value pick.

Recommendations by Budget

Treat the bands below as decision lanes, not instructions to spend to the ceiling. Re-check the dated table before buying.

Use the GPU you already own

Run Qwen3-8B Q4_K_M on a supported card you own and record peak VRAM, time to first token, prompt speed, decode speed, and real context. Upgrade only when you can name the missing capability.

Under $350: reuse first, then RTX 3060 or Arc B580

The used RTX 3060 12GB is the easiest low-cost CUDA recommendation: comfortable for 8B and usable for some modest-context 14B Q4 workloads, but not a 30B card. Arc B580 is the Intel SYCL/Vulkan alternative, but its current used snapshot is also thin at three qualified sales; validate its exact OS, engine, and quant format.

$450–650: RX 9060 XT, RTX 4060 Ti, or Arc Pro B60

RX 9060 XT 16GB is the conventional new capacity pick: enough for 14B Q4 with headroom, at 160W, after validating the AMD OS and backend. A used RTX 4060 Ti 16GB is the efficient CUDA fallback, though its two-sale price sample is weak. RTX 5060 Ti 16GB is a skip at the observed premium. Arc Pro B60 offers unusual 24GB capacity, but only for a model, quant, Intel backend, OS, and serving mode proven before purchase.

$750–950: used RX 7900 XTX

RX 7900 XTX is the strongest observed capacity value here for a validated Linux ROCm or llama.cpp buyer. Its 24GB can host Qwen3-30B-A3B Q4 with headroom and makes dense 32B Q4 around 8K plausible. The nine-sale snapshot is still only a dated market observation, and CUDA-dependent applications may not transfer; wait if CUDA is mandatory.

$1,200–1,500: RTX 3090, R9700, or Arc Pro B70

Choose by stack. A healthy RTX 3090 remains a broadly supported 24GB CUDA option, but its 350W limit, age, memory thermals, case space, and return terms belong in total cost. R9700 provides 32GB, with ECC available on Linux, plus meaningful 30B/32B context headroom for a validated Linux ROCm buyer; wait when new pricing exceeds the band. Arc Pro B70 also reaches 32GB, but vLLM XPU’s Python/kernel constraints and beta pipeline parallelism make it a prevalidated Intel deployment only.

$2,200–3,000: RTX 4090 or dual RTX 3090

RTX 4090 is a throughput, gaming, rendering, or mixed-use purchase. Its 24GB does not open a larger resident-model tier than a 3090, and no controlled evidence here supports a speed multiplier. Dual 3090s expose 48GB through sharding but bring a combined 700W board-power ceiling plus slot, lane, chassis, cooling, PSU, and engine complexity.

$3,800–5,000: RTX A6000 or RTX 5090

A used RTX A6000 is the cleaner single-card 48GB CUDA route in this comparison: ECC, two slots, 300W, and a mature stack. A 43.1GB 70B Q4 file can load, but cache and buffer margin is narrow. RTX 5090 is the speed-first 32GB option in the highest-spec current GeForce tier, at poor VRAM value. RTX Pro 6000 Blackwell’s 96GB is a professional ceiling to compare with rental and high-memory Apple systems.

RTX 3090 vs RTX 4090 vs RTX 5090

These three cards answer different questions.

The most important comparison is between the 3090 and 4090: both stop at 24GB. A 4090 can be much more desirable for prompt processing, generation, gaming, or mixed precision work, but it does not make a 43.1GB 70B Q4 artifact fit. Buy it for measured throughput on your stack, not because “newer” implies a larger model.

The 5090’s 32GB does move the capacity boundary. It gives dense 32B Q4 substantially more room for cache and runtime than 24GB. It still does not make a 70B Q4 model a normal fully resident workload. Its current price also puts it near used 48GB workstation territory, so compare the value of speed against the value of an additional 16GB.

No percentage speed claim appears here because the required controls were not available. A defensible test must hold GPU power policy, driver, OS, engine revision, model revision and hash, quantization, context, cache dtype, prompt length, output length, CPU, RAM, and flash-attention settings constant.

NVIDIA vs AMD vs Intel

The brand decision is really a software decision.

NVIDIA: the safe default, not always the value winner

NVIDIA remains lowest-risk when an application requires CUDA or conventional vLLM packaging. Ollama supports modern GeForce cards, llama.cpp ships CUDA, and troubleshooting knowledge is broad. Pay the price premium only when it removes a real compatibility risk.

AMD: supported, with exact conditions

ROCm has current matrices, vLLM lists supported AMD families on Linux, and Ollama and llama.cpp provide AMD paths. Windows and Linux support differ, and third-party apps may contain CUDA-only extensions. Confirm the exact GPU, OS, runtime, model format, quantization, and deployment mode.

Intel: viable for deliberate experimenters

llama.cpp SYCL covers Arc A770 and B580 on Windows/Linux, vLLM has XPU, and Intel publishes LLM Scaler for B60/B70. But vLLM describes basic-inference support, Python 3.12 XPU-kernel requirements, and beta pipeline parallelism. Prove the workload first.

Quantization can reverse the answer: vLLM lists GGUF on NVIDIA/AMD but not Intel GPU, AWQ/GPTQ on NVIDIA/Intel but not AMD, and FP8 W8A8 on selected NVIDIA/AMD architectures but not Intel. Two “4-bit” tests may not be comparable.

Multi-GPU Reality

Two GPUs can make a model runnable through engine-supported sharding, not create one larger, faster GPU. llama.cpp can split layers and KV cache, including unequally; its experimental tensor mode requires flash attention, excludes quantized KV cache, and has architecture/topology constraints. vLLM’s tensor and pipeline parallelism are likewise engine-specific.

Dual RTX 3090s expose 48GB of aggregate device memory to a runtime configured to shard and can make a 43.1GB 70B Q4 artifact runnable, with little cache/runtime margin. The pair also brings a combined 700W board-power ceiling, PCIe traffic, spacing, heat, PSU, lane, and configuration constraints.

One A6000 offers 48GB ECC at 300W without intra-model PCIe sharding. Two can use NVLink, at higher cost. Never add card bandwidth as application bandwidth; engine communication and allocation determine performance.

GPU PC vs Mac Mini

Compare complete systems and usable model memory, not a GPU line item against a finished computer.

A high-memory Mac reaches capacity tiers that are expensive in one GPU, but shares memory with macOS and uses a non-CUDA stack. A workstation may be faster for supported serving, but the card is only part of its cost.

Total Build-Cost and Compatibility Checklist

Before buying, turn the GPU choice into a system bill of materials:

• PSU and cables: follow the exact board manual for power, connector, adapter, lead, bend, and transient guidance.
• Case and cooling: check length, height, slots, connector clearance, radiator interference, and sustained core, hotspot, and memory temperatures.
• Motherboard: for multi-GPU, map physical spacing, electrical lanes, CPU/chipset routing, and bifurcation.
• Host: offload, loading, tokenization, and workers still require CPU, system RAM, and fast storage.
• Software: verify the OS, driver, and engine installation path.
• Operations: measure wall power, heat, noise, warranty, and return protection.

For RTX 5090 Founders Edition, use NVIDIA’s current R2 guide. For partner cards, use the exact AIB manual rather than inferring PSU safety from the FE, a launch article, another region, or an old table.

Buying Used Without Getting Burned

Used hardware can be the right value, but “it displays an image” is not a sufficient test.

Request:

• Photos of model label, serial area, connector, PCB edge, cooler, fans, screws, and ports.
• Screenshots confirming device identity, VRAM, sustained-load temperatures, and VRAM stress stability.
• Disclosure of firmware or power mods, water blocks, repairs, corrosion, smoke, mining, and continuous load.
• Original invoice and written warranty-transfer terms.

Mining history is a risk input, not automatic rejection. Prefer buyer protection and a return window, then repeat stress tests immediately. For RTX 3090, inspect memory thermals, fans, connector, pads, and cooler changes.

Five-Minute Software Smoke Test

Run nvidia-smi, rocminfo, or sycl-ls for the chosen backend and confirm device, driver, and VRAM.

For a simple interactive test with Ollama:

ollama run qwen3:8b

Confirm in runtime output that intended layers are GPU-resident, then repeat at the real context. A short chat proves installation, not long-context fit or serving throughput.

Use Ollama or llama.cpp interactively; use vLLM when its serving, quantization, and hardware path match.

Rent or Buy?

Ownership favors frequent use, local data, and a long holding period. Rental favors irregular experiments, several GPU tiers, or an unproven configuration.

Use a simple comparison:

ownership cost per useful hour = card and system depreciation + electricity + maintenance + your setup time, divided by actual utilized hours

Compare rental GPU-hours, storage, transfer, idle resources, commitments, and engineering. Idle hardware may take years to win; daily private use adds latency and data-locality value.

For continuous serving, measure concurrency, latency percentiles, throughput, uptime, and operations. Do not compare batched cloud throughput with one local stream.

What I Would Buy

These are scenario decisions under the August 25, 2026 evidence—not permanent rankings.

• Existing PC: Use the current supported GPU and upgrade only after measuring a capacity or latency constraint.
• New budget build: Choose RX 9060 XT 16GB after an AMD backend proof, or a returnable used RTX 4060 Ti 16GB when CUDA matters. Skip the observed RTX 5060 Ti premium.
• Best used value: Consider RX 7900 XTX for validated Linux ROCm/llama.cpp. For CUDA, buy a thermally verified RTX 3090 or wait.
• 32GB fit tier: Choose R9700 for validated Linux ROCm at a defensible price, B70 only for a prevalidated Intel/XPU deployment, or RTX 5090 when a measured CUDA workload justifies the top-end GeForce hardware.
• 70B-class requirement: Treat 48GB as a Q4 weights-fit experiment, not comfortable long context. Prefer 64GB+ usable memory or rent before building.
• Quiet or low-power requirement: Compare a high-memory Mac with the complete GPU workstation.

Buy the cheapest card that fits the target model, cache, and runtime on the stack you use. AMD can reward buyers with validated workloads, Intel remains the more experimental path, and used NVIDIA remains easier when software certainty matters.

Frequently Asked Questions

What is the best GPU for running local LLMs?

There is no universal winner. Buy the least expensive card that fits your model, quantization, context, and runtime with headroom. In August 2026, the used RTX 3060 was the budget CUDA pick, the used RTX 3090 a broadly supported 24GB CUDA pick, and the used RTX A6000 a practical single-card 48GB CUDA tier.

How much VRAM do I need for a local LLM?

Budget for weights, KV cache, runtime buffers, display use, batching, and headroom. Roughly, 12GB suits 8B and some modest-context 14B Q4 workloads, 16GB suits 14B Q4 near 8K, 24GB reaches 30B-class Q4, 32GB adds context headroom, and 48GB only begins the 70B-Q4 weights-fit tier.

Can a 24GB GPU run a 70B model?

Not as a normal fully GPU-resident 70B Q4 workload. A representative Llama 3.1 70B Q4_K_M file is 43.1GB before cache and buffers. A 24GB card requires CPU offload or more aggressive quantization, which is a different performance or quality class.

Is an RTX 3090 still worth buying for local AI?

Yes, when you need 24GB of broadly supported CUDA memory and can verify a healthy used card. It is no longer an automatic bargain: the August 21 sample had a $1,369 median across 17 qualified sales. Include its 350W board power, thermals, age, warranty, PSU, and case fit.

Is an RTX 5090 worth it over a 4090 for LLM inference?

Only when the RTX 5090’s 32GB and top-end GeForce specifications justify its price and platform demands for your measured workload. The RTX 4090 stops at 24GB. At August 2026 observed prices, the 5090 was a premium 32GB GeForce purchase, not the VRAM-per-dollar pick.

Can two GPUs combine their VRAM?

A compatible runtime can shard weights and cache across GPUs, so aggregate capacity can make a model runnable. The cards do not become one latency-free memory pool. Results depend on the engine, split mode, topology, model, cache, and quantization; power, cooling, and slot spacing also matter.

Is AMD good for local LLM inference?

Yes, when the exact GPU, OS, runtime version, and quantization are supported. ROCm, Ollama, llama.cpp, and vLLM provide AMD paths, but their matrices differ. Validate the intended model and backend before buying; AMD is supported, but it is not a universal CUDA replacement.

Should I buy a GPU or a Mac Mini?

Choose a discrete GPU for CUDA, throughput, upgradeability, or supported serving stacks. Choose a high-memory Mac for quiet operation, efficiency, compactness, and unified-memory capacity. Compare complete-system cost and usable memory for the target model, not a GPU’s price against a finished Mac.

Sources and Methodology

How the recommendations were built

Capacity uses first-party specifications. Model fit combines official artifacts, calculated KV cache, and runtime headroom. Compatibility uses runtime documentation. MSRP, new street prices, and completed used sales remain separate.

No author-run cross-GPU benchmark is claimed. A future matrix must pin GPU power policy, driver, OS, engine, model revision/hash, quantization, context, cache dtype, flash attention, GPU split, host, and prompt/output lengths. llama-bench should separate pp512 and tg128 over five repetitions at 8K and 32K; server tests should add concurrency 1/4/8, latency percentiles, and aggregate throughput.

Official GPU specifications

Runtime and platform compatibility

Model artifacts and cache inputs

Price evidence and used-market method

Reproducible benchmark reference