Best NVIDIA GPUs for Local LLMs: 8GB to 48GB Guide

After building a private AI home lab with Open WebUI, vLLM, and llama.cpp, one question keeps coming up: What GPU should I buy for local AI?

The answer isn’t simply “the newest one.” A GPU that can hold the model you actually want to use can be more valuable than a faster card that runs out of memory. And once you start adding longer conversations, document retrieval, and multiple users, that memory disappears faster than you might expect.

Let’s break it down without turning this into a guide to every CPU, motherboard, and memory kit on the market.

Affiliate disclosure: This post contains Amazon affiliate links. As an Amazon Associate I earn from qualifying purchases. Prices, availability, product variants, and seller terms can change; check the listing before ordering. These are research-based recommendations, not claims that every card was personally benchmarked.

First: What Actually Matters for Local LLMs?

VRAM determines what fits. The model’s weights need memory, and so do its context cache, temporary buffers, and anything else using the GPU. Running an operating-system display or a second AI application can reduce what is available.

Memory bandwidth helps determine speed. Once a model fits, bandwidth and compute affect how quickly the GPU processes prompts and generates output. A more expensive GPU with the same VRAM may respond faster, but it cannot automatically hold a larger model.

Quantization reduces model size. A quantized model stores weights using fewer bits, trading some fidelity for lower memory use. Common GGUF options include Q4_K_M, Q5_K_M, and Q8_0. Q4_K_M is a useful starting point; Q5 or Q8 may preserve more fidelity but need more memory. Quality varies by model and task—don’t assume all Q4 models are identical.

Context uses memory too. The context window is the working space for instructions, conversation history, retrieved documents, the current prompt, and generated tokens. A model’s advertised context limit is not the context your GPU can necessarily afford.

For the recommendations below, assume one GPU, one user, text-only inference, and a Q4_K_M or similar 4-bit model, unless stated otherwise. These are planning estimates—not performance benchmarks or guaranteed maximum context lengths.

Model Sizes: What Fits Is More Than the Download

Here are real examples from Ollama’s model catalog. These are approximate download sizes, not total GPU memory requirements:

Model and quantization	Approximate download size	A useful starting point
Qwen2.5 7B Instruct Q4_K_M	4.7 GB	Try on 8 GB or larger
Qwen2.5 14B Instruct Q4_K_M	9.0 GB	Try on 12 GB with a restrained context; 16 GB provides more headroom
Qwen2.5 32B Instruct Q4_K_M	20 GB	Try on 24 GB with a modest context; 32 GB provides more room
Llama 3.3 70B Instruct Q4_K_M	43 GB	Candidate for 48 GB with a short context and careful memory management

Sizes are rounded and reported in decimal GB by the sources. GPU tools may report GiB, and memory allocation depends on the runtime. The 43 GB Llama download is not a promise that 70B plus a long context will fit on every 48 GB card. As another example, Ollama lists Qwen2.5 72B Q4_K_M at roughly 47 GB—leaving much less room for context and overhead.

Sources: Qwen2.5 model variants and sizes and Llama 3.3 70B Q4_K_M.

A practical way to think about context

4,096 tokens: A reasonable first test for basic chat and short prompts.
8,192 tokens: A useful next step for longer conversations and smaller document excerpts.
16,384–32,768 tokens: More room for coding or retrieval, but noticeably more memory pressure.
64K and beyond: Possible with suitable models and hardware, but plan for the cache and verify it rather than assuming it will fit.

These are starting settings, not guaranteed limits for any card. A 7B model on a 16 GB GPU may handle far more context than a 14B model on the same GPU. Multiple simultaneous requests can require substantially more cache. Weight quantization and KV-cache quantization are separate settings.

To make the tradeoff concrete: Llama 3.3 70B’s published architecture has 80 layers and eight KV heads. With a conventional FP16 key/value cache, a simple calculation puts 8,192 tokens at roughly 2.5 GiB of KV cache for one sequence, before runtime buffers and other overhead. Different cache precision or runtime strategies change that number. The model’s published 128K context capability does not mean a 48 GB GPU can provide it at Q4.

Quick Recommendations by VRAM

GPU memory	Suggested direction	Representative starting point
8 GB	Use what you own or buy inexpensively; RTX 2070 SUPER or RTX 3060 Ti are used options	Qwen2.5 7B Q4_K_M; start at 4K and test 8K
12 GB	RTX 3060 12 GB for budget; RTX 5070 for a newer, faster option	Qwen2.5 14B Q4_K_M; start at 4K and watch headroom
16 GB	RTX 5060 Ti 16 GB; RTX 5070 Ti for more speed; RTX 2000 Ada for compact systems	Qwen2.5 14B Q4_K_M; try 4K–8K first
20 GB	RTX A4500 or RTX 4000 Ada when workstation features, size, or power matter	Qwen2.5 14B Q4_K_M with more headroom; evaluate smaller 20–24B variants individually
24 GB	Used RTX 3090 for capacity; RTX 4090 for higher-end performance	Qwen2.5 32B Q4_K_M; start at 4K and verify fit
32 GB	RTX 5090 for a performance-focused desktop; RTX PRO 4500 Blackwell for a professional workstation	Qwen2.5 32B Q4_K_M; try 4K–8K and increase as memory permits
48 GB	RTX PRO 5000 Blackwell, RTX 6000 Ada, or a used RTX A6000	Llama 3.3 70B Q4_K_M; start at 4K, then test 8K

A 20 GB card is not a dependable fully GPU-resident choice for the specific 20 GB Qwen2.5 32B download in the table above. The model file alone consumes approximately the entire nominal capacity before context and overhead.

The Cards: What I Would Consider

The links below point to the specific examples discussed. GPU specifications are reference or published board values; card dimensions, connectors, power limits, warranty, and included adapters can differ by manufacturer and seller.

8 GB: Gigabyte GeForce RTX 2070 SUPER

Best for: Reusing an existing GPU or building a low-cost starter system.

VRAM: 8 GB GDDR6
Architecture: Turing; PCIe 3.0
Reference GPU power: Approximately 215 W; board designs vary.
Models to try: Qwen2.5 7B Instruct Q4_K_M for general tasks, or Qwen2.5-Coder 7B Q4_K_M for coding. Begin at 4K context and test 8K while watching VRAM.
Power planning: Around 650 W for a typical desktop; check the exact card and connector requirements.

This is a useful way to get started, especially if it is already in your PC. But 8 GB gives you limited headroom for larger models and long contexts. If buying specifically for local AI, compare the price with a 12 GB or 16 GB card before committing.

View the Gigabyte RTX 2070 SUPER on Amazon

12 GB: GeForce RTX 3060 12 GB or RTX 5070

Best for: An economical entry into 14B-class quantized models, or a faster 12 GB desktop.

Make sure an RTX 3060 listing explicitly says 12 GB; there is also an 8 GB version. The RTX 5070 has 12 GB too, with newer hardware and higher memory bandwidth, but the same nominal model-memory capacity.

Model to try: Qwen2.5 14B Instruct Q4_K_M, about 9.0 GB as an Ollama download. Start at 4K context; 8K may work depending on available VRAM and runtime settings. If it does not fit fully, try a smaller model or shorter context.

Power planning: A quality 650 W supply is a reasonable starting point; check the exact card’s recommendation. This is a useful tier, but if the price difference is manageable, I would look seriously at 16 GB.

16 GB: ASUS Dual GeForce RTX 5060 Ti 16 GB

Best for: A sensible starting point for a new local-LLM desktop.

VRAM: 16 GB GDDR7
Architecture and interface: Blackwell, PCIe 5.0 x8 electrical
Reference memory bandwidth: 448 GB/s
Reference GPU power: 180 W
Physical design: ASUS Dual OC model, approximately 2.5 slots; verify dimensions and the card’s exact power connector.
Power planning: 650 W is a sensible purchase for a typical single-GPU system; follow ASUS’s requirement for your configuration.

This is the card I would compare first when someone wants to run models locally without jumping straight to workstation pricing. Qwen2.5 14B Instruct Q4_K_M is a useful general-purpose test; Qwen2.5-Coder 14B Q4_K_M is a coding-focused alternative. Start at 4K or 8K and increase the context after checking real memory use.

The 16 GB model is important: an 8 GB 5060 Ti exists too, and it does not have the same model capacity.

View the ASUS Dual RTX 5060 Ti 16 GB on Amazon

16 GB: ASUS Prime GeForce RTX 5070 Ti

Best for: A faster 16 GB system when the models you want already fit in that capacity.

VRAM: 16 GB GDDR7
Interface: PCIe 5.0 x16
Reference memory bandwidth: 896 GB/s
Reference GPU power: 300 W
Power planning: NVIDIA lists 750 W for its reference configuration; 850 W is a practical new-PSU choice for headroom. Check the ASUS card and system combination.

The 5070 Ti has twice the rated memory bandwidth of the 5060 Ti, but that does not mean exactly twice the tokens per second. It does mean this is a more performance-oriented 16 GB choice. The same 14B Q4 examples apply; the purchase decision is whether speed is worth the extra cost when VRAM capacity remains the same.

View the ASUS Prime RTX 5070 Ti 16 GB on Amazon

16 GB: NVIDIA RTX 2000 Ada Generation

Best for: A small, lower-power workstation where a larger gaming card will not fit.

VRAM: 16 GB GDDR6 with ECC
Memory bandwidth: 224 GB/s
Board power: 70 W; normally powered through the PCIe slot
Form factor: Low-profile, dual-slot workstation card
Power planning: A 450–550 W quality PSU is a conservative starting point for a typical build. Existing approved systems may use less; verify slot power and OEM compatibility.

The same Qwen2.5 14B Q4_K_M example is a reasonable starting target at 4K–8K. Its real advantage is getting 16 GB into a compact, power-constrained machine—not beating a larger 16 GB gaming GPU on raw inference throughput.

View the RTX 2000 Ada 16 GB on Amazon

20 GB: PNY NVIDIA RTX A4500

Best for: A workstation upgrade when you find the card at a price that makes sense.

VRAM: 20 GB GDDR6 with ECC
Architecture: Ampere; PCIe 4.0 x16
Memory bandwidth: 640 GB/s
Board power: 200 W
Power connector: PCIe 8-pin on NVIDIA’s reference design
Power planning: Around 650–750 W for a typical single-GPU workstation, adjusted for the rest of the system.

Qwen2.5 14B Q4_K_M gives this card room for more context than a tightly packed 12 GB setup. Some smaller-file 20–24B models may be candidates, but check the particular quantization and leave room for the cache. A 32B model at Q4 is not a safe default fit on 20 GB.

Compare the price with a used 24 GB RTX 3090. The professional form factor and ECC can be useful, but they are not automatically worth a premium for every home lab.

View the PNY RTX A4500 on Amazon

20 GB: PNY NVIDIA RTX 4000 Ada Generation

Best for: A full-height workstation where a single-slot GPU and moderate power draw matter.

VRAM: 20 GB GDDR6 with ECC
Architecture: Ada Lovelace; PCIe 4.0 x16
Memory bandwidth: 360 GB/s
Board power: 130 W
Form factor: Full-height, single-slot design
Power planning: Around 650 W for a typical desktop; verify the exact card’s 16-pin power connection and approved adapter.

This is the full-height RTX 4000 Ada, not the RTX 4000 SFF Ada. It has the same 20 GB capacity as the A4500 but a different form factor, power profile, and bandwidth. For 14B Q4 models it provides comfortable planning headroom; for larger models, check the actual download and cache requirements.

The SFF model is a different card: it is low-profile, uses 70 W, and has 280 GB/s of memory bandwidth. Model names matter here—NVIDIA’s naming department apparently enjoys keeping us alert.

View the PNY RTX 4000 Ada 20 GB on Amazon

24 GB: GeForce RTX 3090 or RTX 4090

Best for: A serious enthusiast system without moving immediately to professional-GPU pricing.

A used RTX 3090 can be attractive when priced well. It has 24 GB of VRAM and a 350 W reference board rating. The RTX 4090 also has 24 GB, but is a higher-performance 450 W reference card. Exact board requirements vary.

Model to try: Qwen2.5 32B Instruct Q4_K_M. Its approximately 20 GB download makes 24 GB a plausible fit at a modest context, but the remaining space for cache and buffers is limited. Start at 4K and verify full GPU loading before increasing it.

Power planning: I would look at an 850 W quality PSU for a typical 3090 desktop and 1,000 W for a new 4090 build. Follow the GPU vendor’s requirements. For used cards, check the seller, return policy, fan condition, signs of repair, and missing power adapter.

32 GB: ASUS TUF Gaming GeForce RTX 5090

Best for: A high-performance local-AI desktop with 32 GB of VRAM and substantial cooling capacity.

VRAM: 32 GB GDDR7
Architecture: Blackwell; PCIe 5.0 x16
Reference memory bandwidth: 1,792 GB/s
Reference GPU power: 575 W; the ASUS board can have different limits.
Physical design: Large 3.6-slot cooler—measure the case and power-cable clearance.
Power planning: NVIDIA lists 1,000 W for its reference system. Consider 1,200 W for a high-power system and follow the exact ASUS PSU guidance.

A 32B Q4 model such as Qwen2.5 32B is a natural starting point, with more room for context than on a 24 GB card. Start at 4K–8K, then increase while monitoring memory. It is a strong 32 GB card, but a typical 43 GB Llama 3.3 70B Q4 download will not fit entirely in its VRAM.

Watch the total build cost: GPU prices change, and the PSU, case, cooling, and connector requirements can add substantially to the project.

View the ASUS TUF RTX 5090 32 GB on Amazon

32 GB: PNY NVIDIA RTX PRO 4500 Blackwell

Best for: A 32 GB professional workstation where power, size, ECC, and a dual-slot layout matter.

VRAM: 32 GB GDDR7 with ECC
Architecture: Blackwell; PCIe 5.0 x16
Memory bandwidth: 896 GB/s
Board power: 200 W
Form factor: Full-height, dual-slot workstation card
Power planning: Around 650–750 W for a typical single-GPU system; check the exact workstation and approved 16-pin cabling.

This has the same nominal VRAM capacity as the 5090 but a very different profile: lower rated power and bandwidth, a professional form factor, and ECC memory. Qwen2.5 32B Q4_K_M at 4K–8K is a sensible test target. It can be an excellent fit for a workstation, but its value depends heavily on whether those professional features matter to you.

View the PNY RTX PRO 4500 Blackwell 32 GB on Amazon

48 GB: PNY NVIDIA RTX PRO 5000 Blackwell

Best for: A single-card workstation focused on larger local models.

VRAM: 48 GB GDDR7 with ECC
Architecture: Blackwell; PCIe 5.0 x16
Memory bandwidth: 1,344 GB/s
Board power: 300 W
Form factor: Full-height, dual-slot workstation card
Power planning: Around 750–850 W for a typical single-GPU workstation; higher-power CPUs, extra drives, or additional GPUs need their own sizing.

This is where 70B-class Q4 models become practical candidates on one GPU. Llama 3.3 70B Q4_K_M is an example, but begin at 4K context and verify actual GPU residency before trying 8K. A model with a larger Q4 file, a long context, or multiple requests can still exceed available memory.

This is expensive professional hardware, so compare it with a used 48 GB RTX A6000 and other suitable workstation options. The right buy depends on capacity, speed, warranty, available space, and total system cost.

View the PNY RTX PRO 5000 Blackwell 48 GB on Amazon

Before You Buy: PCIe, Power, and Physical Fit

You don’t need the newest CPU just to run a local model on a GPU. A reasonably modern desktop with a supported operating system and an available PCIe x16-size graphics slot is a useful starting point.

PCIe 4.0 or newer is preferred; PCIe 3.0 is generally usable. The cards discussed here can normally negotiate a slower supported PCIe generation. The older connection can matter more when loading models, transferring data between CPU and GPU, or splitting work across cards. When model weights and cache stay on the GPU, the PCIe link may be less important for sustained generation.

That does not guarantee every old office PC will accept every GPU. Check BIOS and vendor compatibility, slot power, available lanes, PSU capacity, and physical space. An x16-size slot can be wired for fewer lanes, and the RTX 5060 Ti itself uses an x8 electrical interface.

A simple PSU shopping guide

GPU category	Practical PSU recommendation for a typical one-GPU desktop
70 W slot-powered workstation GPU	450–550 W is a conservative starting point; approved compact systems may use less
RTX 3060, RTX 5060, RTX 5060 Ti	Around 650 W
RTX 5070 Ti or RTX 3090	Around 850 W for a new PSU purchase
RTX 4090	Around 1,000 W
RTX 5090	At least the vendor’s specified system rating; 1,000–1,200 W is a typical planning range
200–300 W professional workstation GPU	Around 650–850 W, depending on the workstation and other components

These are conservative purchasing guidelines, not universal minimums. A high-power CPU, multiple GPUs, numerous drives, or an overclocked board can require more. Use the higher figure when the exact card or system manufacturer specifies one. GPU watts and PSU watts are not the same thing: the PSU powers the entire computer.

Choose a well-reviewed PSU with proper protections, a suitable warranty, and the correct connectors. ATX 3.1 and a native approved 12V-2×6 cable are useful for a new high-power build. An 80 PLUS Gold rating is a reasonable efficiency target, but it is not by itself proof of PSU quality.

Extra parts you might need

Correct GPU power cables or the manufacturer-approved adapter: Match the exact PSU and GPU; check whether the card includes an adapter.
GPU support bracket: Helpful for large, heavy cards; some cards include one.
Case fans: Add airflow if the enclosure cannot handle sustained GPU load.
Low-profile bracket: Relevant to compact workstation cards, especially used purchases.
Display adapter: Some workstation GPUs use mini DisplayPort; a headless compute card does not need a monitor attached.

A riser is generally unnecessary for a normal desktop installation. If you do need one, use a suitable riser and follow the manufacturer’s guidance on PCIe speed.

A few cable warnings worth repeating

Do not mix modular PSU cables between power supplies unless the PSU manufacturer explicitly confirms compatibility. Connectors can look identical while the PSU-side wiring differs—even between models from the same brand.

Avoid SATA-to-GPU and Molex-to-GPU power adapters as a workaround for an undersized supply. Use the correct approved cable or GPU adapter, follow instructions for the required number of separate PCIe power feeds, and make sure the connector is fully seated. Leave enough room so the side panel does not sharply bend or pull on the cable.

A standard CPU/EPS 8-pin cable and PCIe GPU 8-pin cable are not interchangeable. One noteworthy exception: the professional RTX A6000 uses an EPS-style connector on the GPU; follow NVIDIA’s specific instructions and approved adapter for that card.

What About Multiple GPUs?

Two 24 GB GPUs may allow a supported runtime to split a larger model across both cards, but they do not automatically behave as one simple 48 GB GPU. The runtime must support the configuration, and how memory is distributed matters. Inference can be affected by PCIe transfers, and the build becomes more demanding on space, airflow, motherboard layout, power connectors, and the PSU.

For a first local-AI system, one GPU with enough VRAM is generally the simpler path. Multi-GPU becomes worth considering when you know which larger model or workload you are trying to support.

Don’t Forget Storage

An SSD with enough free space for your model collection is useful: you may keep several quantizations and model families even though only one is loaded at a time. Faster storage can help model load times; it does not replace GPU VRAM.

If you are building a storage-heavy home server, a SAS expander is a separate storage component that can help a compatible SAS controller connect to more drives. It is not a standalone controller, an AI accelerator, or extra VRAM. Most local-LLM desktops do not need one.

Checking What Is Actually Running on the GPU

On a system with NVIDIA drivers, start by checking GPU visibility and memory use:

nvidia-smi

With Ollama, check whether a loaded model is on the GPU:

ollama ps

Ollama reports whether the model is fully GPU-resident or split between CPU and GPU. With llama.cpp or vLLM, review the server’s startup logs as well as nvidia-smi; allocation alone is not proof every model layer is GPU-resident. If you are using containers, GPU passthrough and driver/runtime compatibility must also be correct.

For the linked Qwen examples, these commands select explicit Q4_K_M variants:

ollama run qwen2.5:7b-instruct-q4_K_M
ollama run qwen2.5:14b-instruct-q4_K_M
ollama run qwen2.5:32b-instruct-q4_K_M

Start with a deliberately modest configured context, send a representative prompt, and check memory. Then increase context or try a larger model. Ollama, llama.cpp, and vLLM support different model formats and quantization methods; a GGUF Q4_K_M file is not the same thing as an AWQ or GPTQ model. Check the current runtime documentation and support for your GPU architecture.

My Bottom Line

Already own an 8 GB card? Start there and learn what a smaller model can do.
Buying new on a budget? Look carefully at a 16 GB RTX 5060 Ti.
Want more speed with 16 GB? Consider the RTX 5070 Ti, but remember that model capacity is unchanged.
Want a larger-model sweet spot? Compare a good used 24 GB RTX 3090 with the total cost of newer alternatives.
Want a powerful 32 GB desktop? The RTX 5090 is compelling if your budget, power supply, and case can handle it.
Need a compact or professional workstation? The RTX 2000 Ada, RTX 4000 Ada, and RTX PRO 4500 offer different combinations of capacity, form factor, ECC, and power.
Want a 70B-class model on one GPU? A 48 GB card such as the RTX PRO 5000 is a serious option—just budget for context and test the exact model.

The best local-AI GPU is not necessarily the one with the biggest number on the box. It is the one that runs the model and the context you need, at a speed you can live with, in a system you can power and cool.

Choosing an NVIDIA GPU for Local LLMs: A Practical Guide from 8 GB to 48 GB