Local AI sizing desk

What can your GPU run?

Choose your card. Get the models that fit, the quant to use, and a command you can paste.

Evidence and method

58 supported GPU profiles · 41 models · 8 benchmark runs on cards we own

Browse measured fit receipts →Read the sizing method →

Start from

GPU or Apple chip

CUDA · The speed range is directional unless this exact GPU and model pair is marked measured.

Fine-tune results · Chat / Assistant · Quality · 8K

Workload

Optimize for

Context length

5/5 free runs left today

Your result

RTX 3070 8GB

8 models fit at this context.

8 GBavailable memory

Recommended first run

RTX 3070 8GB → Llama 3.2 3B → Q8_0 → Ollama

Scope. The desk calculates memory fit and uses measured or estimated speed. It does not measure answer quality.

Llama 3.2 3B · Q8_0 · Fits fully

Chosen for full residency, headroom, runtime support, and your context. Rankings below still follow your priority.

Speed

~16.7-32.2 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Ollama·detecting platform

Estimated evidence: fit and speed use model weights, VRAM, context, and KV-cache assumptions.

Change runtime or platform

Runtime

Platform

Choose your platform and the desk will assemble the install, launch, and verification steps.

llama.cpp fallback

llama-cli -m llama-3-2-3b-Q8_0.gguf --n-gpu-layers 28 -c 8192

Agent check

Not checked yet for Llama 3.2 3B Q8_0. Checks cover the models this desk recommends most. See the checked models.

Want checks on your own agent work? BMD runs them after each Codex or Claude Code task. Download BMD free.

Save or share this result

Keep this setup

Save it or get fit alerts.

Your hardware and context stay together. Alerts are optional.

Connecting sign-in

Preview share card

Other fits (7)ranked alternatives

gpt-oss 20B

3.6B active / 20B MoE

native MXFP4 reasoning model that stays practical on 16GB+ cards.

Partial offload

Quant

MXFP4 native

VRAM use

13.5 / 8 GB

GPU layers

13/24

Speed

~14.2-27.3 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, MXFP4 native weights, 8 GB VRAM, 0.2 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run gpt-oss:20b

llama.cpp

llama-cli -m gpt-oss-20b-mxfp4.gguf --n-gpu-layers 13 -c 8192

This native MXFP4 native profile is not a verified GGUF filename. Convert it to a compatible GGUF or use Ollama, then replace -m with the resulting path.

Inspect the VRAM math Hugging Face

Qwen3.5 4B

4B dense

fast native multimodal Qwen for small GPUs.

Fits fully

Quant

Q8_0

VRAM use

5.5 / 8 GB

GPU layers

32/32

Speed

~13.8-26.5 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q8_0 weights, 8 GB VRAM, 0.3 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run qwen3.5:4b

llama.cpp

llama-cli -m qwen-3-5-4b-Q8_0.gguf --n-gpu-layers 32 -c 8192

Point -m at your Q8_0 GGUF. All 32 layers fit on the GPU.

Inspect the VRAM math Hugging Face

Qwen3.5 9B

9.7B dense

current multimodal Qwen that fits an 8GB card at 4K context.

Partial offload

Quant

Q8_0

VRAM use

10.5 / 8 GB

GPU layers

24/32

Speed

~7.7-14.9 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q8_0 weights, 8 GB VRAM, 0.3 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run qwen3.5:9b

llama.cpp

llama-cli -m qwen-3-5-9b-Q8_0.gguf --n-gpu-layers 24 -c 8192

Point -m at your Q8_0 GGUF. The flag offloads 24/32 layers; the rest runs on the CPU.

Inspect the VRAM math Compare quant levels Hugging Face

Ministral 3 8B

8B dense

current multimodal edge model with a long context window.

Partial offload

Quant

Q8_0

VRAM use

10.5 / 8 GB

GPU layers

25/34

Speed

~8-15.3 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q8_0 weights, 8 GB VRAM, 1.1 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run ministral-3:8b

llama.cpp

llama-cli -m ministral-3-8b-Q8_0.gguf --n-gpu-layers 25 -c 8192

Point -m at your Q8_0 GGUF. The flag offloads 25/34 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Gemma 4 12B

12B dense

current multimodal quality for 12–16GB cards.

Partial offload

Quant

Q8_0

VRAM use

13 / 8 GB

GPU layers

28/48

Speed

~5.8-11.2 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q8_0 weights, 8 GB VRAM, 0.5 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run gemma4:12b

llama.cpp

llama-cli -m gemma4-12b-Q8_0.gguf --n-gpu-layers 28 -c 8192

Point -m at your Q8_0 GGUF. The flag offloads 28/48 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Gemma 4 E4B

4B dense / 8B total

small current multimodal Gemma for constrained cards.

Partial offload

Quant

Q8_0

VRAM use

8.6 / 8 GB

GPU layers

38/42

Speed

~9.7-18.7 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q8_0 weights, 8 GB VRAM, 0.1 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run gemma4:e4b

llama.cpp

llama-cli -m gemma4-e4b-Q8_0.gguf --n-gpu-layers 38 -c 8192

Point -m at your Q8_0 GGUF. The flag offloads 38/42 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

DeepSeek-R1 Distill Qwen 14B

14.8B dense

mid-size R1 distill for 24GB-class cards.

Partial offload

Quant

Q6_K

VRAM use

13.4 / 8 GB

GPU layers

27/48

Speed

~6.1-11.8 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 8K context from model size, Q6_K weights, 8 GB VRAM, 1.5 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run deepseek-r1:14b

llama.cpp

llama-cli -m deepseek-r1-distill-qwen-14b-Q6_K.gguf --n-gpu-layers 27 -c 8192

Point -m at your Q6_K GGUF. The flag offloads 27/48 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Run it

Buy RTX 3070 Used RTX 3070 Run it on a cloud GPU

Some links here are affiliate links. If you buy or rent through them I may earn a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.

About saved rigs and alerts

Something look wrong?

Tell me what failed. This goes to a private correction queue. It does not change the free answer, subscribe you to marketing email, or put your note into analytics.

Email the full configuration report

Your Llama 3.2 3B on RTX 3070 8GB config report

The full report for this rig: recommended quant, the VRAM tradeoff table across every quantization, and the llama.cpp launch command with the right --n-gpu-layers. Sent once, immediately. No account needed.

Get the requested artifact now, then at most one evidence-backed Local AI Lab Note on Friday when there is something worth sharing. One-click unsubscribe. No sponsored placements. Privacy.

Common questions

What can my GPU run locally?

The Local LLM Sizing Desk at bmdpat.com/desk covers 58 verified GPU profiles. Pick your card and it lists which of 41 local models fit in your VRAM, the GGUF quant to use, and the exact llama.cpp run command to paste.

Are the tokens-per-second numbers measured or estimated?

Both, and the desk labels which is which. It carries 8 benchmark runs measured on cards the site's author owns; everything else is a VRAM-fit calculation, shown as a calculation rather than a measurement.

Which GGUF quant should I pick for my VRAM?

Pick the largest quant that fits your model, context length, and KV cache with VRAM headroom. Q4_K_M is the fit-first choice, Q5 or Q6 is the balance point, and Q8_0 is for quality when memory is available. The desk does this arithmetic per GPU and model.

What if I know the model I want, not the GPU?

Use "I want this model" on the sizing desk. It ranks catalog GPUs that can run that model and marks the smallest card that still fits fully.

Is the Local LLM Sizing Desk free?

Yes: 5 free runs per tool per day with no account. Sign in to save one GPU rig and its workload and context for free, with optional new-model fit email alerts. Saved rig settings are not benchmark or run-receipt history. Changing the saved GPU requires legacy Pro access.