Local AI sizing desk

What can your GPU run?

Choose your card. Get the models that fit, the quant to use, and a command you can paste.

Evidence and method

58 supported GPU profiles · 40 models · 8 benchmark runs on cards we own

Browse measured fit receipts →Read the sizing method →

Start from

GPU or Apple chip

CUDA · The speed range is directional unless this exact GPU and model pair is marked measured.

Fine-tune results · Chat / Assistant · Quality · 4K

Workload

Optimize for

Context length

5/5 free runs left today

Fit receipt

RTX 4090 24GB

8 models fit at this context.

24 GBavailable memory

Recommended first run

RTX 4090 24GB → gpt-oss 20B → MXFP4 native → Ollama

Scope. The desk calculates memory fit and uses measured or estimated speed. It does not measure answer quality.

gpt-oss 20B · MXFP4 native · Fits fully

Chosen for full residency, headroom, runtime support, and your context. Rankings below still follow your priority.

Speed

~51.4-98.8 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Ollama·detecting platform

Estimated evidence: fit and speed use model weights, VRAM, context, and KV-cache assumptions.

Change runtime or platform

Runtime

Platform

Choose your platform and the desk will assemble the install, launch, and verification steps.

llama.cpp fallback

llama-cli -m gpt-oss-20b-mxfp4.gguf --n-gpu-layers 24 -c 4096

Keep this setup

Save it or get fit alerts.

Your hardware and context stay together. Alerts are optional.

Connecting sign-in

Preview share card

Other fits (7)ranked alternatives

Qwen3.5 35B-A3B

3B active / 35B MoE

Qwen3.5 MoE quality with a small active path.

Partial offload

Quant

Q8_0

VRAM use

36.6 / 24 GB

GPU layers

26/40

Speed

~29.2-56.2 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 0.1 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run qwen3.5:35b

llama.cpp

llama-cli -m qwen-3-5-35b-a3b-Q8_0.gguf --n-gpu-layers 26 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 26/40 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Gemma 4 26B-A4B

4B active / 26B MoE

multimodal MoE quality with a compact active path.

Partial offload

Quant

Q8_0

VRAM use

27.2 / 24 GB

GPU layers

26/30

Speed

~31.2-60 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 0.2 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run gemma4:26b

llama.cpp

llama-cli -m gemma4-26b-Q8_0.gguf --n-gpu-layers 26 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 26/30 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Qwen3.5 27B

27B dense

current Qwen3.5 quality for 24GB-class cards.

Partial offload

Quant

Q8_0

VRAM use

28.6 / 24 GB

GPU layers

53/64

Speed

~11.4-22 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 0.3 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run qwen3.5:27b

llama.cpp

llama-cli -m qwen-3-5-27b-Q8_0.gguf --n-gpu-layers 53 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 53/64 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Qwen3.8 27B

27.8B dense

new Qwen3.8 multimodal model for 24GB-class cards.

Partial offload

Quant

Q8_0

VRAM use

28.6 / 24 GB

GPU layers

53/64

Speed

~11.4-22 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 0.3 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

llama.cpp

llama-cli -m qwen-3-8-27b-Q8_0.gguf --n-gpu-layers 53 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 53/64 layers; the rest runs on the CPU.

No verified Ollama tag for this model yet. Use the llama.cpp command with a GGUF build.

Inspect the VRAM math Hugging Face

Gemma 4 31B

31B dense

largest current dense Gemma 4 option.

Partial offload

Quant

Q8_0

VRAM use

32.4 / 24 GB

GPU layers

44/60

Speed

~9.9-19 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 0.6 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run gemma4:31b

llama.cpp

llama-cli -m gemma4-31b-Q8_0.gguf --n-gpu-layers 44 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 44/60 layers; the rest runs on the CPU.

Inspect the VRAM math Hugging Face

Qwen3 32B

32.8B dense

current Qwen3 dense quality for mid-to-large local cards.

Partial offload

Quant

Q8_0

VRAM use

34.3 / 24 GB

GPU layers

44/64

Speed

~9.2-17.7 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q8_0 weights, 24 GB VRAM, 1 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run qwen3:32b

llama.cpp

llama-cli -m qwen-3-32b-Q8_0.gguf --n-gpu-layers 44 -c 4096

Point -m at your Q8_0 GGUF. The flag offloads 44/64 layers; the rest runs on the CPU.

Inspect the VRAM math Compare quant levels Hugging Face

Llama 3.3 70B

70B dense

best general chat quality before the models get huge.

Partial offload

Quant

Q4_K_M

VRAM use

42.1 / 24 GB

GPU layers

45/80

Speed

~8.2-15.7 tok/s

estimated

Calculation only. It is not a benchmark on your machine.

Estimated. Speed is estimated for Chat/Assistant at 4K context from model size, Q4_K_M weights, 24 GB VRAM, 1.3 GB KV cache, and verified model attention data. It is not a benchmark on your machine.

Run commands

ollama

ollama run llama3.3:70b

llama.cpp

llama-cli -m llama-3-3-70b-Q4_K_M.gguf --n-gpu-layers 45 -c 4096

Point -m at your Q4_K_M GGUF. The flag offloads 45/80 layers; the rest runs on the CPU.

Inspect the VRAM math Compare quant levels Hugging Face

Run it

Buy RTX 4090 Used RTX 4090 Run it on a cloud GPU

Some links here are affiliate links. If you buy or rent through them I may earn a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.

Something look wrong?

Tell me what failed. This goes to a private correction queue. It does not change the free answer, subscribe you to marketing email, or put your note into analytics.

Your gpt-oss 20B on RTX 4090 24GB config report

The full report for this rig: recommended quant, the VRAM tradeoff table across every quantization, and the llama.cpp launch command with the right --n-gpu-layers. Sent once, immediately. No account needed.

Get the requested artifact now, then at most one evidence-backed Local AI Lab Note on Friday when there is something worth sharing. One-click unsubscribe. No sponsored placements. Privacy.

Common questions

What can my GPU run locally?

The Local LLM Sizing Desk at bmdpat.com/desk covers 58 verified GPU profiles. Pick your card and it lists which of 40 local models fit in your VRAM, the GGUF quant to use, and the exact llama.cpp run command to paste.

Are the tokens-per-second numbers measured or estimated?

Both, and the desk labels which is which. It carries 8 benchmark runs measured on cards the site's author owns; everything else is a VRAM-fit calculation, shown as a calculation rather than a measurement.

Which GGUF quant should I pick for my VRAM?

Pick the largest quant that fits your model, context length, and KV cache with VRAM headroom. Q4_K_M is the fit-first choice, Q5 or Q6 is the balance point, and Q8_0 is for quality when memory is available. The desk does this arithmetic per GPU and model.

What if I know the model I want, not the GPU?

Use "I want this model" on the sizing desk. It ranks catalog GPUs that can run that model and marks the smallest card that still fits fully.

Is the Local LLM Sizing Desk free?

Yes: 5 free runs per tool per day with no account. Sign in to save one GPU rig and its workload and context for free, with optional new-model fit email alerts. Saved rig settings are not benchmark or run-receipt history. Changing the saved GPU requires legacy Pro access.