Unload Local LLMs After Every Test
A local model test is not over when text appears. I unload the model, read idle VRAM, and record the result before I start another run.
A local model test is not over when the model returns text. It ends after I unload the model and read the idle GPU state.
Short answer: I explicitly unload a local model after each measured test. I record the loaded VRAM, the idle VRAM after unload, and the runtime setting that kept it loaded. This stops one warm model from changing the next test.
Canonical URL: https://bmdpat.com/blog/unload-local-llms-after-tests-2026
Why unload a local model after a test?
Ollama can keep a model in memory after a request. That is useful when the next request uses the same model and settings. It is a problem when I want a clean comparison.
A warm model can make the next run look better than it is. It can hide load time. It can also hold VRAM that another model needs. I cannot call a new result a cold-load result if the previous model still owns memory.
On 2026-08-15, I ran hf.co/unsloth/Qwen3.8-27B-GGUF:Q3_K_S through Ollama on the owned RTX 3070 in FLUARMN. I explicitly stopped the model after the run and recorded the next idle reading in the local receipt.
That reading belongs in the receipt. It shows whether the target runtime released the model state. It does not show that every process on the host stopped. The idle baseline can include the display, drivers, and other work.
What does an unload check prove?
An unload check proves a narrow fact: the target runtime released the model state it held for that request. It does not prove model quality, stable speed, or a useful context size.
I keep those questions separate. The local LLM benchmark decision record keeps task quality, speed, VRAM, power, and failure state in the same decision. The unload result is one field in that record.
The check also helps find a bad comparison. If VRAM stays high after I ask the runtime to stop the model, I do not start another model test and hope for a clear result. I record the state, stop the test sequence, and find the process that still holds memory.
How do I make the check repeatable?
I use the same short sequence for each run.
- Record the idle GPU memory before the model loads.
- Run one named model, runtime, context, and prompt.
- Record the memory during the request and the model's keep-alive setting.
- Stop the model through the target runtime.
- Read GPU memory again and save the post-unload result.
The sequence makes a warm run explicit. If I want warm-request latency, I leave the model in memory and name the result warm. If I want cold-load time, I unload first and record the idle state. I do not mix the two rows.
This matters most on a small card. A model can leave little headroom for a second resident model or a large display workload. The VRAM headroom guide explains why a model that barely fits is not yet a useful route.
Which Ollama setting should I record?
Record keep_alive with the request. It tells the runtime how long it may keep the model ready after the response. A test can use a short keep-alive value, but I still send an explicit unload before I label the next run cold.
Also record the model tag, context size, output cap, and runtime version. A model can use a different amount of VRAM after a context or runtime change. The unload receipt does not replace those fields.
I treat a missing idle reading as missing evidence. The run may still show generation speed. It cannot support a claim about clean release, idle memory, or cold-load conditions.
When should I keep a model loaded?
Keep a model loaded when the next request is part of the same measured warm lane. That can reduce wait time for a real workload. State the lane in the record.
Unload before a new model, quant, context, or runtime test. Unload before a fit decision on a small GPU. Unload before a test that must measure first-request time.
This is a small step, but it keeps local measurements honest. A model that looks fast only because another run left memory warm has not earned a route.
Accompanying prompt
What the prompt does: This prompt makes a local model test receipt that separates loaded, unloaded, warm, and cold GPU states.
Copy/paste this prompt:
Copy-ready prompt
Paste the exact block into your coding agent.
No article chrome, no footnotes, no formatting drift.
This prompt and every other one we publish live in the free prompt library.
Copy the block above.
Get the next measured local AI run in the 5090 Reports email list.
Get the Local AI Field Kit
Four copy-ready tools now, then one evidence-backed Local AI Lab Note on Friday when there is something worth sharing.
Get the requested artifact now, then at most one evidence-backed Local AI Lab Note on Friday when there is something worth sharing. One-click unsubscribe. No sponsored placements. Privacy.
Patrick Hughes
Building BMD HODL — a one-person AI-operated holding company. Nashville, Tennessee. Twenty-Two agents.
More writing
- 5 min
How I Benchmark Local LLMs Before I Trust Them
A local LLM benchmark should end with a decision. I record task quality, tokens per second, VRAM, power, and failure state before I add a model.
- 5 min
How I Test a 30B Local Model Before I Load It
A 30B local model can fit on paper and still fail the job. This test plan checks memory, tool use, speed, and repeatability first.
- 5 min
Why Local LLM Benchmarks Need Power Data
A local LLM benchmark that reports only tokens per second hides the operating cost. My RTX 5090 run shows what power data adds.
- 5 min
VRAM Fit Is Not Runtime Support
A local model can fit in VRAM, download cleanly, and still fail before the first token. My 5090 test adds runtime support as a separate gate.
- 6 min
Search Old Results Before Publishing an LLM Test
An independent QA pass caught my second post about the same Ollama batch sweep. Here is the duplicate check I now run before publishing an LLM result.