Why Your Local Model Fits and Still Fails at Long Context

A model loading into VRAM does not prove your long-context workload will fit. I check the memory left after loading, then test the context the job needs. The file size alone cannot answer that question.

Short answer: Local inference needs memory for weights, context state, and runtime buffers. Compare controlled loads at different context sizes to estimate the change in total memory use; inspect runtime cache logs before attributing that change to KV cache. A fit verdict still needs a run at the target context.

What does a context size flag allocate?

The llama.cpp server reference documents -c / --ctx-size, separate key and value cache types, and KV offload settings. Those settings matter alongside the model file. The amount and placement of context memory depend on the model architecture and runtime configuration.

Slot behavior needs a build-specific check. The reference exposes both unified and non-unified KV modes. Unified mode shares a buffer across sequences, so dividing context by slot count is not a universal rule. Save your build identifier, --help output, and startup context settings with each comparison.

Why does the error message point at the wrong pool?

I separate GPU allocations from host RAM before changing settings. Weights, cache, and compute buffers can occupy different places. A host-memory refusal is not evidence that the GPU ran out of VRAM.

On 2026-06-13 a bounded warmup call to Ollama for gemma4:26b returned HTTP 500 in 4.5 seconds: model requires more system memory (15.1 GiB) than is available (15.0 GiB). That error names system memory. It does not measure KV-cache cost or demonstrate that a smaller context would fix the load.

My prior diagnosis blamed a timeout. The recorded refusal changed it: waiting longer would not add free RAM. I keep that failed run in my notes because it stops me treating every local-model failure as a GPU sizing problem.

Host memory can also block a load before any request goes out. My host RAM headroom note records a separate 2026-08-29 preflight that refused at 7.02 GiB free against a 16.0 GiB cold-load floor, so that run has no token rate at all.

How do I measure the cost instead of estimating it?

Take an idle memory sample before anything loads. Load the model at a short context, wait for load to settle, and sample again. Stop the server. Load the same model at a longer context and sample once more. Subtract.

The difference estimates the change in total allocated VRAM between those two settings. It is not an isolated KV-cache measurement. Other buffers, allocation rounding, and background activity can affect the total. Compare the runtime logs for cache allocation and actual context before naming the cause.

Hold everything else fixed while you do it. Same llama.cpp build, same model file, same quant, same offload setting, same batch settings, same cache type, same parallel slots and KV mode. If you change the context and the quant together, the difference you measured belongs to neither one.

What ruins the comparison?

Background GPU use can spoil the comparison. Record idle memory before each load. NVIDIA documents device totals and process reporting in the nvidia-smi reference, but per-process GPU memory is unavailable under Windows WDDM. Keep unavailable fields unknown; use runtime allocation logs alongside device totals.

Changed offload between runs is the sneaky one. If the first load put every layer on the GPU and the second one silently spilled layers to the CPU, your delta mixes two effects. My n_gpu_layers walkthrough covers how to read that boundary. Cache type and build version belong in the same fixed list.

What decision does the number change?

The comparison helps choose the next context setting to test. An idle load is only the first check. Run a representative long prompt and output, record peak GPU and host memory, and check the answer. Neither an idle sample nor a straight-line projection proves the job fits.

When it does not fit, you have three levers and they are not the same. Ask for less context. Try a smaller cache type supported by the build and test answer quality. Or drop to a smaller quant to free room for the cache, which my GGUF quant comparison walks through. Host memory is the fourth constraint and it hides behind all three, which is why the host RAM headroom note exists.

Pick one change that matches the logged failure. If the logs do not identify a limiting pool, keep the diagnosis unknown and collect that evidence first.

Accompanying prompt

What the prompt does: It compares two controlled memory samples, separates total VRAM changes from cache evidence, and identifies the next workload test.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are checking the memory cost of context for one local model. Context: llama.cpp build identifier: [ ] Model file and quant: [ ] Offload setting and batch settings: [ ] Cache type, parallel slots, KV mode, and actual context from logs: [ ] Runtime cache allocation logs for both loads: [ ] Idle GPU and host memory sample: [ ] Sample after load at short context: [ ] Sample after load at long context: [ ] Target context for the real workload: [ ] Task: 1. Confirm both loads used identical settings apart from context size. 2. Subtract the loaded samples and label the result total VRAM delta. 3. Compare cache allocation logs separately; do not infer exact cache cost. 4. Identify missing controls or unavailable measurements. 5. Propose a target-context workload test with peak-memory recording. Output: - Fixed settings list - Total VRAM delta and separate cache-log evidence - Unknowns that prevent a diagnosis - One next test and the evidence needed to call the workload a fit Constraints: - Do not invent measurements or infer a workload fit from idle samples. - Do not assume context is always divided equally between slots. - Mark the result UNKNOWN when any setting differed between the loads. - Name the pool that ran out before suggesting a fix. - Suggest one change at a time so the next measurement stays readable.

33 lines/1341 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get the next measured local-AI field note by email: The 5090 Reports.