Pin Your Local LLM Context Size Before You Build a Router

I expected the 26B model to be slower than the 8B model. I did not expect one context setting to add more than two minutes before generation started.

Short answer: Pin num_ctx for each loaded local model, then include that value in the model router's cache key. My July 9 RTX 5090 sweep showed that changing Ollama context size forced a full model reload, while the actual generation step still ran above 180 tokens per second. Canonical URL: https://bmdpat.com/blog/pin-local-llm-context-size-before-routing-2026

What did the RTX 5090 sweep measure?

I ran two Q4_K_M models through Ollama 0.31.1 on an RTX 5090. The models were llama3.1:8b and gemma4:26b. Each model handled three jobs: a 256-token generation, a summary with about 4,400 input tokens, and an agent-style coding task capped at 512 output tokens.

The test used temperature 0 and one warmup request per model. Ollama reported token counts and timing. nvidia-smi sampled VRAM and power every half second. The full six-row table is in the July 9 5090 Report.

The smaller model produced 227.79 tokens per second on the coding task. The larger model produced 207.41. That gap matters, but it was not the expensive part.

Why did context size dominate the run?

The test changed num_ctx from 4096 to 8192 for the long summary, then back to 4096 for the coding task. Every change forced Ollama to reload the model.

The 26B model paid 138 to 142 seconds of load time on each measured request. Its generation work took less than 2.5 seconds. A router that ignores context size can therefore spend about a minute loading for every second spent generating.

This also explained an earlier failed test. A five-second request timeout looked reasonable for fast local inference. It could never survive a 26B model load. The timeout was measuring the wrong boundary.

What should be part of a local model route?

Model name is not enough. I now treat the loaded runtime shape as the route:

model + quantization + context size + runtime + GPU

If any item changes, the router should assume a cold path. It can reject the request, send it to an already warm worker, or allow a longer startup budget. What it should not do is call the swap a normal inference request.

For a single-GPU machine, the boring setup is often best. Pick one context size per model. Keep the common model warm. Route unusual long-context work into a separate batch window instead of changing the active model on every request.

Should every local request use the largest context window?

No. More context consumes more memory and creates a larger failure surface. In this sweep, peak VRAM moved from 7,826 MiB for the 8B model to 20,233 MiB for the 26B model. Both fit on the 5090, but fitting is not the same as being cheap to swap.

Most agent steps do not need the largest available window. File classification, command selection, short code edits, and structured extraction usually work with a bounded input. Reserve the larger window for jobs that prove they need it.

That decision should come from task traces, not a default slider. Record input tokens, requested context, load duration, generation duration, and whether the model was already resident. Enough measured runs will expose repeated reloads without promising a fixed observation window.

How should timeouts handle cold and warm paths?

Use two clocks.

The startup clock covers model load and runtime preparation. The generation clock starts after the model is ready. A cold 26B request may deserve several minutes to start while still getting a tight generation limit. A warm 8B request can fail much sooner.

This split also makes alerts useful. "Generation exceeded 20 seconds" points to the model or prompt. "Model load exceeded 150 seconds" points to disk, memory pressure, runtime state, or an accidental context swap. One timeout cannot tell those failures apart.

What is the practical routing rule?

Default to the smallest warm model that passes the task verifier. Escalate only when the output fails, the task needs more context, or the task class has already proved the larger model is worth its load cost.

That rule turns the 5090 from a model carousel into a production worker. Fast tokens help. Stable runtime state helps more.

Accompanying prompt

What the prompt does: This prompt turns local inference logs into a context-aware routing policy with separate cold-start and generation limits.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local LLM runtime engineer reviewing model routing traces. Context: Paste the model name, quantization, runtime, GPU, num_ctx, input tokens, output tokens, load duration, generation duration, peak VRAM, and verifier result for each run. Task: 1. Group runs by model, quantization, runtime, GPU, and num_ctx. 2. Separate cold loads from warm inference. 3. Identify context changes that caused reloads. 4. Recommend one default route and any justified escalation routes. Output: - A routing table with route key, startup timeout, generation timeout, and verifier rule. - A list of requests that paid an avoidable reload cost. - A pass or fail verdict for the current router. Constraints: - Keep it short. - Use exact numbers and file paths when available. - Do not invent missing measurements.

21 lines/811 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

If your local agents need hard runtime budgets around those routes, add AgentGuard before the next unattended run.