Why I Test 3 Workloads Before Sizing a Local LLM

A model can look fast on one prompt and still be the wrong choice for the work you plan to run.

The short answer: I size local models with at least three workload shapes: short generation, long-context summarization, and a code task. In my July 9, 2026 RTX 5090 sweep, gemma4:26b ranged from 180.18 to 207.41 generated tokens per second across those jobs. The range mattered more than the fastest row. Canonical URL: https://bmdpat.com/blog/local-llm-workload-rates-rtx-5090-sweep-2026

Why is one tokens-per-second number not enough?

Tokens per second describes a run. It does not describe every job the model will see.

A short chat prompt spends little time reading input. A summarization job may read thousands of tokens before it writes one. A coding agent can stop early when it completes the requested output, or burn the full output cap when it misses the stop condition.

Those paths stress different parts of the runtime. If I publish only the best generation row, I hide the slow case that may dominate the real queue.

I made this mistake when I first treated model speed as one field in a routing table. The field looked precise. The route was not. Now I attach the workload, context, input count, output count, runtime version, and completion reason to every rate.

What did the RTX 5090 sweep measure?

I tested two Q4_K_M models through Ollama 0.31.1 on one RTX 5090. The three jobs were short-gen-256, long-context-summarize, and agent-code-task-512.

For gemma4:26b, measured generation was 198.81 tokens per second on the short job, 180.18 on the long-context job, and 207.41 on the code task. Peak loaded memory reached 20,233 MiB.

For llama3.1:8b, the same job shapes measured 228.87, 206.74, and 227.79 tokens per second. Peak loaded memory reached 7,826 MiB.

These are measurements from my machine, driver, model files, and prompts. They are not promises for another rig. They are useful because every row keeps its workload attached.

What does the workload range change?

The gemma4:26b code row was 15.1 percent faster than its long-context row. The llama3.1:8b short row was 10.7 percent faster than its long-context row.

That spread is large enough to change a capacity plan. A worker sized from the top row can miss its queue target when summarization becomes the common job. A user waiting on one response may care more about the slow row than the average.

I do not turn these rates into a universal model ranking. The smaller Llama model was faster and used less memory in this sweep. I still use Gemma for some drafting work because speed and fit are only two gates. The output also has to pass the task check.

This is the same reason I kept num_batch 256 after my Ollama batch sweep. The largest number in one column did not decide the setting.

Why does cold load belong in the sizing decision?

The steady generation rows hide startup time. In the same sweep, Gemma load duration was about 138 to 142 seconds. Llama loaded in about 3.28 to 3.31 seconds.

That does not make Gemma unusable. It changes the deployment. A scheduled job that starts a fresh model process for every task pays that load cost again. A persistent server can keep the model resident and make the steady rate relevant.

Context changes can also trigger reloads. I measured that failure mode separately in my Ollama context reload test. Model, quant, and context therefore belong in the routing key.

For a desktop chat, a cold wait may be annoying. For a five-minute agent schedule, it can consume nearly half of the run window. I record both steady generation and cold load before I call a model deployable.

How do I size a local model from the results?

First, reject any model that does not fit with room for the KV cache and the rest of the process. Peak loaded memory is a gate, not a footnote.

Second, test the job shapes your queue will run. I use a short prompt, a long input, and one tool or code task as the minimum set. Keep context, output caps, and runtime versions in the record.

Third, capture cold load. If it is large, decide whether the service will stay resident. Do not mix a persistent-server rate with a process-per-task design.

Fourth, run a task verifier. A fast answer that breaks the schema, misses the stop rule, or fails the code check is not capacity.

The final sizing record should name the model, quant, GPU, peak memory, rate by workload, cold load, and verifier result. That is enough to make a route you can test again after a model or runtime update.

Accompanying prompt

What the prompt does: It turns local LLM benchmark rows into a workload-aware sizing decision without inventing missing measurements.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local LLM capacity reviewer. Context: Paste benchmark rows for one GPU. Include model, quant, runtime, workload, context, input tokens, output tokens, generation rate, peak memory, load duration, and verifier result when available. Task: 1. Group comparable rows by model and workload. 2. Report the fastest and slowest measured workload for each model. 3. Reject any route that does not fit memory or pass its verifier. Output: - A compact workload-rate table. - A recommended resident model and route. - The exact measurement still needed before deployment. Constraints: - Keep measured values separate from estimates. - Do not compare rows with different job shapes as if they were equal. - Do not invent missing memory, load, or verifier results.

22 lines/770 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.