Why I Benchmark Local LLM Input and Output Separately

A local model can print tokens quickly and still make the user wait.

The short answer: I measure model loading, prompt ingestion, output generation, and task correctness as separate phases. One generation rate cannot tell me how a long prompt feels, whether a cold request stalls, or whether the answer works. Canonical URL: https://bmdpat.com/blog/local-llm-input-output-benchmark-2026

What does one local LLM request actually contain?

I split a request into four parts.

First, the runtime loads the model. Second, it reads the prompt. Third, it generates output. Fourth, my verifier checks the result against the job I asked it to do.

Ollama exposes the first three parts in its generate response. Its API returns load_duration, prompt_eval_count, prompt_eval_duration, eval_count, and eval_duration. That gives me raw fields for separate rates instead of one blended stopwatch result.

The fourth part belongs to my test setup. For code, that may be a focused test. For extraction, it may be a schema check. For a summary, it may be a citation check against the supplied text.

What did the RTX 5090 runs show?

My July 2026 RTX 5090 measurements recorded both prompt and output phases for llama3.1:8b and gemma4:26b in Ollama 0.31.1.

The short Llama run read 46 input tokens at 679.4 tokens per second, then generated 256 tokens at 228.87 tokens per second. The long-context run read 4,344 input tokens at 12,108.7 tokens per second, then generated 128 tokens at 206.74 tokens per second.

The Gemma rows made the separation even more useful. The short run read 53 input tokens at 14.5 tokens per second and generated 256 tokens at 198.81 tokens per second. The long-context run read 4,470 input tokens at 6,178.9 tokens per second and generated 128 tokens at 180.18 tokens per second.

Those are measurements from one machine and one sweep. They are not model rankings. The prompt lengths, load behavior, and task shapes differ. That is the point: each row needs its workload and phase labels.

Why is output speed alone a weak deployment test?

Output speed answers one narrow question: how fast did the runtime produce tokens after prompt processing?

It does not answer how long the model took to load. It does not describe a large context. It does not show whether the response stopped at the right place. It does not prove that the answer passed the task.

This matters for local agents because they often carry more input than a chat prompt. A coding worker may receive a file, an error log, tool output, and instructions before it writes one line. A retrieval worker may spend more time reading supplied context than producing the final answer.

My three-workload rate sweep keeps short generation, long-context summarization, and code work separate for the same reason. Workload labels keep a fast row from becoming a false promise.

How do I build a useful local LLM timing table?

I start with the exact route I plan to run: model, quant, runtime version, context setting, batch setting, and output cap.

Then I capture six values from each completed request: load duration, input token count, prompt evaluation duration, output token count, generation duration, and total duration. I calculate input and output rates from their own counts and durations.

I run at least one short prompt and one production-sized prompt. I also record whether the model was already resident. A cold request and a resident request answer different operating questions, so I do not merge them.

Finally, I attach a task result. My guide to honest local LLM eval numbers explains why a denominator and a verifier matter. A fast failed answer is still a failed route.

How do phase metrics set runtime boundaries?

The table should help me choose and bound a route for a named job.

For an interactive tool, total delay may matter most. For a batch summarizer, prompt ingestion and task pass rate may dominate. For a coding agent, generation speed matters only after the code passes its focused check.

Phase measurements also give runtime guardrails a factual base. I can set request, token, and retry limits around observed work instead of guessing. I use AgentGuard when an agent needs those boundaries. It does not make a failed answer correct. The task verifier still owns that decision.

I do not need one winner across every job. I need a measured route whose slow phase is acceptable, whose work stays inside its limits, and whose output clears the verifier. I can test that claim again after the model, runtime, or workload changes.

Accompanying prompt

What the prompt does: It turns raw Ollama timing fields and a task result into a phase-by-phase local LLM benchmark.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are reviewing a local LLM benchmark for a production workload. Context: Paste each run with the model, quant, runtime, workload, residency state, load duration, prompt token count and duration, output token count and duration, total duration, and task-check result. Task: 1. Calculate prompt ingestion and output generation rates separately. 2. Keep cold and resident requests in separate rows. 3. Compare only runs with clearly named workload shapes. 4. Reject any route whose task check failed. Output: - A compact phase-by-phase timing table. - The slow phase for each route. - The best measured route for the named workload. - The next test needed before production use. Constraints: - Keep measured values separate from estimates. - Do not rank models from output speed alone. - Do not hide load time inside a blended average. - Do not treat a fast failed result as a pass.

25 lines/892 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.