My local LLM eval hid four token caps

I compared two local eval scores and treated them as one test. They were not. One path stopped at 50 tokens. Another allowed 256.

Short answer: My 5090-rig eval stack uses four output caps: 50, 180, 192, and 256 tokens, under three field names. A score that omits the cap is not a model result. Canonical URL: https://bmdpat.com/blog/local-llm-eval-four-token-caps-2026

Where are the four caps in my eval code?

I listed every default output limit in 5090-rig. Four values are still in the source and logs.

The router gate posts to /v1/chat/completions with "max_tokens": 50. That lives in 5090-rig/phase1/gates/gate_01_router.py. The prompt asks for one Python function. Fifty tokens can be enough for that job.

Phase 1 and phase 2 call Ollama /api/generate with "num_predict": 180. That default is in 5090-rig/phase1/phase1_eval.py and again in 5090-rig/phase2/data_flywheel.py. The Ollama generate API documents num_predict as the max tokens to generate after the prompt.

The sweep probe uses a different default. 5090-rig/sweep/probe.py sets num_predict=192 on ollama_generate. Same API. Different budget. A table that mixes probe rows with phase-1 rows is already mixing caps.

Phase 3 is a third API. I did not find max_new_tokens in the Python source. The run log 5090-rig/phase3/SCRATCH/phase3-final-run.log shows Hugging Face generate used max_new_tokens=256. The same lines say max_length=131072 was also set, and that max_new_tokens takes precedence. Hugging Face text generation is the independent page for that field.

Why do four caps hide a model ranking?

The field names do not match. max_tokens, num_predict, and max_new_tokens all mean "stop after this many new tokens." A reader who only sees a score cannot tell which one ran.

The jobs do not match either. A 50-token router check is a short function test. A 180-token Ollama generate can write more. A 256-token Hugging Face generate can write more still. Ranking those scores ranks the budget.

I already covered the receipt side of this in The 26B Model Hit the Cap. The 8B Finished.. That post is about done_reason. This post is the other half: find the configured cap in the eval code before you trust the table.

A fast row can still be a truncated row. Honest local eval numbers start with the settings that shaped the answer.

What should every eval row record?

I now want these fields on every local eval row:

• The cap value (50, 180, 192, or 256 in this stack).
• The field name (max_tokens, num_predict, or max_new_tokens).
• The API path (/v1/chat/completions, Ollama /api/generate, or Hugging Face generate).
• Tokens produced.
• Stop reason when the runtime gives one.
• Task pass or fail.

If two rows differ on any of those, they are not the same test. Do not average them. Do not pick a winner.

The speed and quality gate already treats quality as a release check. The cap belongs in that check. A pass under 256 tokens is not a pass under 50 tokens.

Short caps are useful. The failure is silent mixing. If production uses 50, the table must say 50.

How do I test a model against its production cap?

I use two runs on the same prompt, model, quant, and runtime path.

Run 1 uses the production cap for that path. For the router gate, that is 50. For phase 1, that is 180.

Run 2 uses a larger cap on the same path. If run 1 fails the task and run 2 passes, the model needed more tokens than production allows. That is a product limit. It is not a mystery quality drop.

Do not raise the eval cap and then ship the tight one. If production will send num_predict=180, the keep/reject decision must use 180. The large-cap run is evidence, not a substitute score.

A model that hits the cap can look fast because it never had to finish. Record the cap, the tokens produced, and the task check together.

Accompanying prompt

What the prompt does: Lists every output-token cap in a local eval codebase and flags scores that omit the cap.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You audit a local LLM eval codebase for output-token caps. Context: Paste the eval files, configs, and one result table. Task: 1. Find every max_tokens, num_predict, max_new_tokens, and n_predict default. 2. Group them by API path and by file. 3. Match each result-table row to the cap that produced it. 4. Flag any score whose cap, stop reason, or token count is missing. Output: - A table of file, field name, cap value, and API path. - Rows that mixed two caps. - The two-run check: production cap, then a larger cap on the same path. Constraints: - Use the exact numbers from the files. - Do not invent a cap that is not in the source or logs. - Do not rank models across different caps.

21 lines/701 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports