---
title: "My local LLM eval hid four token caps"
excerpt: "My 5090-rig eval uses four output caps: 50, 180, 192, and 256 tokens. A score without the cap is not a model result. Record the cap on every row."
author: "Patrick Hughes"
published_at: 2026-08-21T16:24:55.58+00:00
read_time_min: 5
tags: ["local-llm", "ollama", "rtx-5090", "5090-reports", "model-evals"]
canonical: https://bmdpat.com/blog/local-llm-eval-four-token-caps-2026
---

# My local LLM eval hid four token caps

I compared two local eval scores and treated them as one test. They were not. One path stopped at 50 tokens. Another allowed 256.

**Short answer:** My `5090-rig` eval stack uses four output caps: 50, 180, 192, and 256 tokens, under three field names. A score that omits the cap is not a model result. Canonical URL: https://bmdpat.com/blog/local-llm-eval-four-token-caps-2026

![Local LLM output cap comparison that separates hidden limits from measured results](/blog-assets/local-llm-output-token-cap-eval-2026.png)

## Where are the four caps in my eval code?

I listed every default output limit in `5090-rig`. Four values are still in the source and logs.

The router gate posts to `/v1/chat/completions` with `"max_tokens": 50`. That lives in `5090-rig/phase1/gates/gate_01_router.py`. The prompt asks for one Python function. Fifty tokens can be enough for that job.

Phase 1 and phase 2 call Ollama `/api/generate` with `"num_predict": 180`. That default is in `5090-rig/phase1/phase1_eval.py` and again in `5090-rig/phase2/data_flywheel.py`. The [Ollama generate API](https://docs.ollama.com/api/generate) documents `num_predict` as the max tokens to generate after the prompt.

The sweep probe uses a different default. `5090-rig/sweep/probe.py` sets `num_predict=192` on `ollama_generate`. Same API. Different budget. A table that mixes probe rows with phase-1 rows is already mixing caps.

Phase 3 is a third API. I did not find `max_new_tokens` in the Python source. The run log `5090-rig/phase3/SCRATCH/phase3-final-run.log` shows Hugging Face generate used `max_new_tokens=256`. The same lines say `max_length=131072` was also set, and that `max_new_tokens` takes precedence. [Hugging Face text generation](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation) is the independent page for that field.

## Why do four caps hide a model ranking?

The field names do not match. `max_tokens`, `num_predict`, and `max_new_tokens` all mean "stop after this many new tokens." A reader who only sees a score cannot tell which one ran.

The jobs do not match either. A 50-token router check is a short function test. A 180-token Ollama generate can write more. A 256-token Hugging Face generate can write more still. Ranking those scores ranks the budget.

I already covered the receipt side of this in [The 26B Model Hit the Cap. The 8B Finished.](https://bmdpat.com/blog/local-llm-benchmark-stop-reason-2026). That post is about `done_reason`. This post is the other half: find the configured cap in the eval code before you trust the table.

A fast row can still be a truncated row. [Honest local eval numbers](https://bmdpat.com/blog/local-llm-honest-eval-numbers-2026) start with the settings that shaped the answer.

## What should every eval row record?

I now want these fields on every local eval row:

- The cap value (50, 180, 192, or 256 in this stack).
- The field name (`max_tokens`, `num_predict`, or `max_new_tokens`).
- The API path (`/v1/chat/completions`, Ollama `/api/generate`, or Hugging Face generate).
- Tokens produced.
- Stop reason when the runtime gives one.
- Task pass or fail.

If two rows differ on any of those, they are not the same test. Do not average them. Do not pick a winner.

The [speed and quality gate](https://bmdpat.com/blog/local-llm-speed-quality-gate-2026) already treats quality as a release check. The cap belongs in that check. A pass under 256 tokens is not a pass under 50 tokens.

Short caps are useful. The failure is silent mixing. If production uses 50, the table must say 50.

## How do I test a model against its production cap?

I use two runs on the same prompt, model, quant, and runtime path.

Run 1 uses the production cap for that path. For the router gate, that is 50. For phase 1, that is 180.

Run 2 uses a larger cap on the same path. If run 1 fails the task and run 2 passes, the model needed more tokens than production allows. That is a product limit. It is not a mystery quality drop.

Do not raise the eval cap and then ship the tight one. If production will send `num_predict=180`, the keep/reject decision must use 180. The large-cap run is evidence, not a substitute score.

A model that hits the cap can look fast because it never had to finish. Record the cap, the tokens produced, and the task check together.

### Accompanying prompt

**What the prompt does:** Lists every output-token cap in a local eval codebase and flags scores that omit the cap.

Copy/paste this prompt:

```text
Role:
You audit a local LLM eval codebase for output-token caps.

Context:
Paste the eval files, configs, and one result table.

Task:
1. Find every max_tokens, num_predict, max_new_tokens, and n_predict default.
2. Group them by API path and by file.
3. Match each result-table row to the cap that produced it.
4. Flag any score whose cap, stop reason, or token count is missing.

Output:
- A table of file, field name, cap value, and API path.
- Rows that mixed two caps.
- The two-run check: production cap, then a larger cap on the same path.

Constraints:
- Use the exact numbers from the files.
- Do not invent a cap that is not in the source or logs.
- Do not rank models across different caps.
```

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

---

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-eval-four-token-caps-2026
