My 8B Model Failed a 400-Word Task

I gave Llama 3.1 8B a simple job: turn a set of verified daily artifacts into a 400-to-700-word devlog. It returned 208 to 299 words across three live runs. A retry collapsed the required structure before the post reached the floor.

The short version: Small local models are good when the task is bounded and the output has a hard verifier. Three failed 8B devlog trials gave me the evidence to keep that model on short scoring work. I assigned long-form synthesis to Gemma 4 26B and gate both task classes with code.

Canonical URL: https://bmdpat.com/blog/local-llm-long-form-model-selection-2026

What failed in the three 8B runs?

The prompt asked for six required sections and a 400-word minimum. Llama 3.1 8B produced 208 to 299 words across three runs on my RTX 5090. It could identify the source facts. It could not keep the full long-form shape intact.

That distinction matters. The result did not make the model useless. The July 10 handoff records the same 8B model as a fit for research triage and scoring, where the answer is short, structured, and easy to check.

The measured numbers in this post come from my July 10 local-inference handoff report. That report records the three 8B attempts, the passing 26B result, and the runtime settings that changed the outcome.

Why did another retry not solve the problem?

Retries help when failure is noisy. They help less when the model repeats the same limit.

The July 10 handoff records a structure collapse on retry. It does not prove why the model collapsed, so I do not guess. The useful fact is simpler: three attempts stayed below the contract or lost the required shape.

This is where a verifier earns its keep. I did not ask the model whether its draft was complete. Code checked the required sections, word range, forbidden terms, and output status. The task assignment changed because the artifact failed, not because the prose felt weak.

The same principle applies to generation after retrieval. I preview the files before local inference, then verify the generated artifact after inference. Both gates isolate the failed stage.

What changed when I routed the task to Gemma 4 26B?

The first larger-model attempts exposed a different failure. Gemma 4 26B is a thinking model in Ollama 0.31. With its thinking mode left on, it consumed the output budget internally and returned an empty response.

Setting think to false fixed that boundary. Increasing num_predict to 2600 kept path-heavy sections from being cut off. The configured model produced a 485-word devlog with all six required sections and passed the deterministic gate.

The lesson was not "use the biggest model." The lesson was "route to the smallest model that can pass the artifact contract, then configure that model for the job." Model size, context, thinking mode, and output budget are one runtime shape.

That runtime shape must stay stable. My RTX 5090 context-size sweep showed that changing num_ctx can force a full reload. A good route includes those settings instead of treating the model name as the whole key.

Why not send every task to the 26B model?

The 8B model is still my choice for bounded work. In a separate July 9 sweep, it peaked at 7,826 MiB of VRAM while the 26B model peaked at 20,233 MiB. A classifier that returns a score and one reason does not need the larger memory footprint of my long-form model.

I separate tasks by output shape. Short extraction, scoring, tagging, and command selection go to the 8B route. Multi-source narrative work with six sections goes to the 26B route. The verifier decides whether either result may advance.

The task classes are assigned before inference. Triage and scoring use the 8B route. The six-section devlog uses the 26B route. Each output carries its own verifier result, so the system does not silently swap models and call both outcomes equivalent.

What should the verifier check before a route advances?

Start with rules that code can judge. Check required sections, minimum and maximum length, forbidden phrases, empty output, truncated output, and whether cited paths came from the approved source manifest. Record the model, quant, context size, thinking mode, output budget, and verifier result with the artifact.

Do not turn the verifier into another model unless the rule truly needs judgment. Word counts and section names belong in code. A human or separate model can review tone later. The first gate should be fast, exact, and repeatable.

The resulting policy is boring: test each task shape, keep the smallest model that passes, pin that assignment, and run the same gate on every output. That is enough to stop a weak result from shipping and a larger model from handling every task by default.

Accompanying prompt

What the prompt does: This prompt turns local-model run records into a verifier-driven routing rule for short and long-form tasks.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local LLM routing engineer. Context: Paste the task, model settings, output artifact, and verifier result for each local run. Task: 1. Group runs by task shape and required output. 2. Identify the smallest model that passed every deterministic rule. 3. Name the exact failure that justifies each larger-model route. Output: - A routing table with task shape, model, runtime settings, and pass rules. - One default route and any justified escalation routes. - A list of checks that should run in code before human review. Constraints: - Keep it short. - Use exact numbers and file paths when available. - Do not invent missing measurements. - Do not recommend a larger model without a recorded verifier failure.

21 lines/730 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish the measured local-model runs, failures, and fixes in The 5090 Reports. Join the list to get the next artifact.