Why I Did Not Promote My Smaller Local Model

A local model can exist, load, and answer prompts without earning a place in a working route. I hit that boundary on July 20, 2026, when a smaller candidate failed to beat the baseline on my RTX 5090 box.

The short answer: I keep the current model until a candidate wins under the same promotion rules. A smaller file or a successful load is not enough. The candidate must improve the work without breaking the checks that make the current route useful. Canonical URL: https://bmdpat.com/blog/local-model-promotion-gate-2026

What happened in the local model lab?

My baseline was llama3.1:8b. The candidate was a local Qwen adapter stored on the same machine. The candidate file existed, so this was not a missing-path or download failure.

I ran the promotion check four times on July 20. Each run returned the same result: candidate_not_promotable. The recorded reason was direct. The candidate did not beat the baseline under the promotion rules.

That is a useful result. The lab protected a working route from a change that had not earned its place. I kept the baseline and saved the failed decision as data for the next training or evaluation pass.

Why is a successful model run not enough?

Loading confirms the runtime can open the model. Generating text confirms the model can produce output. Neither shows the output is better for the job.

This distinction matters more with smaller local models. A smaller model may use less memory or return text faster. It may also miss required details, ignore output structure, or fail a verifier that the baseline passes. Speed only helps when the answer still clears the task gate.

I use the same rule in my local model verifier loop. The model does not grade itself. A separate check decides whether the result meets the contract.

What should a local model promotion gate compare?

Start with the work, not the model card. Give the baseline and candidate the same prompt, context, runtime settings, and output limit. Run the same verifier against both outputs.

Then compare the traits that matter for that route:

• Did the output satisfy every required field?
• Did the verifier pass without a manual repair?
• Did the model stay inside the allowed time and memory boundary?
• Did repeated runs keep the same useful behavior?

The exact weights depend on the job. A drafting model can tolerate a repair step if that repair is measured and cheap. A model that writes commands for an unattended job needs a much stricter correctness gate.

I do not combine those jobs into one vague quality score. I write the pass rule before the run. That keeps a fast but incomplete answer from winning because it feels better in the moment.

Why keep the failed result?

A rejection tells me what the next experiment must change. If I only store winners, I lose the boundary that made the winner meaningful.

For this run, the durable fact is simple: the adapter existed, the promotion code ran, and the candidate did not beat llama3.1:8b. I do not have enough evidence from that result to claim the smaller model is bad in general. I only know it did not win this route under this gate.

That phrasing matters. Local model work is full of broad claims drawn from narrow tests. A failed route decision is not a universal model ranking. It is evidence about one workload, one setup, and one acceptance rule.

The same discipline applies to throughput. My consumer GPU inference write-up explains why fit and speed are only part of production readiness. A model that fits the card still has to finish the real job correctly.

When should the smaller model get another chance?

I would retest after a real change: better training data, a corrected adapter export, a prompt contract that matches the target job, or a runtime setting that fixes a measured failure. I would not rerun the same candidate until random variation hands me a pass.

The next test should also keep the baseline fixed. If the baseline, prompt, and verifier all change at once, the comparison stops answering a clear question.

This is the boring part of local AI that saves the most time. Keep the route that works. Change one input. Record the result. Promote only when the candidate wins the job you actually need done.

Accompanying prompt

What the prompt does: It turns a local model replacement into a fixed comparison with explicit evidence and a clear promotion decision.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local model promotion reviewer. Context: Provide the baseline model, candidate model, hardware, runtime settings, task prompt, saved outputs, and verifier results. Task: Compare the candidate with the baseline on the same workload. Separate measured facts from assumptions. Decide whether the candidate earned promotion. Output: - Baseline result - Candidate result - Failed checks - Performance tradeoffs - Promote, reject, or retest verdict - Next experiment that changes one variable Constraints: - Do not treat a successful load as task success. - Do not let the model grade its own output. - Do not invent missing measurements. - Keep the current route unless the candidate clears the written gate.

22 lines/723 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish measured local AI runs and the failures behind them in The 5090 Reports. Join the email list for the next result.