Your local LLM benchmark is probably lying to you

Summary: A local model benchmark number can be completely true and still tell you nothing. I run local models on an RTX 5090 for agent work, and the numbers that fooled me were never fake. They were real numbers reported without the context that made them mean something. This post is about reading eval numbers honestly, both the ones you produce and the ones other people hand you.

Most eval advice tells you to build a test rig. Good advice, covered elsewhere. This post assumes you already have numbers. The harder skill is reading them without fooling yourself. Here are the three ways a local model benchmark lies, and how to catch each one.

Lie one: a pass rate with no denominator

One of my recorded rows says a local Llama 3.1 8B run passed 5 of 5 tasks. That is a real row from a real run. It is also close to meaningless on its own.

Five out of five is a five-task suite. A model that fails it is disqualified. A model that passes it has earned one thing: the right to be tested harder. If I published "100 percent pass rate" and left out the number five, that would be marketing wearing a lab coat.

This is the most common way benchmarks mislead, including in public writeups. The percentage is true. The sample behind it is tiny. When you read any pass rate, your first question is not "how high" but "out of how many, and chosen by whom."

The reading habit: never accept a rate without its denominator. A number like 5/5 and a number like 470/500 can both print as "100 percent" and "94 percent," and the second pair tells you something the first cannot.

Lie two: the missing third result

Most people think an eval result has two states. Pass or fail. It has three. Pass, fail, and could-not-run.

My gate scripts check the environment before a run counts. Is the model server up. Is the gateway answering on its port. Is a proper sandbox available. In my setup a strict Docker sandbox is not available, so the gate records that step as attempted, not passed.

That word choice is the whole game. The moment a log records could-not-run as pass, every green result in the file is suspect. The moment it records could-not-run as fail, you throw away a working model for an environment problem. Both mistakes come from collapsing three states into two.

A row from my gate output, unedited in spirit:

gate_00 probe : ollama up, port 4000 ready -> PASS
gate_03 docker : docker not present in this env -> ATTEMPTED (not pass, not fail)
phase1_eval 8b : 5/5 asserts on 5-task suite -> PASS (denominator: 5)

The reading habit: when a benchmark shows only passes and fails, ask what it did with the runs that never happened. If the answer is "there were none," be skeptical. On real hardware there are always some.

Lie three: the machine itself

The benchmarks folder on my rig sits next to a hardware incident report. On July 1 the machine BSOD-looped in the middle of a run. I spent that evening writing a crash triage runbook instead of collecting eval data.

Numbers from an unstable machine are not numbers. If the box crashed twice during a benchmark session, the throughput figure from that session is fiction, and you have no way to know which rows are clean after the fact. On consumer hardware this is not rare. It is Tuesday.

The reading habit: treat the stability of the machine as part of the eval. If you cannot vouch for the hardware during the run, you cannot vouch for the row. Record the machine state alongside the result, or accept that the result is a guess.

How to read any benchmark in one minute

Whether the numbers are yours or someone else's, three questions catch most of the lying:

1. Out of how many, and who picked the tasks?
2. What happened to the runs that could not complete?
3. Was the machine trustworthy while the numbers were collected?

A benchmark that answers all three is worth acting on. One that answers none is a vibe with a decimal point. Most published local model numbers answer none, which is exactly why a small honest CSV beats a big confident chart.

Related reading

• Test Retrieval Before Your Local LLM Writes
• Why I Did Not Promote My Smaller Local Model
• A verifier loop beats a faster local model
• A 7 GB 27B Model Lost to My 17 GB Default

Accompanying prompt

What the prompt does: This prompt makes a model report its own benchmark result with the context that keeps the number honest, instead of a bare pass rate.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local model evaluation reviewer. Context: I am about to record a benchmark result for a local model and I want the row to be honest, not just high. Task: Given a run result, produce one record that includes the pass rate WITH its denominator, an explicit count of tasks that could not run, and a field for machine stability during the run. Output: Return a single table row with these columns: model, tasks_total, tasks_passed, tasks_could_not_run, machine_stable_yes_no, and a one-line honesty note. Constraints: Never report a percentage without the denominator. Never fold could-not-run into pass or fail. Mark any unknown value as unknown rather than guessing.

9 lines/684 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

The same honesty applies to what your agents spend, not just how they score. A model that reads well on a benchmark can still burn through an API budget in one bad loop. That is what AgentGuard handles: runtime budget caps, loop detection, and a kill switch for AI agents. Honest evals tell you the model works. AgentGuard keeps working from turning into expensive.