Build Local LLM Eval Data From Real Failures

A clean demo tells me a local coding model can pass one task. A saved failure tells me what to test next.

The short answer: I turn each local model attempt into one replayable row that keeps the prompt, generated code, tests, verifier result, model, route, and timestamp together. Passing rows set a baseline. Failed rows become regression cases. Canonical URL: https://bmdpat.com/blog/build-local-llm-eval-data-from-failures-2026

Why are failed local LLM runs useful?

I used to look at a bad answer, change the prompt, and run again. That fixes the moment. It does not create a durable test.

My local coding rig now saves both good and bad attempts. A sample row contains the prompt, generated code, test code, verifier result, pass fraction, source, timestamp, model, generation method, and route. That record lets me replay the same task after I change a model, quant, system prompt, or runtime setting.

The failure is useful because it is specific. One saved add(a, b) attempt returned subtraction. Its two tests both failed. A saved is_prime(n) attempt used the wrong remainder check. One of its two tests passed, while the other failed. Those rows show different failure shapes even though both answers were wrong.

I do not need a model judge to explain either result. The tests provide the boundary.

What should one eval row contain?

I keep the smallest record that can reproduce the decision:

prompt
model_output
test
verifier_result
pass_fraction
model
generation_method
route
source
captured_at

The prompt and output show what the model saw and wrote. The test defines the behavior I expected. The verifier result records what happened. The route tells me whether the request went direct to Ollama or through another layer.

That last field matters. A model name alone does not identify a run. Prompt templates, context settings, and routing code can change the answer before the weights do.

This is the same reason I use a verifier loop before trusting a local coding model. The verifier turns a plausible answer into a pass or fail result. The eval row keeps that result tied to the exact attempt.

How do failed rows become a regression set?

First, I keep the original failure. I do not replace it with the repaired answer.

Second, I inspect the test. A weak test can make a bad row look useful. The is_prime example caught one bad branch but still passed another case. That partial result is a reason to add boundary cases, not a reason to award half credit and move on.

Third, I replay the task against the next candidate route. I compare the result with the saved baseline. If the new route passes, I still keep the old failure because it proves the test can detect the bug.

This gives me a paired dataset: an input, a rejected output, a passing output, and the same executable check. I can use the pair for prompt repair, route selection, or later training work. I do not have to guess which answer was better.

What can this eval set prove?

It can prove that a named route passed a named set of checks at a recorded time. It can show whether a change fixed known failures or brought old ones back.

It cannot prove that the model is good at coding in general. Three toy functions are starter data, not a production benchmark. Repeating the same seeds across iterations also does not create new task coverage.

I keep that limit visible because small evals are easy to oversell. My note on honest local LLM eval numbers explains why the denominator and failed runs belong beside the score.

The next useful step is to add failures from real work. I want tasks with repository context, file edits, tool calls, and tests that reflect the jobs my local agents actually do. A synthetic starter confirms the storage and replay path. Production failures make the set valuable.

How do I keep the loop cheap?

I save the row as soon as verification finishes. I do not wait for a later cleanup pass. The run already has the prompt, output, route, and test result in memory, so that is the cheapest moment to write the receipt.

I also separate collection from promotion. A row may enter the dataset after verification, but a model only earns a production route after it passes the full gate. This lets me collect useful failures without lowering the bar for deployment.

The result is a small loop I can trust: run a real task, verify the output, save the receipt, add the failure to the replay set, then test the next route against it.

Accompanying prompt

What the prompt does: It turns local coding-agent run logs into replayable eval cases without inventing missing tests or measurements.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are building a regression set for a local coding model. Context: Paste one task prompt, model output, test output, model name, runtime route, and capture time. Task: 1. Preserve the original prompt and output. 2. Extract the executable checks and their results. 3. Classify the failure by observed behavior. 4. List missing boundary cases that need new tests. Output: - One replayable eval row with named fields. - A pass or fail verdict tied to the saved tests. - A short list of added test cases, if needed. Constraints: - Do not repair or overwrite the failed output. - Do not invent a test result or runtime setting. - Keep measured facts separate from suggestions. - Mark missing fields as unknown.

23 lines/717 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.