Test Retrieval Before Your Local LLM Writes

I used to judge a local LLM workflow by reading the final answer. That was too late. A polished answer can hide a weak source set, and a bad answer does not tell me whether retrieval or generation failed.

The short answer: I now stop the workflow after retrieval and inspect the source plan before the model writes. The preview shows what context the model would receive, why each source was selected, and whether a private or stale file slipped in. Canonical URL: https://bmdpat.com/blog/local-llm-retrieval-preview-gate-2026

Why test retrieval before generation?

Retrieval and generation are separate failure points. If I run both at once, the final answer mixes them together. I can see that the output is wrong, but I still have to work backward to find the cause.

A retrieval preview makes the boundary visible. It returns the selected file paths, match reasons, scores, and section assignments. It does not ask the local model to write yet.

That one pause changes the debugging job. I can inspect a small source plan instead of reading a full draft and guessing which sentence came from which file. The check is faster because the evidence is compact.

What did the preview expose in my local model lab?

I run a local vault assistant on owned hardware. Its job is to collect private notes for a daily brief, then hand the selected context to a local model. The source files and logs stay on the same machine.

In my July 20, 2026 lab run, preview mode built the brief plan and stopped before generation. It grouped sources into decisions, coding, learning, and growth. It also recorded the ranked files and the terms that caused each match.

That receipt let me inspect the retrieval path without treating fluent prose as proof. I could see when an exact keyword match beat a more useful semantic match. I could also see when an old reflection scored well because it shared many terms with the query.

The important result was not a better paragraph. It was a clear place to tune the system. I could change recency weight, path weight, or section rules, then rerun the preview against the same input.

What should a retrieval preview contain?

I keep the preview narrow. It needs enough data to explain the choice, but not so much that it becomes another report nobody reads.

For each selected source, I want:

• The exact file path or document ID.
• The section that will receive the source.
• The score and the parts that produced it.
• The matched terms or semantic reason.
• The source date and any private-data flag.

I also want the rejected sources near the cutoff. A ranking looks sensible until the best excluded file is clearly better than the weakest included file. That boundary is where tuning work pays off.

This is the same idea behind my local model promotion gate. A component does not earn trust because it runs. It earns trust when an outside check shows that it met the job contract.

How do I keep the preview deterministic?

I freeze the query, source snapshot, retrieval settings, and result limit. Then I save the preview as a run artifact. If I change one scoring rule, I can compare the new ranking with the old one.

I do not ask the local model to explain the ranking. The retriever should expose its own score parts. A model-written explanation can sound right while describing logic the retriever never used.

The preview also needs a fail-closed privacy rule. A blocked path should stop the run before generation, not appear as a warning after private text has entered the prompt. Local inference protects data only when retrieval and logs follow the same boundary.

My local AI research workbench uses the same split. Collection, ranking, generation, and review are separate stages. Each stage leaves an artifact I can inspect.

When should the local model finally write?

Generation starts only after the source plan passes its checks. I look for current sources, useful section coverage, no blocked paths, and no obvious ranking mistake near the cutoff.

Then the model gets a smaller, cleaner context. If the answer still fails, I know retrieval already cleared its gate. The next investigation can focus on the prompt, model, or output verifier.

This does not make every answer correct. It makes failures easier to locate. That is the point. A local AI system gets easier to improve when each stage can fail on its own terms.

Accompanying prompt

What the prompt does: This prompt reviews a retrieval preview and finds source-selection problems before a local LLM generates an answer.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a retrieval QA reviewer for a local LLM workflow. Context: Paste the query, selected sources, rejected sources near the cutoff, score parts, dates, and path rules. Task: 1. Check whether every selected source helps answer the query. 2. Find stale, duplicate, blocked, or weakly matched sources. 3. Compare the weakest selected source with the strongest rejected source. Output: - A table with source, keep or reject, reason, and the scoring rule to change. - A pass or fail verdict for generation. Constraints: - Keep it short. - Use exact numbers and file paths when available. - Do not invent missing measurements.

19 lines/634 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

I publish measured local AI tests, failure notes, and owned-hardware build details in The 5090 Reports. Join the email list to get the next report.