Log Local LLM Fallbacks Before You Score the Output

My local writing job produced a draft on September 7. That sounds like a working local model until I read which model wrote it.

A fallback can finish generation while the local path remains unavailable. Record the requested model, the actual writer, and the output's review result separately. A returned document proves only that something returned a document.

What happened on my local writing job?

My preferred writer is gemma4:26b through Ollama on the RTX 5090 machine. The September 7 draft records a cold-load refusal: free_ram_gib=15.91 minimum=16.0. That minimum is my host admission policy. It is not a published memory requirement for Gemma or Ollama.

The writer code checks capacity before it sends the generation request. In this case, the saved reason says the local path stopped at that check. There is no local generation speed to report from that attempt.

The fallback produced text. The draft metadata names claude-sonnet-5 as the writer and claude-cli as the runtime. Those are the identifiers recorded by my pipeline, not a comparison of model capability. The append-only draft receipt also records the Sonnet writer.

Independent review then requested changes. It found unsupported claims, public-pricing placeholders, and a claim about an ongoing hardware comparison without a matching receipt. The draft stayed unpublished. These observations come from my September 7 writer metadata, draft receipt, and review record. They are one workflow result, not a benchmark of either model family.

Which result belongs to the local model?

The local result is an admission refusal. The hosted result is a generated draft that did not pass review. Combining them into one successful local run would give Gemma credit for text it never wrote. Combining them into one local quality failure would blame Gemma for text it never wrote.

That is the mistake I want a run record to prevent. Keep the local attempt visible even when another provider completes the request. Attach review findings to the output and the writer that produced it.

I covered the memory boundary in host RAM headroom for local LLMs. This case adds the next question: after a refusal, where did the work go?

What should I save before scoring a fallback?

I would put these fields beside the existing job result: requested model, actual model, runtime, fallback reason, whether local generation started, and final review status. Keep the original attempt and the fallback connected by the same job identifier. Avoid replacing the requested model field with the fallback model and losing the route history.

Save the failure reason as observed. A host-memory refusal, an unavailable endpoint, and a malformed response need different fixes. A single error label such as model failed cannot tell me whether to adjust scheduling, repair a connection, or examine the output parser.

For an Ollama diagnostic snapshot, the official FAQ documents ollama ps. It lists loaded models and shows whether they occupy GPU memory, system memory, or both. That helps inspect residency. It does not prove which model wrote an earlier file. The request and response record must establish that link.

Does fallback completion mean the workflow recovered?

I use separate questions. Did the preferred local path run? Did a permitted alternative return output? Did that output pass the check required for the job? On September 7, the evidence answers no, yes, and no.

Those answers suggest different actions. The first calls for investigating available host capacity. The second confirms that generation continued through the configured alternative. The third keeps the document out of publication until its factual problems are fixed.

My local LLM benchmark decision record puts quality before speed. Fallback adds an attribution rule before either: establish which model produced the result. Otherwise, a quality score or duration can end up attached to the wrong machine.

When should I keep the job local and wait?

If the task requires private hardware throughout, a cloud fallback changes the data boundary. I would keep that job pending or select an allowed local alternative. The existence of a callable hosted model does not establish permission to send the input there.

For tasks that allow a hosted fallback, retain the failed local attempt and label the actual output. Keep each provider's duration and usage separate. Do not include fallback text in a local model's acceptance rate or describe it as measured GPU output.

My rule is simple: score the model that did the work, and record why the preferred model did not. That makes a recovered generation useful evidence without turning it into a false local-model result.

Accompanying prompt

What the prompt does: Audits a local model job for fallback attribution and separates generation from accepted output.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You review local model run records. Context: I will supply a job record, request metadata, response metadata, and the output review result. Task: Identify the requested model and actual writer. State whether local generation started and why fallback occurred. Separate availability, returned output, and review acceptance. Identify missing evidence before assigning a quality or speed result. Output: A short verdict for each attempt and a list of missing fields. Constraints: Do not infer success from a file existing. Do not assign hosted output to the local model. Do not invent measurements or assume cloud use is permitted.

20 lines/638 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Join the email list for measured local AI build notes: https://bmdpat.com/5090-reports