Preflight Local AI Before You Benchmark a Model

My local AI benchmark failed before it loaded a model. The GPU was ready. The inference runtime was not.

The short answer: Test the toolchain, runtime build, and model load as separate gates. A setup failure is useful evidence, but it is not a model result. Record the exact failed gate and publish no speed number.

Canonical URL: https://bmdpat.com/blog/local-ai-runtime-preflight-before-benchmark-2026

What failed before the benchmark started?

On August 2, 2026, I tried to build WASTE on my RTX 5090 machine. WASTE is a C inference project with a Makefile. I wanted to test its NVMe expert offload path on hardware I own.

The test stopped at the build gate. cc, gcc, clang, and make were absent from the Windows host. WSL showed Ubuntu 22.04 as running, but each command returned Wsl/Service/E_UNEXPECTED with the text Catastrophic failure.

I recorded the failure in my 5090 report. I did not convert a model. I did not load weights. I did not get RAM, NVMe, latency, or token-rate measurements.

That distinction matters. A blank result row can mean that the model failed, the runtime failed, the host failed, or the test never started. Those outcomes need different fixes.

Which gates should run before model load?

I now split the preflight into three gates.

First, check the build tools. Ask the runtime what it needs. Then run version commands for each compiler, build tool, driver, and GPU API. Save stdout, stderr, and exit codes.

Second, build the runtime without a model. A clean build shows the source tree and host toolchain can produce the binary. It does not prove CUDA, model parsing, or inference works.

Third, load the smallest supported model. The goal is one known-good request, not a speed record. Capture the model file, quant, runtime commit, GPU, driver, load result, and stop reason.

Only then should the measured run begin.

Run these gates on every host, not once per project. My WASTE attempt stopped at the first gate, so the 5090 report carries the host state instead of an empty benchmark row. A gate that passes still earns a line in the receipt. That passed line is what lets a later failure point at the model rather than the machine.

Why is VRAM fit not enough?

VRAM answers one question: can the selected weights and runtime state fit on the GPU? It says nothing about whether the host can build or start the runtime.

I made the same point in VRAM Fit Is Not Runtime Support. That post dealt with feature and runtime support. This WASTE test exposed an earlier boundary. The binary did not exist, so VRAM never entered the decision.

This also changes how I read failed benchmark tables. In Why a Failed Local LLM Benchmark Row Still Matters, I kept a failed row because it showed a real limit. A preflight failure needs the same care, but it belongs in a different field.

Use preflight_status, build_status, load_status, and run_status. Do not put every failure under one error label.

What should the receipt contain?

My minimum receipt has four sections.

The host section names the operating system, GPU, driver, and available memory. The toolchain section stores the compiler and build-tool versions. The runtime section stores the repository URL, commit, build command, and build result. The model section stores the file, quant, load result, and measured-run result.

If a gate fails, I stop there. I keep the command, exit code, and error text. I mark all later measurements as not attempted.

This prevents two common errors. The first is to copy a number from another machine because the local run produced none. The second is to call a setup fault a model fault.

I did neither in the WASTE test. The report says that no inference numbers were produced. That is the only result the evidence supports.

When is the benchmark ready to start?

Start it after one small model completes one known-good request through the exact runtime binary that the measured run will use.

Keep that check cheap. It should catch a missing compiler, broken WSL session, bad CUDA link, unsupported model file, or failed load before the long test reserves the machine.

Then run the real workload. Capture input tokens, output tokens, wall time, token rate, VRAM, power, and stop reason. If the workload fails, the receipt can now place the fault after a passed preflight.

The point is not to avoid failed tests. The point is to make each failure name the boundary it reached.

Accompanying prompt

What the prompt does: It turns a local AI test plan into a fail-closed preflight with separate build, load, and measurement results.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are a local AI benchmark preflight reviewer. Context: I will give you a host, inference runtime, model file, and test plan. Task: 1. List the toolchain commands needed before the runtime build. 2. Define separate pass rules for build, model load, and measured run. 3. Stop at the first failed gate and save its command, exit code, and error text. 4. Mark every later metric as not attempted. Output: - A gate table with toolchain, build, load, and run rows. - The exact evidence needed for each pass. - One final status: ready to measure or blocked at a named gate. Constraints: - Do not copy measurements from another machine. - Do not call a setup failure a model failure. - Do not publish speed, memory, or power numbers unless the measured run started. - Keep the runtime commit, model file, quant, GPU, and driver in the receipt.

22 lines/848 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.