3 Tests Before a GGUF Quant Runs Your Coding Agent

A model can fit in VRAM, answer one prompt, and still fail as a coding agent. I do not keep a quant because one chat reply looks good. I keep it after it completes the same small tool loop more than once.

Short answer: Test a local coding model on file reading, a bounded patch, and a real test command. Keep the prompt, tools, context, and pass rules fixed. Pick the smallest quant that passes every gate across repeated runs.

Why is VRAM fit only the first gate?

Fit tells me whether I can start the model. It does not tell me whether the model can follow a tool schema, respect a file boundary, or report a failed command.

I solve fit first. I check the file size, context target, and GPU offload plan. My GGUF quant guide covers that choice, and my --n-gpu-layers guide covers the offload check.

Then I stop changing runtime settings. A moving setup makes the result hard to compare.

This post decides one thing: which quant file I keep for a coding agent. My Q4_K_M vs Q5_K_M guide picks a quant on paper, before the download. This one picks the file by running it.

What three tests do I run?

I use one small repository fixture. It has a short source file, a focused test, and one rule that the model must preserve.

The first test is read. I ask the model to open the source and test files. It must state the rule before it proposes a change. This catches models that guess from filenames or from the task text.

The second test is edit. I ask for one bounded change in one named file. The model must not touch nearby code, add a package, or rewrite the test to make it pass.

The third test is proof. I ask the model to run the focused test and report the command, exit code, and result. A claim without command output does not pass.

How do I keep the comparison fair?

I keep the same model family, prompt, repository state, context limit, tool definitions, and backend version. I change only the quant under test.

I also reset the fixture before each run. Cached files and prior edits can hide a weak result. I use three runs per quant because one clean run can be luck. This is my acceptance rule, not a universal benchmark.

I set temperature to zero when the backend supports that setting. The output can still vary, so I score behavior instead of exact wording.

The stop reason is the part I can show from a measured run. On 2026-07-09 I ran llama3.1:8b and gemma4:26b, both Q4_K_M, on Ollama 0.31.1 with the agent-code-task-512 workload. Both used num_ctx=4096, num_predict=512, and temperature=0.

That pair shows why the proof gate matters. llama3.1:8b stopped on its own at 459 output tokens. gemma4:26b ran to 512 tokens and stopped because it hit the cap, not because it finished. A truncated patch can still read like a complete one, so I check the stop reason before I score the run.

What makes a quant fail?

I reject a run when the model invents file content, sends invalid tool input, edits outside the named file, changes a test without permission, or calls a failed test successful.

I also reject an incomplete run. A useful explanation does not replace the requested patch and command proof. The agent must finish the small loop.

One safety or boundary failure rejects that run. Style differences do not. I care about the action and its proof.

How do I choose between Q4_K_M and Q5_K_M?

I run the same acceptance set against both files. I do not assume the larger file wins. I keep the smaller quant if it passes all three cases in all three runs.

If Q4_K_M misses a tool call and Q5_K_M passes, I keep Q5_K_M when it still fits my target context. If both pass, I keep Q4_K_M and use the saved memory for context or another process.

If neither passes, I stop. More prompt text rarely repairs a model that cannot follow the basic loop. I test another model family or move that task to a stronger endpoint.

What should the test record contain?

I record the model name, quant, backend version, context limit, GPU layer setting, prompt, tool results, changed files, test command, and exit code. I keep failed rows.

The failed rows explain why a model did not enter the coding-agent lane. They also stop me from repeating the same download and setup work later.

This record turns model choice into a local decision. I can rerun it after a backend update, a new quant, or a tool-schema change. The gate stays the same even when the model changes.

Accompanying prompt

What the prompt does: It tests one GGUF quant against a fixed read, edit, and proof loop for a local coding agent.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are testing a local coding model for safe repository work. Context: Repository path: [ ] Source file: [ ] Test file: [ ] Rule to preserve: [ ] Focused test command: [ ] Allowed edit scope: [one file] Task: 1. Read the source file and test file. 2. State the rule that the change must preserve. 3. Make the smallest change inside the allowed scope. 4. Run the focused test command. 5. Report the command, exit code, changed files, and result. Output: - Rule found - Change made - Changed files - Test command - Exit code - PASS or FAIL Constraints: - Do not invent file content. - Do not add a package. - Do not edit a test unless the task permits it. - Treat a nonzero exit code as FAIL. - Stop and report the blocker if a required file or tool is missing.

32 lines/770 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get the next measured local-AI field note by email: The 5090 Reports.