Agent Memory: Test the Answer After a Correction
Our agent-memory replay checks words in a file. I show what that proves, what it misses, and how to test whether an expert correction changes an answer.
I can save an expert correction in a file and still get the same wrong answer from an agent. The saved text records the correction. I also need to test whether the agent uses it.
That gap matters in our organizational-memory prototype. A September 5, 2026, code check found that its replay reads a proposed file and looks for expected and forbidden strings. It does not rerun the agent on the task that failed. A green result has a narrow meaning.
If you run agents against real work, you have seen the failure mode. Corrections accumulate in notes. A new session loads some of those notes, then repeats an old mistake. More stored context has not settled whether the answer improved.
What Meta's design adds
Meta Engineering described its organizational second brain on September 2, 2026. It separates knowledge files from procedures, asks experts to resolve ambiguity, and turns feedback into proposed edits. Its evaluation reruns the original scenario and judges the new answer without showing the judge the proposed change. Regression tests check other cases before an expert reviews the fix.
Anthropic's context-engineering article separately describes notes stored outside the context window and loaded in later sessions. That supports the storage pattern. It does not establish that our prototype matches Meta's evaluation results.
What our replay actually checks
Our config/feedback_compiler/replay.py opens the file named by a case. judge_output() then checks the text for required and forbidden substrings. The September 5 inspection confirmed that path in the code.
For example, a case can require the phrase recipe in a concept file. That check can catch a missing file or a missing term. It cannot tell me whether an agent now follows the recipe when choosing a model, handling a failed test, or deciding that evidence is missing.
The compiler has a similarly bounded role. It produces a report of proposals, including recurring patterns and Requests that need a human decision. It does not implement Meta's full process for creating and validating file diffs.
I keep that distinction in the completion claim. A file-content check can pass while behavioral verification remains untested. Calling both results "learned" would hide the work left to do.
Test the answer after the edit
The next check needs the original task, the wrong answer, and the expert's expected correction. Freeze those inputs before proposing an edit. Otherwise the agent can make the test easier while trying to repair its mistake.
Run the original task against the edited knowledge in a fresh session. For a local coding agent, record the GGUF file, quantization, runner version, tools, and task inputs. Keep those fixed while testing the memory edit. Otherwise a model swap and a knowledge change become one result that is hard to explain. Give a separate reviewer the new answer and the expert's acceptance criteria. Keep the edit rationale out of that review.
Then run nearby cases that should still work. A correction that fixes one answer but breaks an unrelated boundary needs another edit. Keep the human review step for changes to money, public identity, or the rules that control what an agent may do.
This is a proposed extension to our prototype. The code check above does not establish that we have built or measured that full loop. It establishes why the current file test cannot close that claim.
Keep storage and proof easy to inspect
The storage choice remains useful. My earlier post, Your AI agent needs a file, explains that choice. This post covers the separate failure mode where a test inspects saved words but never observes a new answer.
The same rule applies before a GGUF quant runs a coding agent: write down the task and acceptance check, run the changed path, and report exactly what the evidence supports.
Accompanying prompt
What the prompt does: It separates checks of a saved correction from tests of the agent's answer and drafts a bounded verification plan.
Copy/paste this prompt:
Copy-ready prompt
Paste the exact block into your coding agent.
No article chrome, no footnotes, no formatting drift.
This prompt and every other one we publish live in the free prompt library.
Copy the block above.
Weekly measured local runs: https://bmdpat.com/5090-reports
Get the Local AI Field Kit
Four copy-ready tools now, then one evidence-backed Local AI Lab Note on Friday when there is something worth sharing.
Try the free agent run check firstGet the requested artifact now, then at most one evidence-backed Local AI Lab Note on Friday when there is something worth sharing. One-click unsubscribe. No sponsored placements. Privacy.
Patrick Hughes
I build BMD and publish measured AI runs, failure reports, and reusable checks. Nashville, Tennessee.
More writing
- 5 min
My local daily brief used 0 model calls
On September 4, 2026 my llama3.1:8b daily brief was ready with 0 model calls and 0 seconds. Preview-only file ranking is not a model run.
- 5 min
3 Tests Before a GGUF Quant Runs Your Coding Agent
A GGUF file fitting in VRAM does not prove it can run your coding agent. Use this local acceptance test for tools, patches, and repeat runs.
- 6 min
Build Local LLM Eval Data From Real Failures
I show how I turn failed local coding runs into replayable eval rows with the prompt, model output, tests, route, and verifier result intact.
- 6 min
Local open-model agents just became a product category
LM Studio shipped Bionic, a full agent built on open models with local code projects, voice, and document work. The interesting part is not the app. It is what.
- 6 min
Your local LLM benchmark is probably lying to you
A local model pass rate can be true and useless at the same time. Here are the three ways local LLM benchmarks mislead you, drawn from real rows on my RTX 5090.