My agent wrote 126 empty pages and every gate passed

On September 3, 2026, one batch job in my agent fleet wrote 130 knowledge pages in a single commit. 126 of them were the same four sentences with the page title swapped in. That is 96.9 percent of the commit. Every automated check I own passed the batch and reported success.

Short answer: my checks measured structure and never measured substance. A page with correct fields, working links and no orphan references reads as healthy to a schema validator even when the body says nothing. The repair is to score each output against the rest of its own batch, because filler is defined by repetition, not by a word list. Canonical URL: https://bmdpat.com/blog/verify-ai-agent-output-not-just-schema-2026

What did the run actually produce?

The job reads sources and writes one page per source. Here is the shape those 126 pages shared, with only the title changing between them.

## What it is
The source records a dated signal.

## Key claims extracted
- The source reports the event summarized above. Numeric and vendor
  claims remain source-attributed until direct verification.

## Why it matters for us
This evidence informs the current holdings without changing the
quarter plan by itself.

Three metadata fields carried filler too. All 126 pages carry one identical watch item and one identical warning. The pattern field only looks better: 79 pages share one sentence and the other 47 share a second.

Why did every check pass?

I run a linter over this corpus. It is 2,158 lines. It checks for orphan pages, dangling links, missing backlinks, stale pages, contradictions between pages, pinned corrections, and required fields.

It has no body check at all.

That is the whole failure. Each of those checks answers a question about the shape of a page or its position in a graph. None of them asks whether the page says anything. A generator that fills every required field with plausible filler satisfies all of them, and the more disciplined the generator is about schema, the cleaner its empty output looks.

The daily health check made the same mistake in a different way. It counted new pages by matching filenames, found more than one, and reported that the knowledge base grew. It graded volume. 142 pages carry that date.

Why did the audit look clean?

The same night, a wiki lint ran over the corpus. It audited 836 sources, returned a GREEN verdict, and surfaced 2 orphan candidates.

126 of the pages it audited said nothing.

The lint was not broken. It measured what it measures: inbound links, orphan pages, and pages under a link minimum. It also reported 388 sources below that minimum, which is a real backlog and says nothing about whether a page has content.

That is worse than no audit. A GREEN verdict from a link auditor is a statement about links, and it reads as a statement about health. The bad commit landed at 06:17 CT on September 3. The check that catches this landed at 11:21 CT the next day. For those 29 hours my own numbers said the vault was fine.

How do you detect filler in agent output?

The instinct is to write down the filler and grep for it. That fails on the next run, because the filler changes. I have proof. A different batch three weeks earlier wrote 78 pages carrying one identical template, and a grep for a single sentence from it returns all 78. A word list built on the first family sees none of the second.

My own detector is not clean here either. It flags only 3 of those 78, because that template echoes each page's title into the body, and the unique title text pulls the shared-prose score under the bar. A share threshold tuned on one collapse does not transfer to the next one.

What works is comparing each output to its own batch.

1. Take every page one run produced.
2. Break each body into sentences.
3. Count how many of a page's sentences appear in other pages from the same run.
4. Flag a page when most of its prose is shared.

Filler repeats because the generator has nothing per-item to say. Real output does not, because each source is different. The measure needs no keyword list and no threshold typed in by hand, so it does not go stale when the template changes.

Two limits are worth knowing before you build it. A page that echoes its own title into an otherwise identical body scores lower than it should, because the title is unique text. And a batch of one has nothing to compare against, so a run-scoped check must stay silent rather than guess.

What broke upstream?

The batch was 129 items with 2 fetch failures. That is not unusual. Across 57 batches this one is the third largest, and the two above it, 139 and 130 items, did not collapse. Size alone does not explain it, and I do not have the whole explanation.

What I do have is the last step. The run record flagged the engine's leg as out of quota, and the detector that raises that flag matches the string "usage limit reached" against provider output. The leg was marked failed. Its output became the day's record anyway.

That is the ordinary failure and the expensive one. A step marked failed whose output is used regardless costs whatever that step produced. Here it produced 126 pages that say nothing, and the rest of the vault started citing them. Entity, concept and synthesis pages reference them 529 times across 20 pages.

What rule do I use now?

Structure checks tell you an agent produced the right kind of thing. They cannot tell you it did the work. Those are separate questions and they need separate checks.

I hold the same line about models. A candidate that loads and generates text has not earned the job, which is why I did not promote my smaller local model when it ran fine and lost on the numbers. Running is not the same as working, for a model or for an agent.

This is the same argument I make about agent memory in plain files. A contradiction in a file is visible. Empty output that satisfies a schema is not, and it compounds quietly, because the next agent treats it as evidence.

Accompanying prompt

What the prompt does: It checks a batch of agent output for filler by comparing the files to each other rather than to a word list.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: You are auditing a batch of files that one agent run produced. Context: [Paste 5 to 10 outputs from the same run, with their filenames.] Task: 1. List the sentences that appear in more than one file, verbatim or near-verbatim. 2. For each file, give the share of its prose that is shared text. 3. Name any file whose only unique content is its title or an identifier echoed into the body. Output: - A table of filename, shared-prose share, and unique-sentence count. - One line per file: read its source, or filled a template. - Fail the run when most files are above 90 percent shared prose. Constraints: - Keep it short. - Use exact numbers and file paths when available. - Do not invent missing measurements. - Judge whether each file carries information the others do not, not whether the writing is good.

24 lines/827 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

If you run a fleet and want to see what your agents actually did rather than what they reported, that is the thing I am building: https://bmdpat.com/bmd