My Agents Have to Prove What They Did

I stopped trusting "done."

One of my agents reported three completed actions. Two were false. The model produced a convincing final answer. The repository did not match it.

The tool trace looked busy. The summary sounded sure. Nothing in that chat proved the side effects landed.

So I made a rule for the operation: every material completion needs a check that can fail.

That layer is now open source as showwork. It is on PyPI.

pip install showwork

What does showwork verify?

Observability answers, "What did the agent do?"

showwork answers, "Is the outcome it claimed actually true?"

The first question needs traces. The second needs a falsifiable assertion and a deterministic checker. No LLM judging an LLM.

showwork start --session fix-timeout --agent claude-code

showwork claim --session fix-timeout \
 --claim "bumped billing-sync API timeout to 30s" \
 --type file_contains --path config/api.yaml --pattern "timeout: 30"

showwork finish --session fix-timeout --status ok

If a claim is RED, the clean close is refused. Exit code 2. Fix reality, retract the claim, or finish as blocked.

The command checker has no shell. It only permits a Python script under the project root. It rejects shell metacharacters, path escape, and recursive verification. Other checks cover files, content, path moves, frontmatter, and file counts. Vacuous patterns (regex that match empty strings, always-true counts) are rejected, not blessed.

Why is the ledger append-only?

Agents will be wrong. Operators will also be wrong. Deleting a bad assertion makes the record look cleaner while destroying the most useful evidence.

showwork appends a retraction that names the earlier claim and explains the correction. The original record stays. Failures become training material for the operation.

Does it gate the session?

Yes, through the explicit finish command. Observation and gating stay separate.

showwork finish --session fix-timeout --status ok

The Claude Code Stop-hook adapter records a verdict when a session stops, but always exits 0. It observes. It does not block. The gate is the finish call your prompt contract requires.

What proof came before the package?

I derived a sanitized snapshot from the private ledger that runs the operation:

• 2,158 claims recorded
• 2,152 backed by deterministic checks
• 842 sessions represented
• 152 append-only retractions
• One malformed ledger line surfaced
• Captured verdict: RED, 54 of 60 verified

I kept the RED result. A proof system that only publishes green is not proof.

The derivation publishes aggregates plus a source fingerprint. It does not copy private claim text, paths, strategy, or financial values.

Why publish a specification?

The package can be copied. The record format can outlive it.

spec-v0.1 defines the JSONL schema, checker semantics, anti-vacuous rules, retractions, verdict algebra, session events, and exit-gate contract. Every normative requirement names a behavioral test.

Another team can implement the same evidence format in Go or TypeScript. That is more useful than forcing every agent stack through one Python process.

What is the artifact?

• Installable CLI and Python API: pip install showwork
• Portable ledger specification
• Claude Code Stop-hook adapter
• Genesis receipt where the repository verified its own tests
• Sanitized case study and derivation script
• Marketing demo cut: agents say done, finish is REFUSED, then GREEN after reality matches

Code: github.com/bmdhodl/showwork PyPI: pypi.org/project/showwork

Where runtime limits fit

Verification is one half of trusting an unattended agent. The other half is bounding what the agent can do before it does it.

showwork proves what an agent did after the fact. It does not stop an agent from looping, over-calling a paid API, or draining a token budget on the way to a wrong answer. Those are runtime questions, and they need a runtime limit.

That is the job of AgentGuard: hard caps on tokens, cost, and call rate around an agent run, enforced while the agent is still running. AgentGuard bounds what an agent can spend. showwork proves what an agent actually produced. Run both and an unattended agent has a ceiling on the way in and a receipt on the way out.

Related reading

• Your AI Agent Says "Done." Make It Prove It.
• Pin Your Context Window or Pay the Reload Tax

Accompanying prompt

What the prompt does: turns an agent's vague "done" report into falsifiable claims with deterministic checks you can run yourself.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: senior engineer auditing an AI agent's completion report. Context: an agent reported finishing a task in my repository. I have the report and repo access. Task: 1. Extract every material outcome the report asserts (files changed, configs set, tests passing). 2. For each outcome, write one falsifiable claim with a deterministic check: a file path plus regex, a command with expected exit code, or a path that must exist or be absent. 3. Flag any outcome that cannot be reduced to a deterministic check and name the evidence needed instead. Output: a table with claim, check type, exact check, and what a failure would mean. Constraints: no LLM-judged checks. Every check runs from the repo root and can fail.

8 lines/715 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports