[bmdpat]
Download BMD free

Agent check · measured 2026-10-12

llama3.1:8b Q4_K_M passed 0 of 10 checked coding tasks.

Codex CLI in OSS mode drove each model through Ollama in a container. Each task's check was written before the run and ran after the agent stopped, with run_check.py and tests/ restored from the fixture first.

A task passes when its check exits 0 and the showwork gate reads GREEN. Hardware: one RTX 5090. Agent: codex-cli 0.162.0. Runtime: ollama 0.31.2. Tokens: 19k in, 3k out, $0 in API fees. AgentGuard prices the same tokens at gpt-5.3-codex list rates at $0.08, before cache discounts (price table 2026-09-26).

  1. Add a command-line option

    Failed

    Task: Add a --limit option to main() in cli.py: an integer, default 10, that caps how many lines main prints.

    Why it failed: The agent changed no files, so the check still failed.

    79 s · 2k in / 333 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-cliflag-20261012T184152Z

  2. Handle missing and bad input

    Failed

    Task: Implement load_port in config_reader.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    6 s · 2k in / 611 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-config-20261012T184152Z

  3. Implement an ordered, case-blind dedupe

    Failed

    Task: Implement dedupe in dedupe.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    7 s · 2k in / 745 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-dedupe-20261012T184152Z

  4. Parse a small input format

    Failed

    Task: Implement parse_duration in duration.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    3 s · 2k in / 326 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-duration-20261012T184152Z

  5. Merge nested settings without side effects

    Failed

    Task: Implement deep_merge in merge.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    4 s · 2k in / 312 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-merge-20261012T184152Z

  6. Fix an off-by-one bug

    Failed

    Task: page() in pager.py returns the wrong items. Page numbers start at 1. Fix the bug.

    Why it failed: The agent changed no files, so the check still failed.

    4 s · 2k in / 109 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-paginate-20261012T184152Z

  7. Rename a function across two files

    Failed

    Task: Rename get_usr to get_user everywhere in this project. Remove the old name.

    Why it failed: The agent changed no files, so the check still failed.

    3 s · 2k in / 256 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-rename-20261012T184152Z

  8. Read a CSV file correctly

    Failed

    Task: Implement total in sheet.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    4 s · 2k in / 302 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-sheet-20261012T184152Z

  9. Implement a function from its docstring

    Failed

    Task: Implement slugify in textutil.py so it does what its docstring says.

    Why it failed: The agent changed no files, so the check still failed.

    3 s · 2k in / 190 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-slugify-20261012T184152Z

  10. Fix shared state between calls

    Failed

    Task: Calls to add_tag in tags.py leak tags into each other. Fix the bug without changing the function signature's meaning.

    Why it failed: The agent changed no files, so the check still failed.

    2 s · 2k in / 102 out · showwork gate RED

    Changed: nothing

    showwork session agent-check-llama3-1-8b-tags-20261012T184152Z

What this does not show

Ten small, single-file Python tasks are not a benchmark. A pass means the tests passed, not that the code is good. Other quants, other runtimes and other agents can score differently.

Want checks on your own agent work? BMD runs them after each Codex or Claude Code task. Download BMD free.