Agent check · measured 2026-10-12
qwen3.5:9b Q4_K_M passed 5 of 10 checked coding tasks.
Codex CLI in OSS mode drove each model through Ollama in a container. Each task's check was written before the run and ran after the agent stopped, with run_check.py and tests/ restored from the fixture first.
A task passes when its check exits 0 and the showwork gate reads GREEN. Hardware: one RTX 5090. Agent: codex-cli 0.162.0. Runtime: ollama 0.31.2. Tokens: 2.1M in, 66k out, $0 in API fees. AgentGuard prices the same tokens at gpt-5.3-codex list rates at $4.55, before cache discounts (price table 2026-09-26).
Add a command-line option
PassedTask: Add a --limit option to main() in cli.py: an integer, default 10, that caps how many lines main prints.
88 s · 270k in / 10k out · showwork gate GREEN
Changed: cli.py
The agent's diff
diff --git a/cli.py b/cli.py index fd0a366..09f5f58 100644 --- a/cli.py +++ b/cli.py @@ -4,7 +4,12 @@ import argparse def main(argv: list | None = None) -> int: parser = argparse.ArgumentParser(prog="lines") parser.add_argument("count", type=int, help="how many numbered lines to print") + parser.add_argument("--limit", type=int, default=10, help="cap on how many lines to print (default 10)") args = parser.parse_args(argv) - for n in range(1, args.count + 1): + + # Cap the number of lines at --limit if it's lower than requested count + effective_count = min(args.count, args.limit) + + for n in range(1, effective_count + 1): print(f"line {n}") return 0showwork session agent-check-qwen3-5-9b-cliflag-20261012T181256Z
Handle missing and bad input
PassedTask: Implement load_port in config_reader.py so it does what its docstring says.
16 s · 36k in / 2k out · showwork gate GREEN
Changed: config_reader.py
The agent's diff
diff --git a/config_reader.py b/config_reader.py index 39bdf10..89cadfb 100644 --- a/config_reader.py +++ b/config_reader.py @@ -1,3 +1,6 @@ +import json + + def load_port(path: str) -> int: """Return the "port" value from the JSON file at `path`. @@ -5,4 +8,25 @@ def load_port(path: str) -> int: ValueError when the port is not an integer from 1 to 65535 (a bool is not an integer here). """ - raise NotImplementedError + # Try loading JSON, return default on any read/parse error + try: + with open(path) as f: + data = json.load(f) + except (FileNotFoundError, OSError): + return 8080 + + if "port" not in data or data["port"] is None: + return 8080 + + port = data["port"] + + # Reject booleans explicitly before checking type + if isinstance(port, bool) or not isinstance(port, int): + raise ValueError(f"Port must be an integer from 1 to 65535 (got {type(port).__name__})") + + if port < 1 or port > 65535: + raise ValueError("Port out of range") + + return port + +showwork session agent-check-qwen3-5-9b-config-20261012T181256Z
Implement an ordered, case-blind dedupe
PassedTask: Implement dedupe in dedupe.py so it does what its docstring says.
24 s · 49k in / 3k out · showwork gate GREEN
Changed: dedupe.py
The agent's diff
diff --git a/dedupe.py b/dedupe.py index f2ae1ed..5c63c41 100644 --- a/dedupe.py +++ b/dedupe.py @@ -3,4 +3,13 @@ def dedupe(names: list) -> list: Keep the spelling of the first time a name appears. Do not change the input. """ - raise NotImplementedError + seen = set() # lowercase versions we've already encountered + result = [] # keep original spellings from list + + for name in names: + key = name.lower() + if key not in seen: + seen.add(key) + result.append(name) + + return resultshowwork session agent-check-qwen3-5-9b-dedupe-20261012T181256Z
Parse a small input format
FailedTask: Implement parse_duration in duration.py so it does what its docstring says.
Why it failed: Some tests failed (pytest exit 1).
62 s · 162k in / 8k out · showwork gate RED
Changed: duration.py
The agent's diff
diff --git a/duration.py b/duration.py index 0450915..c8d4668 100644 --- a/duration.py +++ b/duration.py @@ -1,7 +1,36 @@ +import re + + def parse_duration(text: str) -> int: """Return the number of seconds in a duration like "1h30m", "45s" or "2h5m10s". Units are h, m and s, in that order, each at most once. Raise ValueError for anything else, including an empty string. """ - raise NotImplementedError + if not text: + raise ValueError("Empty duration") + + # Match numeric value + unit (h/m/s), allowing multiple components + pattern = re.compile(r'(\d+)([hms])') + matches = list(pattern.finditer(text)) + + if len(matches) != len(re.findall('[hms]', text)): # all h,m,s must be matched with digits before them + raise ValueError("Invalid format: digit-less unit in string") + + seconds = 0 + + for i, match in enumerate(matches): + count_str, unit = match.groups() + count = int(count_str) + + if unit == 'h': + seconds += count * 3600 + elif unit == 'm': + seconds += count * 60 + elif unit == 's': + # Check that s only appears after all h and m components in the string + pass + + return int(seconds) + +showwork session agent-check-qwen3-5-9b-duration-20261012T181256Z
Merge nested settings without side effects
FailedTask: Implement deep_merge in merge.py so it does what its docstring says.
Why it failed: The tests could not run (pytest exit 2: an import or syntax error).
89 s · 279k in / 11k out · showwork gate RED
Changed: merge.py
The agent's diff
diff --git a/merge.py b/merge.py index f224354..481255d 100644 --- a/merge.py +++ b/merge.py @@ -1,7 +1,22 @@ -def deep_merge(base: dict, override: dict) -> dict: - """Return a new dict: `base` with `override` merged in. +def deep_merge(base, override): + + """Return a new dict: base with override merged in. - When both values for a key are dicts, merge them the same way. Otherwise the - value from `override` wins. Never change `base` or `override`. - """ - raise NotImplementedError +When both values for a key are dicts, merge them the same way. Otherwise +the value from override wins. Never change base or override.""" + + def _merge(b_dct, o_dct): + # Merge b_dct and o_dct into result dict + if not (b_dct | o_dct): return {} + + all_keys = set(list(o_dct.keys()) + list(b_dct.keys())) # Union + merged_at_this_level = {} + +_EOF_EOF_ echo ' for key in sorted(all_keys)' >> /work/merge.py && cat << '_EOF_EOF_' >> /work/merge.py +' + b_val = b_dct.get(key) # May be None + +EOF_EOF + +echo "Current state of merge.py after partial writes:" +cat /work/merge.py 2>/dev/null | tail -15showwork session agent-check-qwen3-5-9b-merge-20261012T181256Z
Fix an off-by-one bug
PassedTask: page() in pager.py returns the wrong items. Page numbers start at 1. Fix the bug.
24 s · 51k in / 3k out · showwork gate GREEN
Changed: pager.py
The agent's diff
diff --git a/pager.py b/pager.py index 0bd5957..708ec87 100644 --- a/pager.py +++ b/pager.py @@ -2,5 +2,5 @@ def page(items: list, number: int, size: int) -> list: """Return page `number` (starting at 1) of `items`, `size` items per page.""" if number < 1 or size < 1: raise ValueError("number and size must be at least 1") - start = number * size + start = (number - 1) * size return items[start:start + size]showwork session agent-check-qwen3-5-9b-paginate-20261012T181256Z
Rename a function across two files
FailedTask: Rename get_usr to get_user everywhere in this project. Remove the old name.
Why it failed: The tests passed, but showwork refused the close: the agent changed venv/bin/python: path escapes project root: venv/bin/python (and 3 more).
95 s · 414k in / 9k out · showwork gate RED
Changed: report.py, users.py, venv/bin/python, venv/bin/python3, venv/bin/python3.11, venv/lib/python3.11/site-packages/_distutils_hack/__init__.py, venv/lib/python3.11/site-packages/_distutils_hack/override.py, venv/lib/python3.11/site-packages/distutils-precedence.pth, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/LICENSE.txt, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/METADATA, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/RECORD, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/WHEEL, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/entry_points.txt, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/top_level.txt, venv/lib/python3.11/site-packages/pip/__init__.py and 744 more files
The agent's diff
diff --git a/report.py b/report.py index 8062d48..874fd6e 100644 --- a/report.py +++ b/report.py @@ -1,5 +1,5 @@ -from users import get_usr +from users import get_user def summary(ids: list) -> str: - return ", ".join(get_usr(i) for i in ids) + return ", ".join(get_user(i) for i in ids) diff --git a/users.py b/users.py index b2368c1..670a56f 100644 --- a/users.py +++ b/users.py @@ -1,5 +1,5 @@ _USERS = {1: "ada", 2: "linus"} -def get_usr(user_id: int) -> str: +def get_user(user_id: int) -> str: return _USERS[user_id]showwork session agent-check-qwen3-5-9b-rename-20261012T181256Z
Read a CSV file correctly
FailedTask: Implement total in sheet.py so it does what its docstring says.
Why it failed: The tests could not run (pytest exit 2: an import or syntax error).
95 s · 421k in / 11k out · showwork gate RED
Changed: sheet.py
The agent's diff
diff --git a/sheet.py b/sheet.py index 300f952..0c5e387 100644 --- a/sheet.py +++ b/sheet.py @@ -1,8 +1,3 @@ -def total(path: str, column: str) -> float: - """Return the sum of `column` in the CSV file at `path`. +import csv - The first row is the header. Skip blank cells. Values may be quoted and may - contain commas as thousands separators, like "1,250.50". Raise KeyError when - the column does not exist. - """ - raise NotImplementedError +def total(path, column):showwork session agent-check-qwen3-5-9b-sheet-20261012T181256Z
Implement a function from its docstring
PassedTask: Implement slugify in textutil.py so it does what its docstring says.
60 s · 231k in / 7k out · showwork gate GREEN
Changed: textutil.py
The agent's diff
diff --git a/textutil.py b/textutil.py index f8bf9f4..d294734 100644 --- a/textutil.py +++ b/textutil.py @@ -1,3 +1,5 @@ +import re + def slugify(text: str) -> str: """Return a URL slug: lowercase, runs of a-z and 0-9 joined by single hyphens, nothing else.""" - raise NotImplementedError + return "-".join(re.findall(r"[a-z0-9]+", text.lower()))showwork session agent-check-qwen3-5-9b-slugify-20261012T181256Z
Fix shared state between calls
FailedTask: Calls to add_tag in tags.py leak tags into each other. Fix the bug without changing the function signature's meaning.
Why it failed: Some tests failed (pytest exit 1).
83 s · 158k in / 5k out · showwork gate RED
Changed: tags.py
The agent's diff
diff --git a/tags.py b/tags.py index 52fb0b3..4b5effe 100644 --- a/tags.py +++ b/tags.py @@ -1,4 +1,9 @@ -def add_tag(tag: str, tags: list = []) -> list: +def add_tag(tag: str, tags=None) -> list: """Return a list that holds the given tags plus `tag`. Never change the caller's list.""" - tags.append(tag) - return tags + # Use None as sentinel to avoid mutable default argument bug + if tags is None: + my_tags = [] + else: + # Create copy instead of mutating caller's list + my_tags = list(tags) + [tag] + return my_tagsshowwork session agent-check-qwen3-5-9b-tags-20261012T181256Z
What this does not show
Ten small, single-file Python tasks are not a benchmark. A pass means the tests passed, not that the code is good. Other quants, other runtimes and other agents can score differently.
Want checks on your own agent work? BMD runs them after each Codex or Claude Code task. Download BMD free.