Skip to content
[bmdpat]

Agent check · measured 2026-10-12

qwen3.5:9b Q4_K_M passed 5 of 10 checked coding tasks.

Codex CLI in OSS mode drove each model through Ollama in a container. Each task's check was written before the run and ran after the agent stopped, with run_check.py and tests/ restored from the fixture first.

A task passes when its check exits 0 and the showwork gate reads GREEN. Hardware: one RTX 5090. Agent: codex-cli 0.162.0. Runtime: ollama 0.31.2. Tokens: 2.1M in, 66k out, $0 in API fees. AgentGuard prices the same tokens at gpt-5.3-codex list rates at $4.55, before cache discounts (price table 2026-09-26).

  1. Add a command-line option

    Passed

    Task: Add a --limit option to main() in cli.py: an integer, default 10, that caps how many lines main prints.

    88 s · 270k in / 10k out · showwork gate GREEN

    Changed: cli.py

    The agent's diff
    diff --git a/cli.py b/cli.py
    index fd0a366..09f5f58 100644
    --- a/cli.py
    +++ b/cli.py
    @@ -4,7 +4,12 @@ import argparse
     def main(argv: list | None = None) -> int:
         parser = argparse.ArgumentParser(prog="lines")
         parser.add_argument("count", type=int, help="how many numbered lines to print")
    +    parser.add_argument("--limit", type=int, default=10, help="cap on how many lines to print (default 10)")
         args = parser.parse_args(argv)
    -    for n in range(1, args.count + 1):
    +    
    +    # Cap the number of lines at --limit if it's lower than requested count
    +    effective_count = min(args.count, args.limit)
    +    
    +    for n in range(1, effective_count + 1):
             print(f"line {n}")
         return 0
    

    showwork session agent-check-qwen3-5-9b-cliflag-20261012T181256Z

  2. Handle missing and bad input

    Passed

    Task: Implement load_port in config_reader.py so it does what its docstring says.

    16 s · 36k in / 2k out · showwork gate GREEN

    Changed: config_reader.py

    The agent's diff
    diff --git a/config_reader.py b/config_reader.py
    index 39bdf10..89cadfb 100644
    --- a/config_reader.py
    +++ b/config_reader.py
    @@ -1,3 +1,6 @@
    +import json
    +
    +
     def load_port(path: str) -> int:
         """Return the "port" value from the JSON file at `path`.
     
    @@ -5,4 +8,25 @@ def load_port(path: str) -> int:
         ValueError when the port is not an integer from 1 to 65535 (a bool is not
         an integer here).
         """
    -    raise NotImplementedError
    +    # Try loading JSON, return default on any read/parse error
    +    try:
    +        with open(path) as f:
    +            data = json.load(f)
    +    except (FileNotFoundError, OSError):
    +        return 8080
    +    
    +    if "port" not in data or data["port"] is None:
    +        return 8080
    +    
    +    port = data["port"]
    +    
    +    # Reject booleans explicitly before checking type
    +    if isinstance(port, bool) or not isinstance(port, int):
    +        raise ValueError(f"Port must be an integer from 1 to 65535 (got {type(port).__name__})")
    +    
    +    if port < 1 or port > 65535:
    +        raise ValueError("Port out of range")
    +    
    +    return port
    +
    +
    

    showwork session agent-check-qwen3-5-9b-config-20261012T181256Z

  3. Implement an ordered, case-blind dedupe

    Passed

    Task: Implement dedupe in dedupe.py so it does what its docstring says.

    24 s · 49k in / 3k out · showwork gate GREEN

    Changed: dedupe.py

    The agent's diff
    diff --git a/dedupe.py b/dedupe.py
    index f2ae1ed..5c63c41 100644
    --- a/dedupe.py
    +++ b/dedupe.py
    @@ -3,4 +3,13 @@ def dedupe(names: list) -> list:
     
         Keep the spelling of the first time a name appears. Do not change the input.
         """
    -    raise NotImplementedError
    +    seen = set()  # lowercase versions we've already encountered
    +    result = []   # keep original spellings from list
    +    
    +    for name in names:
    +        key = name.lower()
    +        if key not in seen:
    +            seen.add(key)
    +            result.append(name)
    +    
    +    return result
    

    showwork session agent-check-qwen3-5-9b-dedupe-20261012T181256Z

  4. Parse a small input format

    Failed

    Task: Implement parse_duration in duration.py so it does what its docstring says.

    Why it failed: Some tests failed (pytest exit 1).

    62 s · 162k in / 8k out · showwork gate RED

    Changed: duration.py

    The agent's diff
    diff --git a/duration.py b/duration.py
    index 0450915..c8d4668 100644
    --- a/duration.py
    +++ b/duration.py
    @@ -1,7 +1,36 @@
    +import re
    +
    +
     def parse_duration(text: str) -> int:
         """Return the number of seconds in a duration like "1h30m", "45s" or "2h5m10s".
     
         Units are h, m and s, in that order, each at most once. Raise ValueError for
         anything else, including an empty string.
         """
    -    raise NotImplementedError
    +    if not text:
    +        raise ValueError("Empty duration")
    +    
    +    # Match numeric value + unit (h/m/s), allowing multiple components
    +    pattern = re.compile(r'(\d+)([hms])')
    +    matches = list(pattern.finditer(text))
    +    
    +    if len(matches) != len(re.findall('[hms]', text)):  # all h,m,s must be matched with digits before them
    +        raise ValueError("Invalid format: digit-less unit in string")
    +    
    +    seconds = 0
    +    
    +    for i, match in enumerate(matches):
    +        count_str, unit = match.groups()
    +        count = int(count_str)
    +        
    +        if unit == 'h':
    +            seconds += count * 3600
    +        elif unit == 'm':
    +            seconds += count * 60
    +        elif unit == 's':
    +            # Check that s only appears after all h and m components in the string
    +            pass
    +    
    +    return int(seconds)
    +
    +
    

    showwork session agent-check-qwen3-5-9b-duration-20261012T181256Z

  5. Merge nested settings without side effects

    Failed

    Task: Implement deep_merge in merge.py so it does what its docstring says.

    Why it failed: The tests could not run (pytest exit 2: an import or syntax error).

    89 s · 279k in / 11k out · showwork gate RED

    Changed: merge.py

    The agent's diff
    diff --git a/merge.py b/merge.py
    index f224354..481255d 100644
    --- a/merge.py
    +++ b/merge.py
    @@ -1,7 +1,22 @@
    -def deep_merge(base: dict, override: dict) -> dict:
    -    """Return a new dict: `base` with `override` merged in.
    +def deep_merge(base, override):
    +    
    +    """Return a new dict: base with override merged in.
     
    -    When both values for a key are dicts, merge them the same way. Otherwise the
    -    value from `override` wins. Never change `base` or `override`.
    -    """
    -    raise NotImplementedError
    +When both values for a key are dicts, merge them the same way. Otherwise
    +the value from override wins. Never change base or override."""
    +    
    +    def _merge(b_dct, o_dct): 
    +        # Merge b_dct and o_dct into result dict  
    +        if not (b_dct | o_dct): return {} 
    +        
    +        all_keys = set(list(o_dct.keys()) + list(b_dct.keys()))  # Union
    +        merged_at_this_level = {} 
    +
    +_EOF_EOF_ echo '    for key in sorted(all_keys)' >> /work/merge.py && cat << '_EOF_EOF_' >> /work/merge.py  
    +'        
    +            b_val = b_dct.get(key)   # May be None
    +            
    +EOF_EOF  
    +
    +echo "Current state of merge.py after partial writes:"
    +cat /work/merge.py 2>/dev/null | tail -15
    

    showwork session agent-check-qwen3-5-9b-merge-20261012T181256Z

  6. Fix an off-by-one bug

    Passed

    Task: page() in pager.py returns the wrong items. Page numbers start at 1. Fix the bug.

    24 s · 51k in / 3k out · showwork gate GREEN

    Changed: pager.py

    The agent's diff
    diff --git a/pager.py b/pager.py
    index 0bd5957..708ec87 100644
    --- a/pager.py
    +++ b/pager.py
    @@ -2,5 +2,5 @@ def page(items: list, number: int, size: int) -> list:
         """Return page `number` (starting at 1) of `items`, `size` items per page."""
         if number < 1 or size < 1:
             raise ValueError("number and size must be at least 1")
    -    start = number * size
    +    start = (number - 1) * size
         return items[start:start + size]
    

    showwork session agent-check-qwen3-5-9b-paginate-20261012T181256Z

  7. Rename a function across two files

    Failed

    Task: Rename get_usr to get_user everywhere in this project. Remove the old name.

    Why it failed: The tests passed, but showwork refused the close: the agent changed venv/bin/python: path escapes project root: venv/bin/python (and 3 more).

    95 s · 414k in / 9k out · showwork gate RED

    Changed: report.py, users.py, venv/bin/python, venv/bin/python3, venv/bin/python3.11, venv/lib/python3.11/site-packages/_distutils_hack/__init__.py, venv/lib/python3.11/site-packages/_distutils_hack/override.py, venv/lib/python3.11/site-packages/distutils-precedence.pth, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/LICENSE.txt, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/METADATA, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/RECORD, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/WHEEL, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/entry_points.txt, venv/lib/python3.11/site-packages/pip-23.0.1.dist-info/top_level.txt, venv/lib/python3.11/site-packages/pip/__init__.py and 744 more files

    The agent's diff
    diff --git a/report.py b/report.py
    index 8062d48..874fd6e 100644
    --- a/report.py
    +++ b/report.py
    @@ -1,5 +1,5 @@
    -from users import get_usr
    +from users import get_user
     
     
     def summary(ids: list) -> str:
    -    return ", ".join(get_usr(i) for i in ids)
    +    return ", ".join(get_user(i) for i in ids)
    diff --git a/users.py b/users.py
    index b2368c1..670a56f 100644
    --- a/users.py
    +++ b/users.py
    @@ -1,5 +1,5 @@
     _USERS = {1: "ada", 2: "linus"}
     
     
    -def get_usr(user_id: int) -> str:
    +def get_user(user_id: int) -> str:
         return _USERS[user_id]
    

    showwork session agent-check-qwen3-5-9b-rename-20261012T181256Z

  8. Read a CSV file correctly

    Failed

    Task: Implement total in sheet.py so it does what its docstring says.

    Why it failed: The tests could not run (pytest exit 2: an import or syntax error).

    95 s · 421k in / 11k out · showwork gate RED

    Changed: sheet.py

    The agent's diff
    diff --git a/sheet.py b/sheet.py
    index 300f952..0c5e387 100644
    --- a/sheet.py
    +++ b/sheet.py
    @@ -1,8 +1,3 @@
    -def total(path: str, column: str) -> float:
    -    """Return the sum of `column` in the CSV file at `path`.
    +import csv
     
    -    The first row is the header. Skip blank cells. Values may be quoted and may
    -    contain commas as thousands separators, like "1,250.50". Raise KeyError when
    -    the column does not exist.
    -    """
    -    raise NotImplementedError
    +def total(path, column):
    

    showwork session agent-check-qwen3-5-9b-sheet-20261012T181256Z

  9. Implement a function from its docstring

    Passed

    Task: Implement slugify in textutil.py so it does what its docstring says.

    60 s · 231k in / 7k out · showwork gate GREEN

    Changed: textutil.py

    The agent's diff
    diff --git a/textutil.py b/textutil.py
    index f8bf9f4..d294734 100644
    --- a/textutil.py
    +++ b/textutil.py
    @@ -1,3 +1,5 @@
    +import re
    +
     def slugify(text: str) -> str:
         """Return a URL slug: lowercase, runs of a-z and 0-9 joined by single hyphens, nothing else."""
    -    raise NotImplementedError
    +    return "-".join(re.findall(r"[a-z0-9]+", text.lower()))
    

    showwork session agent-check-qwen3-5-9b-slugify-20261012T181256Z

  10. Fix shared state between calls

    Failed

    Task: Calls to add_tag in tags.py leak tags into each other. Fix the bug without changing the function signature's meaning.

    Why it failed: Some tests failed (pytest exit 1).

    83 s · 158k in / 5k out · showwork gate RED

    Changed: tags.py

    The agent's diff
    diff --git a/tags.py b/tags.py
    index 52fb0b3..4b5effe 100644
    --- a/tags.py
    +++ b/tags.py
    @@ -1,4 +1,9 @@
    -def add_tag(tag: str, tags: list = []) -> list:
    +def add_tag(tag: str, tags=None) -> list:
         """Return a list that holds the given tags plus `tag`. Never change the caller's list."""
    -    tags.append(tag)
    -    return tags
    +    # Use None as sentinel to avoid mutable default argument bug
    +    if tags is None:
    +        my_tags = []
    +    else:
    +        # Create copy instead of mutating caller's list
    +        my_tags = list(tags) + [tag]
    +    return my_tags
    

    showwork session agent-check-qwen3-5-9b-tags-20261012T181256Z

What this does not show

Ten small, single-file Python tasks are not a benchmark. A pass means the tests passed, not that the code is good. Other quants, other runtimes and other agents can score differently.

Want checks on your own agent work? BMD runs them after each Codex or Claude Code task. Download BMD free.