When to Replace Your AI Agent With a Script

Will Larson published a piece this week called "Agents as Scaffolding." His argument: agents are best as temporary prototypes, not permanent production infrastructure.

I run 12 AI agents overnight. They write PRs, draft blog posts, scan for leads, compile morning briefs. After months of running this system, I agree with Larson's thesis. Mostly.

Larson's three-phase framework

Larson lays out a clean pattern:

1. Prototype with agents to understand the problem
2. Refactor deterministic steps to code once you understand the shape
3. Keep agents only for genuinely ambiguous reasoning

His example is perfect. His team tried to automate Dependabot alert triage with a pure-agent approach. The agent couldn't reliably filter to only critical severity alerts. It kept including medium and high severity items. You can't interrupt your colleagues with false positives.

The fix: code handles the webhook filtering (deterministic). The agent handles ownership determination (ambiguous, requires context about teams and codebases).

What I learned running 12 agents

My vault runs a daily agent pipeline: email scanner at 5:15 AM, vault health at 5:30, digest at 5:45, distribution scout at 6:00, and so on through a morning brief at 7:30.

After months of iteration, here's what graduated from agent to code:

Graduated to scripts:

• File routing (which folder does this go in?) became path rules
• TTL enforcement (archive after 3 days) became a cron job
• Sync operations (copy files between systems) became a scheduled PowerShell task
• Format validation (does the YAML frontmatter parse?) became a linter

Stayed as agents:

• Content scoring (is this article relevant to our business?)
• Blog drafting (write a post from intel, match our voice)
• Competitive analysis (is this company a threat?)
• Morning brief synthesis (summarize overnight activity, flag what matters)

The pattern is exactly what Larson describes. Anything deterministic graduated. Anything requiring judgment stayed.

The graduation test

Here's my rule of thumb. Ask two questions:

1. Can I write an if/else tree that handles 95% of cases? If yes, it's code.
2. Does the remaining 5% actually matter? If no, it's code. If yes, keep the agent for that 5%.

Most things are code. The email scanner doesn't need an LLM to check if a subject line contains "AI" or "agent." A regex does that. The LLM's job is deciding whether the article content is relevant to a specific project. That's the ambiguous part.

Where Larson is wrong (slightly)

Larson frames agents as temporary. Build the prototype, then replace it with code. I think there's a third category: agents that stay agents forever because the task is permanently ambiguous.

Content scoring will never be deterministic. Blog drafting will never be a template. Competitive analysis requires judgment that changes weekly. These aren't scaffolding. They're permanent infrastructure. The key is they only handle the ambiguous parts. Everything around them is code.

The cost angle

Every agent call costs money. My overnight runs process 30+ items per sweep. If I ran agents for everything (file routing, TTL checks, format validation, sync), I'd burn 3x the tokens for the same output.

Graduating deterministic steps to code isn't just about reliability. It's about cost. Every if/else you write is a token you don't spend.

If you're running agents in production, audit your pipeline. Find the deterministic steps. Graduate them. Keep the agents for what they're actually good at: reasoning under ambiguity.

Get started with runtime guards for your agents