Topic
Local LLM
26 posts on local llm — guides and lab notes from real runs on hardware we own. New posts land here automatically. Start anywhere, or grab the copy-paste prompts that ship with them.
- 5 min read
Will That Local Model Fit? Do the VRAM Math First
A local LLM needs about half a gigabyte of VRAM per billion parameters at Q4, then KV cache and context stack on top. Here is how to know a model fits before you download 40 GB.
- 5 min read
Local LLMs Need a Timeout Before They Need a Bigger Model
A bigger local model will not fix a stuck runtime. Add a bounded inference doctor first, then trust the benchmark.
- 5 min read
Your local LLM is not a worse Claude. It is a different tool.
Stop scoring your local model on how close it gets to Opus. It is a different tool with a different sweet spot. Here is the line, and which side your work sits on.
- 5 min read
A verifier loop beats a faster local model
Local LLMs are useful when the loop proves the output, not when the benchmark looks good. This is the small gate I use before a local coding agent gets more rope.
- 4 min read
How I Gate a Local Coding Model Before I Trust It
A local model is not ready because it runs fast. It is ready when one verifier loop can prove the output before an agent writes files.
- 5 min read
How I Make Local Model Runs Fail Safely On A 5090
A local model run should prove its safety path before it proves a score. Here is the small guardrail loop I use on my RTX 5090 for QLoRA starter work.
- 5 min read
How to Make a Local QLoRA Starter Fail Safely
A local QLoRA starter should prove data, GPU safety, metrics, tests, and blockers before it claims progress. Here is the small loop I use on owned hardware.
- 5 min read
How to Run Local LLM Verifier Loops on Owned Hardware
A local LLM workflow needs more than a model prompt. It needs a verifier loop that proves the file, command, URL, or report changed before the agent claims done.
- 5 min read
AI Agent Memory in 2026: How It Works and When to Use It
Understanding different memory architectures for AI agents and when persistent, episodic, or vector memory actually pays off in production.
- 2 min read
VRAM Calculator: Estimate Local LLM Requirements
Estimate the VRAM required to run local LLMs like Llama 3 with our interactive calculator. Compare quantization levels like Q4 and Q8 to plan your hardware.
- 5 min read
GGUF Quant Cheat Sheet: Q4 vs Q5 vs Q6 vs Q8 (2026)
Skip the theory — a one-glance decision table for Q4_K_M, Q5_K_M, Q6_K, and Q8_0 on consumer GPUs, with the quality and size tradeoff spelled out for each.
- 5 min read
Tune llama.cpp --n-gpu-layers: VRAM Math & OOM Fixes (2026)
Set --n-gpu-layers too high and you OOM; too low and inference crawls. The VRAM math, KV-cache sizing, and a fast tuning loop to find the right value for your GPU. (2026)
- 5 min read
GGUF Quantization and VRAM: How to Pick Q4, Q5, or Q8 for Your GPU (2026)
VRAM decides your GGUF quant, not vibes. How I assign Q4, Q5, Q8 across an 8GB 3070, 16GB 5070 Ti, and 32GB 5090.
- 7 min read
llama.cpp Multi-GPU: Splitting a Model Across Cards with --tensor-split
Split a 70B model across multiple GPUs with llama.cpp. How --tensor-split, --main-gpu, and --split-mode work on a real consumer rig.
- 6 min read
How to Tune --n-gpu-layers for Your VRAM Budget
How to actually pick --n-gpu-layers: the offload math, finding the number with nvidia-smi, multi-GPU splits, and the top OOM mistakes.
- 6 min read
How to Pick a GGUF Quant Level for Your VRAM Budget
Given your GPU, which GGUF quant do you actually pick? The VRAM math, a card-by-card table, and the quality tradeoff in plain terms.
- 6 min read
llama.cpp -ngl 99 Still on CPU? 5 Fixes, Ranked (2026)
You set -ngl 99 and llama.cpp still pins the CPU — the flag isn't the bug. Here's the 30-second load-log check and the 5 real causes, ranked by how often they bite.
- 6 min read
The AI Whirlwind: Why Your Local Agent Matters More Than Ever
Amidst the big tech AI boom and new policy discussions, discover why building ethical, autonomous AI agents on consumer hardware is critical. Explore practical engineering insights and Python tips for true local control.
- 5 min read
Decoding the AI Summer: Building Accountable Agents for the User
As the AI world heats up, learn how to build AI agents that prioritize user control and transparency. Discover practical strategies for creating observable and accountable automation on your own hardware.
- 7 min read
Localmaxxing isn't theory. Here's what my 3-GPU rig actually does.
Tom Tunguz called it localmaxxing. I run a 3070 + 5070 Ti + 5090 in one box and serve Llama 3.1 8B locally every day. Here are the real tokens-per-second, the real watts, and the real cost per million tokens.
- 8 min read
GGUF Quantization 2026: Q4_K_M vs Q5 vs Q8 — Which to Pick
Short answer: Q4_K_M wins for most local LLMs — 75% smaller with near-zero quality loss. Q5, Q6 and Q8 each win edge cases. Benchmarked on real GPUs — here's the pick for your VRAM. (2026)
- 5 min read
GPU Prices Up 48% in Two Months. I Run LLMs in My Garage.
Blackwell rental hit $4.08/hr. CoreWeave raised prices 20%. Anthropic restricted their newest model to 40 orgs. Meanwhile, consumer GPUs are sitting idle.
- 6 min read
Anthropic's Advisor Tool Is the Cost-Split Pattern You Should Already Be Running
Anthropic shipped a pattern where a cheap model runs the loop and escalates to Opus only when it needs to. The pattern works on any two-model setup. Here is the math and the playbook.
- 7 min read
llama.cpp n-gpu-layers Explained: -1 vs 0 + VRAM Guide (2026)
Setting --n-gpu-layers wrong tanks your tokens/sec or crashes with OOM. Here's exactly what to use (-1, 0, or a number), the VRAM-per-layer math, and 4060-4090 benchmarks.
- 6 min read
Raspberry Pi 5 Offline Voice Assistant: Sub-2s, No Cloud (2026)
Want a private voice assistant with zero cloud and no subscription? A Raspberry Pi 5 runs it offline at sub-2s latency. We tested 6 local models on real hardware — here's the winner. (2026)
- 7 min read
Local LLM on Consumer GPUs: 50 req/s, $0/Call [Benchmarks 2026]
Cloud LLM bills hit $2K/month fast. An RTX 5070 Ti serves Llama 3.1 at 50 req/s for $0 per call — we benchmarked 4 consumer GPUs and built the exact production setup.
The AI agent build notes
Real costs, real tools, no fluff. M-F when I ship, publish, or learn something worth sending.