Topic
Local LLM
33 posts on local llm — guides and lab notes from real runs on hardware we own. New posts land here automatically. Start anywhere, or grab the copy-paste prompts that ship with them.
- 5 min read
A 7 GB 27B Model Lost to My 17 GB Default
A 6.66 GiB ternary 27B model fit my RTX 5090, but it lost the default slot. File size is only the first local model-selection gate.
- 6 min read
My Agents Have to Prove What They Did
I extracted the claims ledger that runs my agent operation into showwork. Falsifiable claims, deterministic checks, and an exit gate that refuses a false done.
- 5 min read
Ollama Raised $65M. What Builders Get
Ollama's $65M round and 8.9 million developers make local AI a funded product category. Here is what changes for builders on owned GPUs.
- 5 min read
Why production AI is moving to open weights
I tested a local-first writing pipeline on an RTX 5090. Gemma 4 writes the draft, deterministic checks catch errors, and frontier QA decides what ships.
- 5 min read
Q4km vs Q5km: Q4_K_M vs Q5_K_M
q4km vs q5km is Q4_K_M vs Q5_K_M in GGUF. Pick by VRAM headroom, quality risk, and CPU fallback. Use the free quant tool before download.
- 6 min read
My 8B Model Failed a 400-Word Task
Three Llama 3.1 8B runs missed a 400-word floor. Here is the verifier-driven route that moved long-form synthesis to Gemma 4 26B.
- 5 min read
I Raised the QA Bar on Blog Infographics
Blog charts need correct labels. I tightened layout geometry, char budgets, claims checks, and vision logs so a bad PNG cannot ship quietly.
- 6 min read
I Rebuilt 5,383 Embeddings After a Dimension Change
I moved a vault search index from 1,536 to 768 dimensions. Here is why I rebuilt all 5,383 chunks instead of mixing vector spaces.
- 6 min read
Preview Retrieval Before Your Local LLM Runs
Preview the files your local LLM will receive before inference. This simple gate catches bad retrieval while the fix is still cheap and clear.
- 6 min read
Pin Your Context Window or Pay the Reload Tax
Change num_ctx in Ollama and it reloads the whole model. On my RTX 5090 that meant 140 seconds of dead air per context swap. Here is how to stop paying it.
- 6 min read
Pin Your Local LLM Context Size Before You Build a Router
Changing context size can reload a local model before every request. A measured RTX 5090 sweep shows why context belongs in the routing key.
- 6 min read
When Claude hits a weekly limit, your agent fleet still needs a third CLI
Claude and Codex both went dark. Here is the tertiary Gemini path I wired, with honest qa_reviewer stamps.
- 8 min read
I built a self-improving code model on one RTX 5090. Here is what actually worked.
Six pieces, one consumer GPU, no cloud. The honest results: some parts worked, some were flat, and one idea changed everything.
- 5 min read
Will That Local Model Fit? Do the VRAM Math First
A local LLM needs about half a gigabyte of VRAM per billion parameters at Q4, then KV cache and context stack on top. Here is how to know a model fits before you download 40 GB.
- 5 min read
Local LLMs Need a Timeout Before They Need a Bigger Model
A bigger local model will not fix a stuck runtime. Add a bounded inference doctor first, then trust the benchmark.
- 5 min read
Your local LLM is not a worse Claude. It is a different tool.
Stop scoring your local model on how close it gets to Opus. It is a different tool with a different sweet spot. Here is the line, and which side your work sits on.
- 5 min read
A verifier loop beats a faster local model
Local LLMs are useful when the loop proves the output, not when the benchmark looks good. This is the small gate I use before a local coding agent gets more rope.
- 4 min read
How I Gate a Local Coding Model Before I Trust It
A local model is not ready because it runs fast. It is ready when one verifier loop can prove the output before an agent writes files.
- 5 min read
How I Make Local Model Runs Fail Safely On A 5090
A local model run should prove its safety path before it proves a score. Here is the small guardrail loop I use on my RTX 5090 for QLoRA starter work.
- 5 min read
How to Make a Local QLoRA Starter Fail Safely
A local QLoRA starter should prove data, GPU safety, metrics, tests, and blockers before it claims progress. Here is the small loop I use on owned hardware.
- 5 min read
How to Run Local LLM Verifier Loops on Owned Hardware
A local LLM workflow needs more than a model prompt. It needs a verifier loop that proves the file, command, URL, or report changed before the agent claims done.
- 2 min read
VRAM Calculator: Estimate Local LLM Requirements
Estimate the VRAM required to run local LLMs like Llama 3 with our interactive calculator. Compare quantization levels like Q4 and Q8 to plan your hardware.
- 7 min read
llama.cpp Multi-GPU: Splitting a Model Across Cards with --tensor-split
Split a 70B model across multiple GPUs with llama.cpp. How --tensor-split, --main-gpu, and --split-mode work on a real consumer rig.
- 6 min read
llama.cpp -ngl 99 Still on CPU? 5 Fixes, Ranked (2026)
You set -ngl 99 and llama.cpp still pins the CPU — the flag isn't the bug. Here's the 30-second load-log check and the 5 real causes, ranked by how often they bite.
- 6 min read
The AI Whirlwind: Why Your Local Agent Matters More Than Ever
Amidst the big tech AI boom and new policy discussions, discover why building ethical, autonomous AI agents on consumer hardware is critical. Explore practical engineering insights and Python tips for true local control.
- 5 min read
Decoding the AI Summer: Building Accountable Agents for the User
As the AI world heats up, learn how to build AI agents that prioritize user control and transparency. Discover practical strategies for creating observable and accountable automation on your own hardware.
- 7 min read
Localmaxxing isn't theory. Here's what my 3-GPU rig actually does.
Tom Tunguz called it localmaxxing. I run a 3070 + 5070 Ti + 5090 in one box and serve Llama 3.1 8B locally every day. Here are the real tokens-per-second, the real watts, and the real cost per million tokens.
- 8 min read
Q4_K_M vs Q5_K_M: Which GGUF Quant?
Compare Q4_K_M vs Q5_K_M vs Q8 GGUF: VRAM fit, quality, speed, and when a smaller quant beats a bigger one that falls back to CPU.
- 5 min read
GPU Prices Up 48% in Two Months. I Run LLMs in My Garage.
Blackwell rental hit $4.08/hr. CoreWeave raised prices 20%. Anthropic restricted their newest model to 40 orgs. Meanwhile, consumer GPUs are sitting idle.
- 6 min read
Anthropic's Advisor Tool Is the Cost-Split Pattern You Should Already Be Running
Anthropic shipped a pattern where a cheap model runs the loop and escalates to Opus only when it needs to. The pattern works on any two-model setup. Here is the math and the playbook.
- 7 min read
llama.cpp --n-gpu-layers: -1, 0, Partial
Set --n-gpu-layers in llama.cpp with clear -1, 0, and partial offload rules, VRAM headroom checks, examples, and CPU fallback fixes.
- 6 min read
Raspberry Pi 5 Offline Voice Assistant: Sub-2s, No Cloud (2026)
Want a private voice assistant with zero cloud and no subscription? A Raspberry Pi 5 runs it offline at sub-2s latency. We tested 6 local models on real hardware — here's the winner. (2026)
- 7 min read
Local LLM on Consumer GPUs: 50 req/s, $0/Call [Benchmarks 2026]
Cloud LLM bills hit $2K/month fast. An RTX 5070 Ti serves Llama 3.1 at 50 req/s for $0 per call — we benchmarked 4 consumer GPUs and built the exact production setup.
The AI agent build notes
Real costs, real tools, no fluff. M-F when I ship, publish, or learn something worth sending.