Topic
Local LLM
54 posts on local llm — guides and lab notes from real runs on hardware we own. New posts land here automatically. Start anywhere, or grab the copy-paste prompts that ship with them.
- 5 min read
Why production AI is moving to open weights
I tested a local-first writing pipeline on an RTX 5090. Gemma 4 writes the draft, deterministic checks catch errors, and frontier QA decides what ships.
- 5 min read
Q4km vs Q5km: Q4_K_M vs Q5_K_M
q4km vs q5km is Q4_K_M vs Q5_K_M in GGUF. Pick by VRAM headroom, quality risk, and CPU fallback. Use the free quant tool before download.
- 6 min read
My 8B Model Failed a 400-Word Task
Three Llama 3.1 8B runs missed a 400-word floor. Here is the verifier-driven route that moved long-form synthesis to Gemma 4 26B.
- 5 min read
I Raised the QA Bar on Blog Infographics
Blog charts need correct labels. I tightened layout geometry, char budgets, claims checks, and vision logs so a bad PNG cannot ship quietly.
- 6 min read
I Rebuilt 5,383 Embeddings After a Dimension Change
I moved a vault search index from 1,536 to 768 dimensions. Here is why I rebuilt all 5,383 chunks instead of mixing vector spaces.
- 6 min read
Preview Retrieval Before Your Local LLM Runs
Preview the files your local LLM will receive before inference. This simple gate catches bad retrieval while the fix is still cheap and clear.
- 6 min read
Pin Your Context Window or Pay the Reload Tax
Change num_ctx in Ollama and it reloads the whole model. On my RTX 5090 that meant 140 seconds of dead air per context swap. Here is how to stop paying it.
- 6 min read
Pin Your Local LLM Context Size Before You Build a Router
Changing context size can reload a local model before every request. A measured RTX 5090 sweep shows why context belongs in the routing key.
- 6 min read
When Claude hits a weekly limit, your agent fleet still needs a third CLI
Claude and Codex both went dark. Here is the tertiary Gemini path I wired, with honest qa_reviewer stamps.
- 8 min read
I built a self-improving code model on one RTX 5090. Here is what actually worked.
Six pieces, one consumer GPU, no cloud. The honest results: some parts worked, some were flat, and one idea changed everything.
- 5 min read
Will That Local Model Fit? Do the VRAM Math First
A local LLM needs about half a gigabyte of VRAM per billion parameters at Q4, then KV cache and context stack on top. Here is how to know a model fits before you download 40 GB.
- 5 min read
Local LLMs Need a Timeout Before They Need a Bigger Model
A bigger local model will not fix a stuck runtime. Add a bounded inference doctor first, then trust the benchmark.
The AI agent build notes
Real costs, real tools, no fluff. M-F when I ship, publish, or learn something worth sending.