Topic
Local LLM
84 posts on local llm — guides and lab notes from real runs on hardware we own. New posts land here automatically. Start anywhere, or grab the copy-paste prompts that ship with them.
- 2 min read
VRAM Calculator: Estimate Local LLM Requirements
Estimate the VRAM required to run local LLMs like Llama 3 with our interactive calculator. Compare quantization levels like Q4 and Q8 to plan your hardware.
- 7 min read
llama.cpp Multi-GPU Guide: --tensor-split & --split-mode (2026)
Running a 70B across 2 GPUs and hitting OOM? The llama.cpp multi-GPU reference: --tensor-split ratios, --split-mode layer vs row, --main-gpu, and the VRAM math from a real two-card rig.
- 6 min read
llama.cpp -ngl Flag Explained: 5 Fixes When It Stays on CPU
Set -ngl 99 in llama-cli and the GPU still sits idle? The flag isn't the bug. What -ngl (--n-gpu-layers) does, the 30-second load-log check, and the 5 real causes ranked by how often they bite.
- 6 min read
The AI Whirlwind: Why Your Local Agent Matters More Than Ever
Amidst the big tech AI boom and new policy discussions, discover why building ethical, autonomous AI agents on consumer hardware is critical. Explore practical engineering insights and Python tips for true local control.
- 5 min read
Decoding the AI Summer: Building Accountable Agents for the User
As the AI world heats up, learn how to build AI agents that prioritize user control and transparency. Discover practical strategies for creating observable and accountable automation on your own hardware.
- 7 min read
Localmaxxing isn't theory. Here's what my 3-GPU rig actually does.
Tom Tunguz called it localmaxxing. I run a 3070 + 5070 Ti + 5090 in one box and serve Llama 3.1 8B locally every day. Here are the real tokens-per-second, the real watts, and the real cost per million tokens.
- 6 min read
Q4_K_M vs Q5_K_M vs Q8: Which GGUF Quant?
Compare GGUF quants by actual file size, runtime memory, and task results. Pick a candidate, test your context, and keep the version that passes.
- 5 min read
GPU Prices Up 48% in Two Months. I Run LLMs in My Garage.
Blackwell rental hit $4.08/hr. CoreWeave raised prices 20%. Anthropic restricted their newest model to 40 orgs. Meanwhile, consumer GPUs are sitting idle.
- 6 min read
Anthropic's Advisor Tool Is the Cost-Split Pattern You Should Already Be Running
Anthropic shipped a pattern where a cheap model runs the loop and escalates to Opus only when it needs to. The pattern works on any two-model setup. Here is the math and the playbook.
- 7 min read
llama.cpp --n-gpu-layers: -1, 0, Partial GPU Offload (2026)
Not sure what to set --n-gpu-layers to? -1 offloads all layers, 0 keeps it on CPU, a number splits the model. VRAM headroom rules, examples, and the CPU-fallback fix. (2026)
- 6 min read
Raspberry Pi 5 Offline Voice Assistant: 6 Models Tested (2026)
Want a private voice assistant — zero cloud, no subscription? A Raspberry Pi 5 runs it offline at sub-2s latency. We tested 6 local models on real hardware; see the winner. (2026)
- 7 min read
Local LLM on Consumer GPUs: 50 req/s, $0/Call [Benchmarks 2026]
Cloud LLM bills hit $2K/month fast. An RTX 5070 Ti serves Llama 3.1 at 50 req/s for $0 per call — we benchmarked 4 consumer GPUs and built the exact production setup.
The AI agent build notes
Real costs, real tools, no fluff. One evidence-backed note on Friday when there is something worth sharing.
Get the requested artifact now, then at most one evidence-backed Local AI Lab Note on Friday when there is something worth sharing. One-click unsubscribe. No sponsored placements. Privacy.