Writing · Tag

#inference

3 posts tagged #inference.

Jun 8, 20267 min read
llama.cpp Multi-GPU: Splitting a Model Across Cards with --tensor-split
Split a 70B model across multiple GPUs with llama.cpp. How --tensor-split, --main-gpu, and --split-mode work on a real consumer rig.
#llama.cpp#local-llm#gpu#multi-gpu
Jun 4, 20266 min read
llama.cpp -ngl 99 Still on CPU? 5 Fixes, Ranked (2026)
You set -ngl 99 and llama.cpp still pins the CPU — the flag isn't the bug. Here's the 30-second load-log check and the 5 real causes, ranked by how often they bite.
#llama-cpp#local-llm#gpu-offloading#n-gpu-layers
Apr 6, 20267 min read
llama.cpp n-gpu-layers Explained: -1 vs 0 + VRAM Guide (2026)
Setting --n-gpu-layers wrong tanks your tokens/sec or crashes with OOM. Here's exactly what to use (-1, 0, or a number), the VRAM-per-layer math, and 4060-4090 benchmarks.
#llama-cpp#local-llm#gpu-offloading#n-gpu-layers

The AI agent build notes

Real costs, real tools, no fluff. M-F when I ship, publish, or learn something worth sending.