KV Cache Q4 vs Q8 in llama.cpp Explained

In llama.cpp server, choosing between a q4 and a q8 KV cache means choosing between the q4_0 and q8_0 runtime cache types instead of the default f16.

Setting a quantized key-value cache changes how your server stores context tokens in memory. When you configure this setting, you are not picking model weight files like Q4_K_M or Q8_0. You are configuring runtime buffer formats for attention keys and values.

What is the difference in llama.cpp between a q4 and q8 KV cache?

The difference comes down to the specific cache format you supply to the server runtime. In llama.cpp, setting a q4 KV cache means selecting q4_0, while a q8 KV cache selects q8_0. Both options compress runtime context compared to the standard default format.

According to the official llama.cpp server documentation, both key and value buffers default to f16. The allowed values for both settings are f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, and q5_1. Notice that k-quants like q4_k or q8_k do not appear on this list.

On the independent format side, the Hugging Face GGUF documentation explains that Q4_0 is a 4-bit round-to-nearest quantization where each block has 32 weights using the formula w = q * block_scale. It describes Q8_0 as an 8-bit round-to-nearest quantization that also uses 32 weights per block with the same scale formula. While Hugging Face defines these formulas for weight tensors, llama.cpp uses the q4_0 and q8_0 naming scheme for its KV cache types.

If you are researching model files rather than context buffers, read our breakdown on GGUF quantization explained and check model weights using the GGUF compare tool.

Which command-line flags control the KV cache?

You control the KV cache types using separate flags for keys and values. The server accepts --cache-type-k or -ctk for the key buffer, and --cache-type-v or -ctv for the value buffer. You can also configure them using environment variables.

Here is how the options look side by side:

The documentation does not state that K and V types must match. It also does not document whether quantizing keys or values is safer for output stability. If you run speculative decoding, the draft model provides separate flags with the exact same allowed values.

You can set these flags directly on the command line:

llama-server -m model.gguf --cache-type-k q8_0 --cache-type-v q8_0

Or test a tighter cache format:

llama-server -m model.gguf --cache-type-k q4_0 --cache-type-v q4_0

The documentation also includes --flash-attn (-fa), which accepts on, off, or auto, with a default of auto. The server documentation does not state that quantized KV types require Flash Attention.

When should you pick q8_0 over q4_0?

You should pick q8_0 when you need memory reduction beyond f16 but want a wider cache representation than q4_0. You pick q4_0 only when memory constraints demand a smaller cache type and your specific task verification still passes.

Start from the documented default. If your required context fits in your memory budget at f16, leave both settings at f16. Neither source documentation provides exact bytes-per-token figures, perplexity deltas, or benchmark scores between q4_0 and q8_0. Because published documents omit quality deltas, treat any switch away from f16 as an experimental change that you must verify against your own evaluation suite.

For broader hardware considerations, read about local LLM inference on consumer GPUs.

What should you do with this?

1. Inspect your current server startup arguments to see if --cache-type-k and --cache-type-v are omitted, which means your server is running the default f16 cache.
2. If you need a smaller cache, launch a test instance using --cache-type-k q8_0 and --cache-type-v q8_0 and run a deterministic test prompt to confirm output consistency.
3. If memory limits still require a smaller cache, test --cache-type-k q4_0 and --cache-type-v q4_0, keeping the configuration only if your task evaluation passes.

Accompanying prompt

What the prompt does: This prompt helps you construct and verify llama.cpp server launch flags for q4 and q8 KV cache configurations.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: Local LLM Server Administrator Context: You configure llama.cpp server instances using documented CLI arguments. The default KV cache type is f16, and allowed options include q8_0 and q4_0. Inputs: - Model path: __ - Desired K cache type (f16, q8_0, q4_0): __ - Desired V cache type (f16, q8_0, q4_0): __ - Context length: __ - Port: __ Task: 1. Review the provided inputs and verify that the requested cache types match the documented allowed list. 2. Generate the exact llama-server CLI command using --cache-type-k and --cache-type-v flags. 3. Provide a test verification plan to compare outputs against the f16 baseline. Output: - Command line snippet - Configuration summary - Task verification steps Constraints: - Use only valid llama.cpp server flags. - Do not assume k-quant names like q4_k or q8_k exist for cache types. - State that output quality must be verified against task outputs.

27 lines/908 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Weekly measured local runs: https://bmdpat.com/5090-reports