q5_k_m vs q5_k_s: what the extra 0.12 GiB changes

On 2026-10-16 I read the llama.cpp quantizer source to answer q5_k_m vs q5_k_s. Google was sending that search to my Q4_K_M versus Q5_K_M post. That post answers a different choice.

Q5_K_M and Q5_K_S are two llama.cpp recipes on one 5-bit tensor type, Q5_K. Q5_K_S keeps the layer weights at Q5_K. Q5_K_M moves selected attention-value and feed-forward-down tensors up to Q6_K. On the llama.cpp README table for Llama 3.1 8B, that costs 0.12 GiB: 5.21 GiB against 5.33 GiB.

Open full-size image (opens in new tab)

What is the difference between Q5_K_M and Q5_K_S?

Both files start from Q5_K, which pull request 1684 lists at 5.5 bits per weight. Q5_K_M then stores selected attention and feed-forward-down tensors as Q6_K, at 6.5625 bits per weight. Q5_K_S has no rule of its own. That is the whole difference.

I checked this at llama.cpp commit 6868a52. In src/llama-quant.cpp, Q5_K_S shows up once, where both presets map to GGML_TYPE_Q5_K. Q5_K_M shows up in three rules. Each rule sets GGML_TYPE_Q6_K:

1. attn_v tensors on the layers a helper named use_more_bits picks.
2. ffn_down tensors on the layers the same helper picks.
3. Every fused attn_qkv tensor, in models that store Q, K and V in one tensor.

The helper is one line:

return i_layer < n_layers/8 || i_layer >= 7*n_layers/8 || (i_layer - n_layers/8)%3 == 2;

It picks the first eighth of the layers, the last eighth, and every third layer between them. I ran it for a 32-layer model. It picked 16 of 32: layers 0 to 3, layers 28 to 31, and 8 layers in between. The 2023 pull request called this "half" of those tensors, and for 32 layers it is.

Both presets share the other exceptions in that file. By default the output tensor goes to Q6_K in both. Architecture rules and tensor shapes can change the mix. So the suffix names a recipe, not an exact tensor list for every model.

How much bigger is a Q5_K_M file?

About 2.3 percent on an 8B model. The llama.cpp README lists Llama 3.1 8B at 5.21 GiB for Q5_K_S and 5.33 GiB for Q5_K_M. Bartowski's Llama 3.1 8B Instruct files are 5.60GB and 5.73GB. His card covers the Instruct model in GB, and the README covers the base model in GiB, so treat them as two data points.

That table comes from the README's meta-llama/Llama-3.1-8B section at commit 6868a52. The README does not name the machine behind the speed columns. On that machine, prompt speed was close. Q5_K_M generated 2.30 tokens per second slower. I would not carry that gap to your GPU. Measure it on yours.

Bartowski's model card lists 5.60GB for Q5_K_S and 5.73GB for Q5_K_M, a 0.13GB gap. His card says he used llama.cpp release b3472 with an importance matrix. A GiB is about 1.074 GB, so the README's 5.21 GiB is about 5.59 GB. Do not compare a GiB column with a GB column.

Does Q5_K_M have lower perplexity?

Yes, in both perplexity sources I read, by a small margin. The quantizer's help text lists +0.1049 perplexity for Q5_K_S and +0.0569 for Q5_K_M on Llama-3-8B. The 2023 pull request had Q5_K_S at 5.9419 and Q5_K_M at 5.9208 on LLaMA 7B, against 5.9066 for F16.

Those are two different models. The help text does not name its baseline. The pull request measured perplexity on the wikitext dataset.

On the 7B row, Q5_K_S sat 0.0353 above F16. Q5_K_M sat 0.0142 above it. Both gaps are small. The 13B row tells the same story: 5.2785 for Q5_K_S, 5.2638 for Q5_K_M, 5.2543 for F16. The same 7B table timed the two files at 16.7 and 16.9 milliseconds per token on an RTX 4080 with 4 threads.

Perplexity measures how well a model predicts a text set. It is not a coding score or a chat score. A lower number on wikitext does not tell you how your agent task will go.

What is Q5_K_P?

I could not find one. The quantizer type table at commit 6868a52 lists Q5_K, Q5_K_S and Q5_K_M. Q5_K is an alias for Q5_K_M. No Q5_K_P is in that table. People still search for q5_k_p vs q5_k_m, so somebody uses the name.

If you have a file named Q5_K_P, ask its publisher what recipe made it. Publishers do add their own labels. Bartowski's repo has a Q5_K_L file, and his card says it uses Q8_0 for the embed and output weights. Q5_K_L is not in the quantize.cpp table either.

Did I test Q5_K_M vs Q5_K_S on my 5090?

No. I checked my benchmark files and my sizing desk corpus on 2026-10-16. Neither has a Q5_K_S row or a Q5_K_M row. Every number in this post comes from llama.cpp or a Hugging Face model card. I have no paired result of my own to add, and I am not going to guess one.

Which one should you download?

If both files fit with room for your context, take Q5_K_M. It costs about 2.3 percent more disk and weight memory, and every public perplexity row I found favors it. If only the smaller file fits, take Q5_K_S. If neither fits, look at the Q4 files instead.

1. Read the size of both files for the same base model and the same publisher. Use bytes when the page gives them.
2. Add your context cost on top. The sizing desk at https://bmdpat.com/desk takes your card and returns the models that fit, the quant, and a command you can paste.
3. If both fit, load each file with the same context and the same prompt. Record tokens per second and peak memory.
4. If quality matters, run one task check on both files. Keep that result apart from speed.

To build both files yourself from one high-precision GGUF, use the positional form the README shows for Q4_K_M:

./build/bin/llama-quantize model-bf16.gguf model-Q5_K_S.gguf Q5_K_S
./build/bin/llama-quantize model-bf16.gguf model-Q5_K_M.gguf Q5_K_M

Leave out --pure. It turns off the k-quant mixtures, and then S and M come out the same.

The Q4 versus Q5 choice is a different post: Q4_K_M versus Q5_K_M. The 4-bit version of this question is q4km vs q4ks. The wider quant map is Q4, Q5, and Q8 explained.

Accompanying prompt

What the prompt does: Compares your own Q5_K_S and Q5_K_M files by size, fit, speed, and one quality check before you keep one.

Copy/paste this prompt:

Copy-ready prompt

Paste the exact block into your coding agent.

No article chrome, no footnotes, no formatting drift.

Copy promptSelect text

Role: Local model operator who picks between two GGUF files of one model. Context: Q5_K_S and Q5_K_M are llama.cpp presets. Both use Q5_K as the base type. Q5_K_M stores part of attn_v and ffn_down as Q6_K, so it is a little larger. A published size or speed is not a result from your machine. Inputs: - Base model and publisher: __ - Q5_K_S file path and size in bytes: __ - Q5_K_M file path and size in bytes: __ - GPU and memory in GB: __ - Context length: __ - Prompt token count: __ - Output token count: __ Task: 1. Convert both sizes to GiB. Say if the page gave GB or GiB. 2. Load each file with the same context length. Record any load error. 3. Run the same prompt and output length on each file. 4. Record tokens per second and peak memory for each file. 5. If I give a task check, run it on both files. Keep that result apart from speed. Output: - A table with file, size in GiB, tokens per second, and peak memory. - One line that names the file to keep and the one fact that decided it. Constraints: - Do not treat a published benchmark as my result. - Do not mix a 7B perplexity row with an 8B speed row. - If a file fails to load, write the error and stop that row.

32 lines/1186 chars

Ready

This prompt and every other one we publish live in the free prompt library.

Copy the block above.

Weekly measured local runs: https://bmdpat.com/5090-reports