---
title: "Full Precision vs Q8 vs Q6 vs Q4: Which Weights to Run?"
excerpt: "Compare full precision vs Q8 vs Q6 vs Q4 weight sizes, memory needs, and measured speeds so you can pick the right GGUF quant for your local setup."
author: "Patrick Hughes"
published_at: 2026-11-19T16:32:52.912+00:00
read_time_min: 5
tags: ["local-ai", "local-llm", "rtx-5090", "ai-infrastructure", "5090-reports"]
canonical: https://bmdpat.com/blog/full-precision-vs-q8-vs-q6-vs-q4-which-weights-to-run-2026
---

# Full Precision vs Q8 vs Q6 vs Q4: Which Weights to Run?

**Your choice between full precision vs Q8 vs Q6 vs Q4 depends on your available memory at your target context length and actual task checks, not an assumed quality score.** Quantization cuts file size and memory footprint, but weight changes can alter model output. Start by checking your baseline format and what fits directly in memory.

![Full precision vs Q8 vs Q6 vs Q4 evaluation steps: baseline formats, file sizes, and hardware measurements](/blog-assets/full-precision-vs-q8-vs-q6-vs-q4-which-weights-to-run-2026.png)

## What is full precision compared to quantized weights?

Full precision refers to unquantized floating-point baseline weights, but the phrase hides whether the file is 32-bit or 16-bit. You must identify whether a model starts as F32, F16, or BF16 before comparing it to integer formats like Q8_0, Q6_K, or Q4_K_M.

According to the [Hugging Face GGUF documentation](https://huggingface.co/docs/hub/gguf), F32 is 32-bit single precision, F16 is 16-bit half precision, and BF16 is a distinct 16-bit format. GGUF stores tensors alongside metadata. You can use the Hub viewer to inspect each tensor's name, shape, and precision.

Quantization encodes weights with fewer bits. For tensor encodings, Q6_K uses 6.5625 bits per weight, while Q4_K uses 4.5 bits per weight, as documented on [Hugging Face](https://huggingface.co/docs/hub/gguf). Those figures describe tensor formats, not whole-model file sizes. When someone asks about full precision, they usually compare a 16-bit original against quantized files. If you want background on how these formats work, read [GGUF quantization explained](https://bmdpat.com/blog/gguf-quantization-q4-q5-q8-explained-2026).

## How do file sizes and bitrates compare across formats?

File sizes drop sharply as you move from 16-bit baselines down to Q4, cutting storage and memory requirements roughly in half between F16 and Q8, and again toward Q4. The exact storage depends on whether the quantization preset applies a uniform bit depth or mixes tensor types.

The [llama.cpp quantize README](https://raw.githubusercontent.com/ggml-org/llama.cpp/ad1b88ca0d37a2171efba1c04f1a3531c78f1b52/tools/quantize/README.md) documents concrete measurements for the `meta-llama/Llama-3.1-8B` model. The published values illustrate the step-down across presets:

| Format | Bits per weight | File size (GiB) |
| :--- | :--- | :--- |
| F16 | 16.0005 | 14.96 GiB |
| Q8_0 | 8.5008 | 7.95 GiB |
| Q6_K | 6.5633 | 6.14 GiB |
| Q4_K_M | 4.8944 | 4.58 GiB |

These numbers belong specifically to Llama-3.1-8B. The upstream project notes that `--pure` disables K-quant mixtures, while `--imatrix file_name` supplies calibration data. You can evaluate memory allocations across different cards with the [GGUF compare tool](https://bmdpat.com/tools/gguf-compare). Remember that total runtime memory exceeds the file size. You must also reserve memory for context state, working buffers, and operating system tasks, especially when running [local LLM inference on consumer GPUs](https://bmdpat.com/blog/local-llm-inference-consumer-gpu-production-2026).

## How do these options behave in local measurements?

Lower bit depths increase generation speed when running inside memory limits, while perplexity tests show minor differences with overlapping error ranges. Local logs show that Q4_K_M runs faster than Q8_0, but the test numbers do not prove task superiority.

In historical records from `Reports/5090/quant-ladder-27b.md` dated 2026-09-21, Qwen3.8-27B-GGUF ran on an RTX 5090 using Ollama server 0.31.2. At a codegen context of 4096 tokens, single-request decode speeds were:
- Q4_K_M: 62.1 tokens per second
- Q6_K: 45.0 tokens per second
- Q8_0: 41.6 tokens per second

Short outputs limit the repeatability of these speed measurements.

Quality was recorded using llama.cpp b10423 on WikiText-2 across 145 chunks at context and batch size 2048 from `Reports/Sizing-Desk/corpus.jsonl` (rows 58-66). The raw logs report:
- `ppl-Q4_K_M.log`: 6.3066 +/- 0.03983
- `ppl-Q6_K.log`: 6.3438 +/- 0.04042
- `ppl-Q8_0.log`: 6.3558 +/- 0.04057

The reported errors overlap. These figures do not establish task equivalence or quality rank. Neither F16, BF16, nor F32 was measured in this local ladder. Neither upstream source supplies universal quality percentages or a universal speed hierarchy across hardware.

## How do you quantize a baseline model file?

You quantize a model using the `llama-quantize` tool by passing an unquantized input file, an output destination, and your target preset name. You should always create quantized files from original high-precision weights rather than requantizing an already quantized file.

The [llama.cpp quantize guide](https://raw.githubusercontent.com/ggml-org/llama.cpp/ad1b88ca0d37a2171efba1c04f1a3531c78f1b52/tools/quantize/README.md) provides this exact command line for converting an existing BF16 file:

```bash
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M
```

Requantizing an already quantized file degrades weight fidelity. Always preserve your original high-precision GGUF file as the source.

## What should you do with this?

1. Identify the base precision of your source weights by inspecting the tensor metadata with the Hub viewer or GGUF tools before picking a quant.
2. Select the highest quantization level that fits your hardware budget: keep original precision if memory allows, use Q8_0 if small quants fail, step down to Q6_K if memory fills up, or select Q4_K_M when you need the smallest footprint.
3. Validate your target prompts and required context length directly on your hardware, keeping prompt templates, model revision, and runtime parameters identical during testing.

### Accompanying prompt

**What the prompt does:** Evaluates whether full precision, Q8, Q6, or Q4 fits your local hardware and test requirements.

Copy/paste this prompt:

```text
Role:
Local AI Systems Engineer

Context:
I am choosing between full precision (F16/BF16), Q8_0, Q6_K, and Q4_K_M for running a model locally. I need to determine memory fit and plan verification steps based on hardware constraints.

Inputs:
- Model name: __
- Base precision (F32, F16, or BF16): __
- Available VRAM: __ GB
- Target context length: __ tokens
- Primary task type: __

Task:
1. Review the input model and memory capacity.
2. Calculate whether full precision, Q8_0, Q6_K, or Q4_K_M fits within the specified VRAM including context buffers.
3. Recommend the highest precision preset that fits the VRAM budget.
4. Provide a verification plan to check output quality on the primary task type.

Output:
- Memory fit assessment for each quantization level
- Selected target quantization preset
- Three concrete verification steps for the specified task

Constraints:
- Do not assume universal quality percentages.
- Account for context memory beyond weight file sizes.
```

Copy the block above.

Weekly measured local runs: https://bmdpat.com/5090-reports

---

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=full-precision-vs-q8-vs-q6-vs-q4-which-weights-to-run-2026
