Full Precision vs Q8 vs Q6 vs Q4: Which Weights to Run?
Evaluates whether full precision, Q8, Q6, or Q4 fits your local hardware and test requirements.
Show prompt
Role: Local AI Systems Engineer Context: I am choosing between full precision (F16/BF16), Q8_0, Q6_K, and Q4_K_M for running a model locally. I need to determine memory fit and plan verification steps based on hardware constraints. Inputs: - Model name: __ - Base precision (F32, F16, or BF16): __ - Available VRAM: __ GB - Target context length: __ tokens - Primary task type: __ Task: 1. Review the input model and memory capacity. 2. Calculate whether full precision, Q8_0, Q6_K, or Q4_K_M fits within the specified VRAM including context buffers. 3. Recommend the highest precision preset that fits the VRAM budget. 4. Provide a verification plan to check output quality on the primary task type. Output: - Memory fit assessment for each quantization level - Selected target quantization preset - Three concrete verification steps for the specified task Constraints: - Do not assume universal quality percentages. - Account for context memory beyond weight file sizes.