Measured · lab only · recorded October 18, 2026
KV cache: less VRAM, slower runs
Compressed caches used less VRAM and had lower median throughput on this fixed short-task suite. Some also passed fewer tasks.
No production profile was promoted.
Download data and source hashesQuestion and method
Does a compressed KV cache improve this short-task workload without reducing its task pass count?
RTX 5070 Ti, 16 GB VRAM; 32 GB RAM; Intel Core i9-13900K. Runtime: Ollama 0.40.6. Runtime, model and source hashes are in the download.
Four models, two allocated context capacities, three cache types, three repeats. Per cell: minimum passed tasks, median recorded e2e_tok_s and median peak VRAM bytes. Each candidate is compared with its own model/context f16 reference. This produced 72 retained profiles. The separate context cohort is excluded.
Recorded results
All eight model/context reference cells retained f16. LFM2.5 scored zero on this suite and stays outside the selected lab profile. Its speed is not a useful recommendation.
On narrow screens, scroll the table sideways to compare speed and memory.
| Model | Context capacity | Cache | Min tasks passed | Median e2e tok/s | Median peak VRAM (GiB) |
|---|---|---|---|---|---|
| gemma4:12b | 16384 | f16 | 16 | 72.1 | 7.81 |
| gemma4:12b | 16384 | q8_0 | 16 | 68.7 | 7.52 |
| gemma4:12b | 16384 | q4_0 | 14 | 69.7 | 7.36 |
| gemma4:12b | 65536 | f16 | 16 | 72.6 | 7.91 |
| gemma4:12b | 65536 | q8_0 | 16 | 68.6 | 7.63 |
| gemma4:12b | 65536 | q4_0 | 14 | 69.8 | 7.47 |
| lfm2.5:8b | 16384 | f16 | 0 | 339.1 | 5.13 |
| lfm2.5:8b | 16384 | q8_0 | 0 | 328.2 | 5.04 |
| lfm2.5:8b | 16384 | q4_0 | 0 | 326.6 | 4.99 |
| lfm2.5:8b | 65536 | f16 | 0 | 339.5 | 6.02 |
| lfm2.5:8b | 65536 | q8_0 | 0 | 327.5 | 5.67 |
| lfm2.5:8b | 65536 | q4_0 | 0 | 328.3 | 5.48 |
| qwen3.5:4b | 16384 | f16 | 13 | 115.8 | 3.40 |
| qwen3.5:4b | 16384 | q8_0 | 10 | 109.2 | 3.20 |
| qwen3.5:4b | 16384 | q4_0 | 13 | 113.9 | 3.07 |
| qwen3.5:4b | 65536 | f16 | 13 | 115.5 | 5.26 |
| qwen3.5:4b | 65536 | q8_0 | 10 | 109.1 | 4.52 |
| qwen3.5:4b | 65536 | q4_0 | 13 | 113.1 | 4.02 |
| qwen3.5:9b | 16384 | f16 | 15 | 89.6 | 5.63 |
| qwen3.5:9b | 16384 | q8_0 | 15 | 87.0 | 5.40 |
| qwen3.5:9b | 16384 | q4_0 | 14 | 88.7 | 5.27 |
| qwen3.5:9b | 65536 | f16 | 15 | 88.9 | 7.55 |
| qwen3.5:9b | 65536 | q8_0 | 15 | 87.3 | 6.71 |
| qwen3.5:9b | 65536 | q4_0 | 14 | 88.3 | 6.21 |
Limits and current disposition
- Short prompts; allocated context is not long-input quality.
- e2e_tok_s includes prefill and generation, subtracts model load and excludes server startup and cleanup; not total wall-clock throughput.
- Cold means empty model residency, not cleared OS or compiler caches.
- All eight model/context reference cells retained f16. LFM scored zero and remains excluded from the selected lab profile.
- No production profile was promoted. This is not a model leaderboard or whole-machine speedup.
The separate Dory cohort was inconclusive. The autonomy trial was unfinished in this recorded packet. Neither supports a completed-result claim here.
Sources and next steps
- Ollama cache documentation explains the runtime settings; it is not the source of these measurements.
- KV cache flag guide covers the separate llama.cpp command-line settings.
- Check what fits your GPU before your own task evaluation.