---
title: "Everyone quantizes the weights and forgets the KV cache"
date: 2026-06-23
series: 14
summary: "Everyone quantizes the weights and forgets the KV cache. On long context, that cache is what blows your VRAM budget."
voice: builder
tags: [kv-cache-quantization, rope-aware-bit-allocation, 2bit-kv, block-gtq, long-context-serving]
image: "/notes/kv-cache-quantization/image.png"
imageAlt: "weights vs KV memory at 128K, and uniform-2bit vs RoPE-aware recall"
linkedin: null
sources: []
dateApprox: false
---

Everyone quantizes the weights and forgets the KV cache. On long context, that cache is what blows your VRAM budget.

You AWQ the model to 4-bit, fit it on the GPU, ship it. Then a user pastes a 200-page contract, and per-token KV at 128K quietly eats more memory than the weights you just shrank.

Low-bit KV cache used to be a trap: push it to 2-bit and needle-in-haystack recall falls apart. New work (Block-GTQ, RoPE-aware bit allocation, Jun 2026) says stop quantizing it uniformly. Under RoPE, key energy concentrates in a few frequency blocks. Spend bits where the signal is, starve the rest.

Paper-stated on Llama-3.1-8B, benchmark your own:
- 3.24x KV-cache compression at fp16-comparable quality
- needle recall 70.6 to 97.4 vs naive low-bit
- 1.34x faster than fp16 attention at 128K on one H800

The trap: uniform 2-bit KV looks clean on short prompts and silently wrecks recall past 32K. You won't catch it in eval until a customer's long document returns the wrong clause.

Quantize the weights for the GPU bill. Quantize the KV cache for the context window. That second one is the on-prem win nobody is pricing in yet.