Everyone quantizes the weights and forgets the KV cache

Everyone quantizes the weights and forgets the KV cache. On long context, that cache is what blows your VRAM budget.

You AWQ the model to 4-bit, fit it on the GPU, ship it. Then a user pastes a 200-page contract, and per-token KV at 128K quietly eats more memory than the weights you just shrank.

Low-bit KV cache used to be a trap: push it to 2-bit and needle-in-haystack recall falls apart. New work (Block-GTQ, RoPE-aware bit allocation, Jun 2026) says stop quantizing it uniformly. Under RoPE, key energy concentrates in a few frequency blocks. Spend bits where the signal is, starve the rest.

Paper-stated on Llama-3.1-8B, benchmark your own:

• 3.24x KV-cache compression at fp16-comparable quality
• needle recall 70.6 to 97.4 vs naive low-bit
• 1.34x faster than fp16 attention at 128K on one H800

The trap: uniform 2-bit KV looks clean on short prompts and silently wrecks recall past 32K. You won't catch it in eval until a customer's long document returns the wrong clause.

Quantize the weights for the GPU bill. Quantize the KV cache for the context window. That second one is the on-prem win nobody is pricing in yet.