KV cache

KV cache (key-value cache) is the store of intermediate attention tensors a transformer keeps around during autoregressive generation so it never recomputes them.

Here is the mechanism. In self-attention, every token produces a query, a key, and a value vector at each layer. When you generate token by token, token 50's output depends on the keys and values of tokens 1 through 49, which do not change once computed. Without a cache, generating each new token would rerun attention over the entire prefix, making a sequence of length n cost roughly O(n^2) work over the whole decode. With the cache, you compute keys and values once per token, append them, and each new step is O(n). Think of it like memoization, except the memoized values are per-layer tensors rather than function return values.

The cost is memory. Cache size scales as 2 * layers * heads * head_dim * seq_len * batch * bytes. A concrete case: a 7B model with 32 layers serving a 4k-token context in fp16 needs on the order of a few hundred megabytes of KV cache per sequence, which is why long contexts and high batch sizes pressure GPU memory more than the weights do. This is what techniques like paged attention, grouped-query attention, and cache quantization target.

It is worth noting what the cache does not hold. It stores activations, not parameters. A LoRA adapter or a swapped weight set changes the projections that produce keys and values, so in a setup like PAW where a small interpreter runs many short tasks, the KV cache is transient per call and modest, while the hot-attached adapter is the part that actually changes behavior.