KV cache

Also written: key-value cache

Memory in which the model stores a key and a value for every token in every attention layer so it doesn't recompute them. It grows with context length and at long contexts can exceed the weights.