KV cache

Also written: KV memory, keys and values

The stored keys and values for every token already processed in a conversation. Weights are shared but each conversation needs its own KV cache, so memory usually limits how many users fit on a GPU.