KV cache
Also written: key-value cache
Stored keys and values of earlier tokens, reused while generating so each step computes only the new token. It makes steps cheaper but uses memory that grows with the context.
Also written: key-value cache
Stored keys and values of earlier tokens, reused while generating so each step computes only the new token. It makes steps cheaper but uses memory that grows with the context.