Skip to content

Glossary: Serving LLMs at Scale

Unified KV pool

Also written: unified KV, --kv-unified

One KV cache pool that any llama-server slot can fill up to the whole context, so long and short requests share memory flexibly. It is the default when -np isn't set.

Where the book teaches it

Related terms

SlotKV cache