Glossary: Serving LLMs at Scale
Unified KV pool
Also written: unified KV, --kv-unified
One KV cache pool that any llama-server slot can fill up to the whole context, so long and short requests share memory flexibly. It is the default when -np isn't set.