Glossary: Serving LLMs at Scale

KV space

Also written: GPU memory budget

The GPU memory left for the KV cache: GPU memory times the utilization cap, minus the weights and about 1 to 3 GB of overhead. Dividing it by the memory per conversation gives how many users fit.