Skip to content

Glossary: Serving LLMs at Scale

Context size

Also written: -c, --ctx-size, context window

The number of tokens of conversation (prompt plus answer) a server can hold. In llama-server, -c sets the total KV cache shared by the slots; in vLLM, --max-model-len caps each request.

Where the book teaches it

Related terms

SlotKV cacheKey vLLM flags