Glossary: Serving LLMs at Scale
Context size
Also written: -c, --ctx-size, context window
The number of tokens of conversation (prompt plus answer) a server can hold. In llama-server, -c sets the total KV cache shared by the slots; in vLLM, --max-model-len caps each request.