Skip to content

Glossary: Serving LLMs at Scale

Key vLLM flags

Also written: --gpu-memory-utilization, --max-model-len, --max-num-seqs, --max-num-batched-tokens

--gpu-memory-utilization is the share of GPU memory vLLM may use (default 0.92); --max-model-len is the longest prompt plus answer (lower it to free KV memory); --max-num-seqs is the most requests decoded at once (lower it to protect per-user speed).

Where the book teaches it

Related terms

KV cacheConcurrencyPreemption