Glossary: Serving LLMs at Scale
Key vLLM flags
Also written: --gpu-memory-utilization, --max-model-len, --max-num-seqs, --max-num-batched-tokens
--gpu-memory-utilization is the share of GPU memory vLLM may use (default 0.92); --max-model-len is the longest prompt plus answer (lower it to free KV memory); --max-num-seqs is the most requests decoded at once (lower it to protect per-user speed).