Glossary: Serving LLMs at Scale

CUDA out of memory

Also written: OOM

An error from running out of GPU memory while serving. Lower --max-num-seqs, --max-num-batched-tokens or --gpu-memory-utilization.