Glossary: Serving LLMs at Scale

Preemption

Also written: preempts

vLLM pausing a request, freeing its KV cache and recomputing it later when memory runs out. The user sees a stall.