Glossary: Serving LLMs at Scale
PagedAttention
vLLM's way of splitting the KV cache into small blocks handed out as a request grows, cutting waste from most of the memory to under 1% and letting blocks be shared.
Glossary: Serving LLMs at Scale
vLLM's way of splitting the KV cache into small blocks handed out as a request grows, cutting waste from most of the memory to under 1% and letting blocks be shared.