Glossary: Serving LLMs at Scale
Block
Also written: blocks, block table
A small fixed-size piece of KV cache memory (16 tokens by default in vLLM). A request's block table lists which blocks, anywhere in memory, belong to it.
Glossary: Serving LLMs at Scale
Also written: blocks, block table
A small fixed-size piece of KV cache memory (16 tokens by default in vLLM). A request's block table lists which blocks, anywhere in memory, belong to it.