Skip to content

Glossary: Serving LLMs at Scale

Slot

Also written: slots, -np

A place in llama-server where one request is worked on, set with -np. All busy slots are decoded together, and extra requests wait in a queue.

Where the book teaches it

Related terms

Continuous batchingUnified KV poolContext size