Glossary: Serving LLMs at Scale
Slot
Also written: slots, -np
A place in llama-server where one request is worked on, set with -np. All busy slots are decoded together, and extra requests wait in a queue.
Glossary: Serving LLMs at Scale
Also written: slots, -np
A place in llama-server where one request is worked on, set with -np. All busy slots are decoded together, and extra requests wait in a queue.