Glossary: Serving LLMs at Scale
Concurrency
The number of requests in progress at the same moment. It is lower than the number of open chats, because most customers are reading or typing.
Glossary: Serving LLMs at Scale
The number of requests in progress at the same moment. It is lower than the number of open chats, because most customers are reading or typing.