Glossary: Serving LLMs at Scale
Throughput
The whole server's work: total tokens or requests per second across all users. Batching raises it a lot and each request's latency a little.
Glossary: Serving LLMs at Scale
The whole server's work: total tokens or requests per second across all users. Batching raises it a lot and each request's latency a little.