Glossary: Serving LLMs at Scale
Latency
How long one user waits: the time to first token, then the speed of later tokens. It trades against throughput.
Glossary: Serving LLMs at Scale
How long one user waits: the time to first token, then the speed of later tokens. It trades against throughput.