Skip to content

Glossary: Serving LLMs at Scale

Latency

How long one user waits: the time to first token, then the speed of later tokens. It trades against throughput.

Where the book teaches it

Related terms

ThroughputTime to first token (TTFT)