Skip to content

Glossary: AI System Design

Latency

How long a user waits, measured as time to first token and total time, set with percentiles rather than averages.

Where the book teaches it

Related terms

TTFT (time to first token)p95Latency budget