Skip to content

Glossary: Serving LLMs at Scale

SLO (service-level objective)

Also written: SLO

A latency target, such as p95 TTFT under 2 seconds. Goodput counts the requests that meet it.

Where the book teaches it

Related terms

GoodputPercentile