Glossary: Serving LLMs at Scale
SLO (service-level objective)
Also written: SLO
A latency target, such as p95 TTFT under 2 seconds. Goodput counts the requests that meet it.
Glossary: Serving LLMs at Scale
Also written: SLO
A latency target, such as p95 TTFT under 2 seconds. Goodput counts the requests that meet it.