Glossary: Serving LLMs at Scale
Goodput
Requests per second that meet the SLO. A server answering 10 a second but making half wait 20 seconds has a goodput of 5.
Glossary: Serving LLMs at Scale
Requests per second that meet the SLO. A server answering 10 a second but making half wait 20 seconds has a goodput of 5.