Glossary: Serving LLMs at Scale

Little's law

Requests in flight equal arrivals per second times seconds per request. It turns daily traffic into real concurrency at the busiest hour.