Skip to content

Glossary: LLMs in Production

Throughput

How many requests a service completes per second. Used with the number of allowed requests to work out the worst wait.

Where the book teaches it

Related terms

GateLoad test