Skip to content

Glossary: Serving LLMs at Scale

Least outstanding requests

A routing rule that sends each request to the replica with the fewest requests in progress, rather than simply the next in line.

Where the book teaches it

Related terms

RouterLoad balancer