Glossary: Serving LLMs at Scale
Least outstanding requests
A routing rule that sends each request to the replica with the fewest requests in progress, rather than simply the next in line.
Glossary: Serving LLMs at Scale
A routing rule that sends each request to the replica with the fewest requests in progress, rather than simply the next in line.