Glossary: Serving LLMs at Scale

Round-robin

A simple load-balancing rule that sends each request to the next replica in turn. It does poorly when request lengths vary a lot.