Glossary: Serving LLMs at Scale
Router
Also written: LLM-aware router
A load balancer that understands LLM traffic, using least outstanding requests, affinity and health checks plus per-user limits. Kettle & Crane's router.py is a small one.
Glossary: Serving LLMs at Scale
Also written: LLM-aware router
A load balancer that understands LLM traffic, using least outstanding requests, affinity and health checks plus per-user limits. Kettle & Crane's router.py is a small one.