Glossary: Serving LLMs at Scale
Load balancer
A component that picks a replica for each request. A general one spreads requests evenly; an LLM-aware router also uses load and cache affinity.
Glossary: Serving LLMs at Scale
A component that picks a replica for each request. A general one spreads requests evenly; an LLM-aware router also uses load and cache affinity.