Glossary: Serving LLMs at Scale
Autoscaling
Also written: autoscale
Adding or removing instances automatically. Scale on what users feel, such as queue length or p95 TTFT, never on CPU use, because a busy GPU is the goal.
Glossary: Serving LLMs at Scale
Also written: autoscale
Adding or removing instances automatically. Scale on what users feel, such as queue length or p95 TTFT, never on CPU use, because a busy GPU is the goal.