Glossary: Serving LLMs at Scale
Drain
Also written: draining
Stopping traffic to a replica before shutting it down, so in-progress requests aren't broken. The failure drill showed why this matters in production.
Glossary: Serving LLMs at Scale
Also written: draining
Stopping traffic to a replica before shutting it down, so in-progress requests aren't broken. The failure drill showed why this matters in production.