Glossary: Serving LLMs at Scale
Cold start
The time from starting a model server until it can answer. It includes getting a machine, pulling the image, fetching the weights, loading them and warming up, so it is much longer than a web app's.
Glossary: Serving LLMs at Scale
The time from starting a model server until it can answer. It includes getting a machine, pulling the image, fetching the weights, loading them and warming up, so it is much longer than a web app's.