Glossary: Serving LLMs at Scale
Scale to zero
Running no instances when idle so nothing is billed, at the cost of a cold start for the first customer afterwards.
Glossary: Serving LLMs at Scale
Running no instances when idle so nothing is billed, at the cost of a cold start for the first customer afterwards.