Glossary: Serving LLMs at Scale
Deployment
Also written: pod, Service
Kubernetes objects. A Deployment keeps N copies of a container running; each copy is a pod; a Service gives the pods one address and spreads requests over the ready ones.
Glossary: Serving LLMs at Scale
Also written: pod, Service
Kubernetes objects. A Deployment keeps N copies of a container running; each copy is a pod; a Service gives the pods one address and spreads requests over the ready ones.