Glossary: Serving LLMs at Scale

Replica

Also written: replicas

One complete copy of the model server. Several replicas give more throughput and survive one failing or being upgraded.