Glossary: LLMs in Production

Replicas

Also written: replica

Several copies of a service behind a load balancer. In-memory state must move to a shared store like Redis before you add them.