Glossary: Serving LLMs at Scale
Consistent hashing
A way of assigning users to replicas so that adding a replica moves only a few users, keeping most caches warm.
Glossary: Serving LLMs at Scale
A way of assigning users to replicas so that adding a replica moves only a few users, keeping most caches warm.