Glossary: Serving LLMs at Scale
Affinity
Also written: prefix affinity
Sending the same user or conversation to the same replica so that replica's prefix cache stays warm. The router falls back to the least busy replica if the home one is overloaded.