Skip to content

Glossary: Serving LLMs at Scale

Affinity

Also written: prefix affinity

Sending the same user or conversation to the same replica so that replica's prefix cache stays warm. The router falls back to the least busy replica if the home one is overloaded.

Where the book teaches it

Related terms

Prefix cachingReplicaRouter