Glossary: Serving LLMs at Scale
Hash
Also written: fingerprint
A fingerprint computed from data. vLLM hashes each full block of tokens with everything before it to find reusable cached blocks; the router hashes a user name to pick a home replica.
Glossary: Serving LLMs at Scale
Also written: fingerprint
A fingerprint computed from data. vLLM hashes each full block of tokens with everything before it to find reusable cached blocks; the router hashes a user name to pick a home replica.