Glossary: Serving LLMs at Scale
Decode
The phase where the model writes the answer one token per step, each step reading the whole KV cache. It is memory-bound and sets inter-token latency.
Glossary: Serving LLMs at Scale
The phase where the model writes the answer one token per step, each step reading the whole KV cache. It is memory-bound and sets inter-token latency.