Glossary: Serving LLMs at Scale
Prefill
The phase where the model reads the whole prompt in parallel and stores keys and values in the KV cache, ending with the first output token. It is compute-bound and mostly sets TTFT.
Glossary: Serving LLMs at Scale
The phase where the model reads the whole prompt in parallel and stores keys and values in the KV cache, ending with the first output token. It is compute-bound and mostly sets TTFT.