Glossary: Serving LLMs at Scale
Compute-bound
Limited by how fast the processor can do arithmetic rather than by memory. Prefill is compute-bound because each weight read is reused for every prompt token.
Glossary: Serving LLMs at Scale
Limited by how fast the processor can do arithmetic rather than by memory. Prefill is compute-bound because each weight read is reused for every prompt token.