Glossary: Serving LLMs at Scale

Memory-bound

Limited by how fast the processor can read memory. Decode is memory-bound because each token reads every weight once for very little arithmetic.