Glossary: Serving LLMs at Scale
Arithmetic intensity
The amount of calculation done per byte read from memory. It decides whether a step is limited by memory or by compute.
Glossary: Serving LLMs at Scale
The amount of calculation done per byte read from memory. It decides whether a step is limited by memory or by compute.