Glossary: Serving LLMs at Scale
Activations
Also written: A (in W4A16)
The numbers that flow between a model's layers while it handles a request. They take working memory, and some quantization schemes shrink them too.
Glossary: Serving LLMs at Scale
Also written: A (in W4A16)
The numbers that flow between a model's layers while it handles a request. They take working memory, and some quantization schemes shrink them too.