Skip to content

Glossary: Serving LLMs at Scale

Quantization

Also written: quantized

Storing a model's numbers in fewer bits to save memory and speed up decoding. On a server it frees memory for more users.

Where the book teaches it

Related terms

FP8AWQ (activation-aware weight quantization)GPTQPerplexity