Glossary: Serving LLMs at Scale
Quantization
Also written: quantized
Storing a model's numbers in fewer bits to save memory and speed up decoding. On a server it frees memory for more users.
Glossary: Serving LLMs at Scale
Also written: quantized
Storing a model's numbers in fewer bits to save memory and speed up decoding. On a server it frees memory for more users.