Glossary: Serving LLMs at Scale
W4A16 / W8A8
Also written: W4A16, W8A8
Quantization scheme names: W is bits for weights and A is bits for activations. W4A16 keeps 4-bit weights and 16-bit activations; W8A8 makes both 8-bit.
Glossary: Serving LLMs at Scale
Also written: W4A16, W8A8
Quantization scheme names: W is bits for weights and A is bits for activations. W4A16 keeps 4-bit weights and 16-bit activations; W8A8 makes both 8-bit.