Glossary: Serving LLMs at Scale
GPTQ
A 4-bit weight quantization method that quantizes one layer at a time, adjusting the remaining weights to cancel each rounding error.
Glossary: Serving LLMs at Scale
A 4-bit weight quantization method that quantizes one layer at a time, adjusting the remaining weights to cancel each rounding error.