Quantization
Also written: quantize, 4-bit, llama-quantize
Storing weights in fewer bits, such as 4 or 8, to save memory and disk. QLoRA stores the frozen base in 4 bits, and GGUF files can be 8-bit (q8_0) or smaller.
Also written: quantize, 4-bit, llama-quantize
Storing weights in fewer bits, such as 4 or 8, to save memory and disk. QLoRA stores the frozen base in 4 bits, and GGUF files can be 8-bit (q8_0) or smaller.