Glossary: Running Open Models Locally

k-quant

Also written: K, S/M/L, IQ levels, Q4_K_M, Q8_0, Q6_K, Q5_K_M

The naming scheme for GGUF quantization levels: the number is the main bit width, K marks the newer k-quant method, and S, M and L say how many sensitive tensors keep higher precision. Q4_K_M is the usual default; IQ levels use a slower but smarter scheme for very low bit counts.