Quantization - Q2, Q4, Q6, Q8 explained
Quantization is how compressed the model's weights are.
Short answer
Quantization is how compressed the model’s weights are. Q4 = 4 bits per weight (smaller, faster, slightly less accurate). Q8 = 8 bits (bigger, slower, closer to original). Q4_K_M is the sweet spot for most people.
Detailed explanation
A model’s original weights are stored as 16-bit floating point numbers. That’s a lot of data - a 7B model at fp16 is ~14 GB. Quantization compresses each weight into fewer bits, trading a small quality loss for big size and speed wins.
The main tiers:
| Quant | Bits/weight | 7B model size | Quality vs original |
|---|---|---|---|
| Q2_K | ~2.5 | 2.8 GB | Noticeable loss |
| Q3_K_M | ~3.4 | 3.5 GB | Small loss |
| Q4_K_M | ~4.5 | 4.5 GB | Very small loss |
| Q5_K_M | ~5.4 | 5.3 GB | Barely detectable |
| Q6_K | ~6.5 | 6.1 GB | Nearly identical |
| Q8_0 | ~8.5 | 7.6 GB | Effectively lossless |
IQ variants (IQ2, IQ3, IQ4) - “importance quantization”, newer method. Same bits/weight as their Q counterparts but smarter compression. Better quality at the same size, but slower on some hardware.
F16 / BF16 - no quantization. Rarely used for local inference (too big, no speed benefit).
The rule of thumb: Q4_K_M is the default for most users. It’s the smallest quant where the model still feels “smart” - going to Q3 or Q2 makes noticeable dumbness for most tasks. Going to Q6/Q8 costs 30-70% more RAM for quality gains you probably won’t notice in normal chat.
When to go smaller (Q3, Q2):
- Running a big model on tight RAM (e.g. 13B on 8 GB laptop)
- You’ve tested and confirmed the smaller quant works for your use case
When to go larger (Q6, Q8):
- Precision-sensitive tasks: math, code, long-form technical writing
- You have RAM to spare and want the best possible quality
- Reproducing benchmark numbers from papers
Why it matters
Choosing quantization is choosing where to spend RAM. Same 13B model at Q4 needs ~9 GB, at Q8 needs ~15 GB. If your PC has 16 GB total, that’s the difference between “fits comfortably” and “fits only after closing everything else”. For most chat use, Q4 gives you 95% of the quality at 55% of the size.
In Abliteration specifically
The quant is auto-detected from the GGUF file’s metadata and shown in the model name as a suffix: Llama 3.2 3B Instruct · Q4. If you download the same model at different quants (Q4 and Q8), Abliteration shows both with distinct suffixes so you can compare.
The Models tab row shows the file size in the subtitle (e.g. 2.1 GB · GGUF); the fit-check dialog uses the actual file size (not a tier estimate) to decide whether it fits your RAM.
The Edit dialog on each model (⋯ → Edit) lets you rename or use the original filename, but doesn’t change the quant - the quant is baked into the file.
Related
- Model naming - how the · Q4 suffix appears
- RAM requirements - quant is the biggest lever on model RAM
- Recommended models by RAM - which quant fits which rig