Quantizations for humanists

What a model's weights are

A model's "weights" are, physically, an enormous list of numbers - for a mid-sized model, several billion of them. Each number records the strength of one connection learned during training. To run the model is to do arithmetic with all of them for every word produced.

The size of that list depends on how precisely each number is stored. Precision is measured in bits. A model's native precision is usually 16-bit floating point (F16 or BF16), meaning each weight occupies two bytes of memory. That is why a 70-billion-parameter model in its native precision needs roughly 140 gigabytes just to hold the weights - more memory than any consumer graphics card, and more than most single data-center accelerators.

F16 is "native" because it is the format the model was trained and released in. Every compressed copy is measured against it.

What quantization is

Quantization is lossy compression for weights. Instead of two bytes per number, store each in fewer bits - 8, 5, 4, 3, even 2 - accepting a small loss of accuracy in exchange for a large saving in memory. At 4 bits, that 70B model drops to under 40 gigabytes and becomes runnable on a single high-end consumer GPU. A 7B model shrinks from about 13.5 GB at F16 to about 4 GB at Q4_K_M, roughly a 3.3x reduction.

The trade is real: fewer bits mean coarser numbers, and coarser numbers mean the model's outputs drift from what full precision would produce. How much they drift depends on the scheme.

The GGUF format

The dominant format for local inference is GGUF - the letters derive from "GGML Universal Format"; GGML in turn carries the initials of its author, Georgi Gerganov. He created it for the llama.cpp project, and it replaced the earlier GGML format in 2023.

Its design virtue is self-containment: a single .gguf file holds the quantized weights, the tokenizer, and all the metadata needed to run the model, so any compatible program can load it without extra files. This is why abliterated GGUFs are the format most laptop users encounter - LM Studio, Ollama, KoboldCpp, and text-generation-webui all consume the same files.

Reading the cryptic names

The strings on a download page are readable once decoded. Two representative examples:

Q4_K_M

• Q4 - 4-bit quantization. Each weight uses 4 bits instead of 16.
• K - the K-quant scheme, a smarter method that stores weights in blocks with their own scaling factors, giving better quality than the older "0" schemes at the same bit count.
• M - the medium variant, which keeps a handful of especially sensitive weights at higher precision.

So Q4_K_M reads as "4-bit, K-quant, medium."

IQ3_XXS

• I - marks an I-quant, which uses a pre-computed importance matrix (a measurement of which weights matter most, taken by running calibration text through the model) to decide where to spend its scarce bits.
• Q3 - 3-bit quantization.
• XXS - the smallest, most aggressive variant. Stands for "extra-extra-small."

So IQ3_XXS reads as "importance-weighted 3-bit, extra-extra-small."

Rough orientation

This is how to read the options, not a recommendation of which to pick:

• The bit count tracks the memory needed. More bits, more memory, closer to native quality.
• The suffix tracks the quality-for-size trade within a given bit count. S / M / L variants (small / medium / large) preserve progressively more of the sensitive weights.
• Higher bit counts (Q6, Q8) sit closer to native quality and need more memory.
• 4-bit K-quants are the common community compromise - Q4_K_M is what most model cards recommend as the value sweet spot.
• Sub-4-bit I-quants squeeze onto small hardware at rising risk of degradation.

Which one fits depends entirely on the memory of the machine doing the running, which is why catalogs list many. A person with 8 GB of VRAM has fundamentally different options from one with 24 GB or 80 GB, and the same underlying model has to be quantized differently for each.

Why one model has twelve files

This is why a single abliterated model appears on a hosting site five, ten, or fifteen times. Each entry is the same underlying weights compressed to a different level for a different class of hardware. The high-volume quantizers - mradermacher (with over 68,000 model repositories), bartowski - exist to produce these families exhaustively.

A typical release page for a mid-sized abliterated model might carry: Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, Q8_0, IQ3_XXS, IQ4_XS. Twelve files, one underlying edit. Nothing in the twelve entries adds a new abliteration or teaches the model a new capability. They are twelve compressions of one artifact.

The philosophical residue

What to download (rough guide)

For most users on most hardware, the answer is Q4_K_M. It is the community sweet spot: small enough to run on modest hardware, high enough quality that prose stays coherent, well-tested across model sizes from 7B to 70B.

Rough shape of the decision:

• Have plenty of memory, want highest quality → Q6_K or Q8_0. Above Q6 the perplexity gains stop scaling with disk space, so Q8 is rarely worth the extra memory.
• Standard consumer hardware, want the default → Q4_K_M. The value sweet spot.
• Modest hardware, accept some quality tradeoff → Q4_K_S, or if further squeezed, IQ4_XS or IQ3_M with imatrix calibration.
• Very tight memory, willing to accept clear degradation → IQ3_XXS or Q2_K. Prose degrades visibly here; use for experimentation, not production.

For the technical reasoning behind these choices, and the huihui-ai _L trick that preserves ablation-affected tensors at higher precision, see B9 M8 Repackaging.

Where to go next

Read B9 M8 Repackaging for the technical side of GGUF quantization - the actual commands, the imatrix calibration workflow, the failure modes. Read C2 Reading benchmarks for how quantization interacts with the benchmark numbers a model card reports. And if you are trying to run a specific abliterated model and are unsure which quant to pick, the model card on Hugging Face usually recommends one - trust the recommendation on the specific model over any general rule.