M8 - Repackaging (GGUF quantization)

What M8 is

M8 is not a refusal-removal method at all. It is the conversion of an already-modified model into a different container or precision, so it runs on hardware the original could not. The dominant operation is quantization to GGUF, the format used by llama.cpp and the tools built on it (LM Studio, Ollama, KoboldCpp). A practitioner takes an abliterated model that someone else produced, runs it through GGUF conversion at a range of precision levels, and republishes the quantized files.

No refusal direction is touched. An M8 model refuses or complies exactly as its parent did, modulo the effects of quantization. That qualifier is not trivial - see the failure modes section for how aggressive quantization can partly undo abliteration through weight perturbation.

How quantization works, briefly

A model's "weights" are, physically, an enormous list of numbers - for a mid-sized model, several billion of them. Each number records the strength of one connection learned during training. To run the model is to do arithmetic with all of them for every word produced.

The size of that list depends on how precisely each number is stored. Precision is measured in bits. A model's native precision is usually 16-bit floating point (F16 or BF16), meaning each weight occupies two bytes. That is why a 70-billion-parameter model in native precision needs roughly 140 gigabytes just to hold the weights - more memory than any consumer graphics card.

Quantization is lossy compression for weights. Instead of two bytes per number, store each in fewer bits - 8, 5, 4, 3, even 2 - accepting a small loss of accuracy for a large saving in memory. At 4 bits, that 70B model drops to under 40 gigabytes and becomes runnable on a single high-end consumer GPU. The trade is real: fewer bits mean coarser numbers, and coarser numbers mean the model's outputs drift from what full precision would produce.

Reading GGUF quantization names

The cryptic names on a download page are readable once decoded. Two representative examples:

Q4_K_M. The "Q4" means 4-bit quantization. The "K" means the K-quant scheme - a smarter method that stores weights in blocks with their own scaling factors, giving better quality than the older "0" schemes at the same bit count. The "M" means the medium variant, which keeps a handful of especially sensitive weights at higher precision. So Q4_K_M reads as "4-bit, K-quant, medium."

IQ3_XXS. "Importance-weighted 3-bit, extra-extra-small." The "I" marks an I-quant, which uses a pre-computed importance matrix (measurement of which weights matter most, taken by running calibration text through the model) to decide where to spend its scarce bits. "XXS" marks the smallest, most aggressive variant.

Full current llama.cpp quantization ladder, smallest to largest:

$ column -t gguf_ladder.tsv

LEVEL      BITS   NOTES
─────────  ─────  ──────────────────────
Q2_K       2.6    smallest; experiments
Q3_K_S/M   3.4    low; prose degrades
Q3_K_L     3.9    noticeable degradation
Q4_0       4.5    legacy 4-bit; use Q4_K
Q4_K_S     4.6    slightly smaller
Q4_K_M     4.8    ★ sweet spot 7B-70B
Q5_K_S/M   5.5    preferred for roleplay
Q6_K       6.6    near-perfect
Q8_0       8.5    rarely needed

IQ2_XXS    2.1    imatrix, tiny, poor
IQ3_XXS    3.1    imatrix, small
IQ4_XS     4.3    imatrix, competitive

  ★  = community sweet spot
  below Q3: prose degrades noticeably
  above Q6: gains stop scaling with size

What you need before starting

• A model in Hugging Face safetensors format.
• Disk space for the F16 GGUF (~2x params in GB) plus each quant.
• Mostly CPU and RAM; a GPU accelerates imatrix computation but is not required for basic quantization.

Get the tools

llama.cpp is the standard - created by Georgi Gerganov, whose initials give GGML (and later GGUF) its name.

$ git clone https://github.com/ggml-org/llama.cpp
$ cd llama.cpp

$ cmake -B build -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --config Release -j

$ pip install -r requirements.txt

# produces binaries in build/bin/:
#   llama-quantize
#   llama-imatrix
#   llama-cli, llama-server (reference)

# platform flags (optional):
#   -DGGML_METAL=ON   ← Apple Silicon
#   -DGGML_CUDA=ON    ← CUDA GPU
# neither needed for CPU-only work

The core procedure

Two steps: convert the HuggingFace safetensors to an F16 GGUF, then quantize the F16 to a target level.

# stage 1: HF safetensors → F16 GGUF
$ python convert_hf_to_gguf.py \
    ./my-abliterated-8b/ \
    --outfile ./my-abliterated-8b-f16.gguf \
    --outtype f16

# stage 2: F16 GGUF → quantized target
$ ./build/bin/llama-quantize \
    ./my-abliterated-8b-f16.gguf \
    ./my-abliterated-8b-Q4_K_M.gguf \
    Q4_K_M

# always quantize from F16, never from an
# already-quantized file. requantizing
# compounds error and produces worse
# results than direct F16 → target.

imatrix - importance-weighted quantization

An importance matrix records which weights matter most, measured by running calibration text through the model. The quantizer then allocates precision where it reduces error most rather than uniformly. This gives significant quality gains at low bitrates (Q4 and below) and is strongly recommended for anything below Q6. It adds no inference overhead - nothing extra is stored in the file, only the quantization decisions are informed by the matrix.

mradermacher and bartowski publish imatrix-calibrated quants as a matter of course. The "i1" tag on quant names (e.g. mradermacher/*-i1-GGUF) indicates imatrix-calibrated variants.

# stage 1: compute the importance matrix
$ ./build/bin/llama-imatrix \
    -m  ./my-abliterated-8b-f16.gguf \
    -f  wiki.train.raw \
    -o  imatrix.dat

# calibration text: several MB of neutral
# text (wikitext or a curated multilingual
# corpus).

# stage 2: quantize with the matrix
$ ./build/bin/llama-quantize \
    --imatrix imatrix.dat \
    ./my-abliterated-8b-f16.gguf \
    ./my-abliterated-8b-IQ4_XS.gguf \
    IQ4_XS

# effect strongest at IQ3/IQ4 tiers.
# at Q6+ imatrix makes little difference.

huihui-ai's precision-preserving _L trick

huihui-ai preserves the ablation-affected tensors at higher precision during quantization, precisely to keep quantization from undoing the abliteration. Their published command uses llama-quantize's per-tensor override to force specific tensors (token_embd, output, ffn_down, ssm_out, attn_output) to Q8_0 (or BF16 in the Q8_0 build), renaming the file with an _L suffix.

# tensor_overrides.txt:
#   token_embd\..*         Q8_0
#   output\..*             Q8_0
#   .*\.ffn_down\..*      Q8_0
#   .*\.ssm_out\..*       Q8_0
#   .*\.attn_output\..*   Q8_0

$ ./build/bin/llama-quantize \
    --allow-requantize \
    --tensor-type-file tensor_overrides.txt \
    ./Huihui-Qwen3.8-27B-abl-f16.gguf \
    ./Huihui-Qwen3.8-27B-abl-Q6_K_L.gguf \
    Q6_K

# result: Q6_K everywhere EXCEPT the
# ablation-affected tensors, which stay
# at Q8_0. filename gets _L suffix.

# a Q2_K_L can end up larger than a plain
# Q3_K because of the preserved tensors.

The mradermacher pipeline

M8 has no inventor; quantization is general infrastructure. But it has a dominant producer: mradermacher, whose account hosts over 68,000 model repositories, the great majority GGUF and imatrix quantizations of other people's models. mradermacher's operation is a semi-automated pipeline run on company hardware (the model cards thank "my company, nethype GmbH, for letting me use its servers") with additional compute from a collaborator ("thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more imatrix quants").

The throughput is high enough that mradermacher functions as de facto distribution infrastructure for the abliterated ecosystem - most abliterated models a laptop user encounters were quantized through this pipeline rather than by the original abliteration producer. The predecessor role was played by TheBloke, whose quantization READMEs mradermacher's cards still reference. bartowski is the other major current M8 producer, particularly known for high-quality imatrix quants.

AWQ, GPTQ, EXL2, MLX - when to prefer

GGUF dominates local CPU/mixed inference, but other formats exist for other runtimes:

• GPTQ / AWQ - 4-bit GPU-side quantization for fast server inference (vLLM). Prefer when serving on GPUs rather than CPU. huihui-ai ships GPTQ-Int4 variants of 70B abliterated models.
• EXL2 - ExLlamaV2's variable-bitrate format, excellent quality-per-bit for NVIDIA GPU inference. Prefer for single-GPU enthusiast setups on NVIDIA.
• MLX - Apple Silicon native format. Prefer on Mac when running inside the MLX ecosystem.
• GGUF - default for CPU and mixed CPU/GPU inference, most widely consumed (LM Studio, Ollama, KoboldCpp).

When it does not work

Quantization interferes with abliteration. Symptom: low quants (Q2, Q3) refuse more than the F16 parent. Cause: aggressive quantization perturbs exactly the low-magnitude weight changes that abliteration relied on - the ablated tensors get rounded back toward their pre-ablation values, partly re-installing refusal through quantization noise. Diagnostic: refusal rate rises monotonically as quantization gets more aggressive. Fix: use imatrix, use huihui-ai's _L trick to keep ablated tensors at higher precision, or ship a larger quant.

Load errors on new architectures. Symptom: create_tensor: tensor 'output.weight' not found or similar. Cause: architecture not yet supported by your llama.cpp build. Fix: update llama.cpp; recently-released architectures may need a build from HEAD rather than a released version.

Garbled output after quantizing. Symptom: incoherent tokens, repeating patterns, garbage. Cause: quantized from an already-quantized source, or broken conversion (interrupted mid-write, wrong outtype). Fix: reconvert from F16 or safetensors source; verify F16 file loads correctly before quantizing.

Cost, time, file sizes

Conversion and quantization are CPU/RAM jobs. A 7B model converts and quantizes in minutes on a normal workstation. imatrix computation is heavier and benefits from a GPU (-ngl to offload layers). A 70B model is disk-heavy and RAM-heavy but still tractable on a workstation.

Observed file sizes for a 27B model (huihui-ai's Qwen3.8-27B):

• Q4_K: 16.8 GB
• Q4_K_L: 20.9 GB (larger due to preserved tensors)
• Q6_K: 22.4 GB
• Q8_0: 29 GB

For a 7B model at Q4_K_M: roughly 4-5 GB. For a 70B at Q4_K_M: roughly 40 GB. The general rule: at 4-bit K-quants, file size is roughly 0.6x the parameter count in GB (a 27B becomes ~16 GB, a 70B becomes ~40 GB).

Related literature

The Q3 2026 academic corpus review verifies no paper that directly evaluates the interaction between GGUF quantization and post-abliteration model behavior. This is one of the clearer coverage gaps in the field: quantization is treated as a solved engineering problem in the llama.cpp community, while the peer-reviewed literature that does exist on quantization (GPTQ, AWQ, EXL2 numerical work) predates the abliteration boom and does not test on abliterated targets.

Editorial note: the corpus review's working notes map three merging papers (Task Vectors arXiv:2212.04089, TIES-Merging arXiv:2306.01708, DARE arXiv:2311.03099) as method_mapping=M8. That is a taxonomy mismatch: those papers describe merging algorithms, which in our classifier is M5 (repackaging via GGUF quantization is a distinct operation). They are attributed correctly in M5 related literature.

What the M8 practitioner needs is documented in the tool codebases and blog posts already cited on this page: the llama.cpp quantization README for format specifications, mradermacher's Hugging Face profile for canonical output examples, and the forthcoming humanist explainer for the concept map. Coverage gap remains: no peer-reviewed measurement of whether low-bit quantization partially undoes abliteration (which the practical wiki notes as an operational concern), and no comparative study of Q4_K_M vs Q5_K_M vs Q6_K on refusal-benchmark output for abliterated models specifically.

Where to go next

Distribute: upload the GGUFs to Hugging Face for others to consume. Test: run the quantized file through the same refusal-count and capability-benchmark checks as the F16 parent - quantization can change both metrics, and low-bit quants can partly undo abliteration. See the practical how-to verification section. For the human-readable explainer of what all the quant names mean, see the upcoming E2 concept article.