base_model: huihui-ai/Huihui-GLM-4.7-Flash-abliterated
base_model_relation: quantized
quantized_by: cognitivers
library_name: gguf
pipeline_tag: text-generation
license: mit
language:
- en
- zh
tags: - gguf
- llama.cpp
- abliterated
- uncensored
- moe
- glm4
- imatrix
GLM-4.7-Flash-abliterated, 12 GB GGUF
GGUF quantizations of huihui-ai/Huihui-GLM-4.7-Flash-abliterated,
built for one specific target: a 12 GB consumer GPU with the routed experts offloaded to system RAM.
These are not another uniform ladder. The bit budget is allocated by the role each tensor plays at
inference time, which is where a Mixture-of-Experts model leaves a lot on the table.
Why a split-aware mix
GLM-4.7-Flash is a 29.94B MoE with roughly 3.3B active parameters per token. llama.cpp's--cpu-moe / --n-cpu-moe moves exactly \.ffn_(up|down|gate|gate_up)_(ch|)exps to the CPU.
Everything else stays resident on the GPU forever:
| Block | Params | Share of model | Lives on |
|---|---|---|---|
| MLA attention (47 layers) | 1.02B | 3.3 % | GPU |
| Shared experts + dense layer 0 | 0.50B | 1.6 % | GPU |
| Token embeddings + output head | 0.63B | 2.0 % | GPU |
| Routed experts (64 x 46) | 28.2B | 90.4 % | RAM, or GPU via -ncmoe |
The GPU-resident part is only 2.15B parameters. Keeping it at q8_0 costs about 2.3 GB, which is
cheap on a 12 GB card. A uniform ladder spends the same bits per weight on those 2.15B as on the
28.2B of routed experts, and that is quality given away for nothing.
So: q8_0 for the router, the MLA attention, the shared experts, the dense layer and the output
head; q6_k for the token embeddings; and the routed experts compressed hard. The first four MoE
layers get one extra step, since early layers tolerate low-bit worse.
Files
| File | Size | Routed experts | Intended use |
|---|---|---|---|
GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf |
18.2 GB | q4_k (first 4 layers q5_k) |
Recommended. 12 GB VRAM + ~16 GB RAM |
GLM-4.7-Flash-abliterated.IQ2_M-SPLIT.gguf |
11.3 GB | iq2_s (first 4 layers iq3_xxs) |
Fits entirely in 12 GB VRAM, at a measured reasoning cost |
imatrix.gguf |
72 MB | The importance matrix used, so the recipe is reproducible |
Both were quantized from a BF16 conversion of the original safetensors (never re-quantized from a
smaller GGUF), with an importance matrix computed on calibration_datav3 over 125 chunks.
How to run it on 12 GB
# Recommended: 26.3 tok/s, 10.19 GB VRAM, 1.68 GB headroom
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 24 -c 8192 -fa on
# Aggressive: 33.5 tok/s, but only 0.39 GB of VRAM left. Long prompts may OOM.
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -ncmoe 20 -c 8192 -fa on
# Minimum VRAM: all experts in RAM, 2.82 GB on the card, 16.4 tok/s
llama-server -m GLM-4.7-Flash-abliterated.Q4_K-SPLIT.gguf -ngl 99 -cmoe -c 8192 -fa on
-ncmoe N keeps the routed experts of the first N layers (out of 46) in system RAM. Lower N means
more experts on the GPU and more speed, until you run out of VRAM.
Measured on an RTX 4070 Ti (11874 MiB usable), context 8192, llama-bench with -p 512 -n 128 -r 2.-ncmoe 18 and below abort with a CUDA allocation error on this card.
Quality
Everything below is measured against a BF16 conversion of the same checkpoint, not against a
smaller quantization, on wikitext-2-raw test (sha256 173c87a5..., 565 chunks, ctx 512).
| Model | Size | PPL ratio | Mean KLD | Median KLD | KLD p99 | Same top-token |
|---|---|---|---|---|---|---|
Q8_0 (reference point) |
31.8 GB | 1.017 | 0.0298 | 0.0020 | 0.106 | 96.43 % |
Q4_K-SPLIT |
18.2 GB | 1.006 | 0.0871 | 0.0088 | 0.633 | 92.63 % |
IQ2_M-SPLIT |
11.3 GB | 1.166 | 0.2302 | 0.0723 | 2.993 | 82.35 % |
A PPL ratio near or below 1.0 is not evidence of being better than the reference. Low-bit
quantization can smooth a corpus and lower perplexity while still diverging from the original
distribution, which is exactly why top-token agreement and KLD are reported next to it.
Reasoning is not free below ~3 bits per expert
GSM8K, 300 problems, paired against the same baseline, bootstrap CI over paired per-item differences:
| Model | Accuracy drop | 95 % upper bound | Verdict at a 3 pp bar |
|---|---|---|---|
Q4_K-SPLIT |
0.33 pp | 2.33 pp | passes |
IQ2_M-SPLIT |
5.33 pp | 8.00 pp | fails |
Chain-of-thought length did not inflate (+1.8 %, upper bound +8.8 %) and tool-calling was unchanged
(call rate, JSON validity and argument correctness all 1.00 in both).
Be aware of what this means: the 11.3 GB file measurably costs you reasoning accuracy. It is
published because it is the best option we could measure at that size, not because it is lossless.
If you can spare the system RAM, use the 18.2 GB file with -ncmoe.
Head to head at the same size
The natural comparison is mradermacher/Huihui-GLM-4.7-Flash-abliterated-i1-GGUF, a
well-made imatrix ladder over the same abliterated checkpoint. Its i1-IQ3_XXS (11.65 GB) is
the closest neighbour in size to our 11.30 GB file, so we are competing 0.35 GB smaller.
Both were measured in the same session, against the same BF16 base logits, with the same llama.cpp
build and the same corpus. Numbers published elsewhere are not comparable to these; these are.
Ours, IQ2_M-SPLIT |
i1-IQ3_XXS |
|
|---|---|---|
| Size | 11.30 GB | 11.65 GB |
| Mean KLD vs BF16 | 0.2302 | 0.3916 |
| Median KLD | 0.0723 | 0.1749 |
| KLD p99 | 2.993 | 4.201 |
| Same top-token | 82.35 % | 75.25 % |
| GSM8K drop | 5.33 pp | 9.33 pp |
Smaller file, 41 % lower mean KLD, 7.1 points more top-token agreement, and 43 % less damage to
reasoning. That is the case for allocating bits by role instead of uniformly.
The abliteration survives quantization
This matters more than perplexity for a model published as abliterated, and it is the one claim a
quantizer can easily break without noticing.
Measured with a refusal-onset detector over StrongREJECT-small (n=60), identical protocol for all
three models, including a positive control:
| Model | Refusal rate |
|---|---|
zai-org/GLM-4.7-Flash (unmodified base, positive control) |
93.33 % |
Q4_K-SPLIT |
1.67 % |
IQ2_M-SPLIT |
3.33 % |
The control matters. A quantization scoring 0 % refusals proves nothing on its own, because a broken
detector also scores 0 %. Here the same detector fires at 93.3 % on the unmodified base model, so
the low scores are evidence rather than an artifact.
Scope, stated honestly: this is a refusal-onset detector built on lexical markers over the first
400 characters of the reply. It measures whether the model starts refusing. It is not the
StrongREJECT harmfulness grader and it is not a safety evaluation. It supports exactly one claim,
that quantization did not restore refusal behaviour, and nothing beyond that.
Reproducibility notes
- Architecture:
Glm4MoeLiteForCausalLMconverts to GGUF archdeepseek2with native MLA. The MTP
head is excluded (--no-mtp) and can be exported separately as a speculative draft. - The
Q8_0andBF16conversions came out byte-for-byte identical when produced independently
on an A40 and on an A100 in different datacenters, so the source chain is deterministic. imatrix.ggufis included so the recipe can be reproduced or extended.
Known gaps
Published deliberately with these stated rather than hidden:
- Behavioural metrics (GSM8K, tool-calling) were measured against a
Q8_0of the same checkpoint
rather than the BF16. In this model that proxy tracked the BF16-referenced KLD to within 1-9 % with
an identical ranking, but it is a proxy. - Divergence@32 is an internal proxy set (GSM8K + MBPP + fixed prompts), not the published
Divergence-300@32; absolute values are not comparable to anyone else's. - Long-context behaviour is untested. The model supports 202k context; nothing here was measured
above 8192. - A
Q8-vs-Q8weight-space comparison against the unmodified base, which would quantify how much
the abliteration changed, has not been run. The refusal delta above establishes it functionally.
Credits
Base model by huihui-ai, built onzai-org/GLM-4.7-Flash. Quantized by
cognitivers. mradermacher's ladder was used as the comparison baseline and is a fine choice if
you want a conventional set of sizes.