base_model: huihui-ai/Huihui-GLM-5.2-abliterated
base_model_relation: quantized
license: mit
library_name: llama.cpp
pipeline_tag: text-generation
tags:

• gguf
• glm
• moe
• abliterated
• uncensored
• iq1_s
• mixed-precision
• llama.cpp

Huihui-GLM-5.2-abliterated — Q8_0 + IQ1_S routed experts (GGUF)

A 744B-parameter abliterated MoE squeezed to 156.5 GiB (1.78 BPW) — runs at ~6 tok/s on a single desktop with 64 GB VRAM + 128 GB RAM.

This is a mixed-precision requant of huihui-ai/Huihui-GLM-5.2-abliterated built for one purpose: making the full model RAM-resident on a 128 GB machine, so decode speed is bounded by memory bandwidth instead of NVMe. No REAP/pruning, all 256 experts and all 8 active experts per token are intact.

The recipe

• Total: 156.5 GiB, 1.78 BPW effective
• Source: Huihui's UD-Q3_K_M shards, requantized with llama-quantize --allow-requantize
• Why the Unsloth imatrix is valid for the abliterated model: Huihui's README states the first 12 layers and all expert modules were not ablated — the expert tensors are bit-identical to the original GLM-5.2, so the original-model imatrix applies exactly where it's used (the experts).

The philosophy is the inverse of a uniform low-bit quant: spend bits where every token flows (attention + shared expert at Q8), starve the sparsely-activated experts (only 8 of 256 fire per token, so per-expert error partially averages out across the ensemble).

Measured performance (the reason this exists)

Rig: RTX 5090 32 GB + 2× RTX 5060 Ti 16 GB (64 GB VRAM), 128 GB DDR5, Core Ultra 9 285K, Windows 11, llama.cpp b10241.

Verified zero disk I/O during generation (working set 105.8 GB fully page-cached). ~32 GiB of expert layers pinned to VRAM, the remaining ~102 GiB of experts live in system RAM.

How to run it (llama.cpp)

llama-server.exe ^
  -m Huihui-GLM-5.2-abliterated-Q8-IQ1S-exps-00001-of-00004.gguf ^
  -ngl 999 --split-mode layer -ts 30,16,16 --main-gpu 0 ^
  -ot "blk\.([3-9]|10)\.ffn_.*_exps\.=CUDA0,blk\.(3[8-9]|4[0-2])\.ffn_.*_exps\.=CUDA1,blk\.(5[8-9]|6[0-2])\.ffn_.*_exps\.=CUDA2,exps=CPU" ^
  -fa on -ctk q8_0 -ctv q8_0 ^
  -c 32768 -b 2048 -ub 512 --no-warmup ^
  -t 24 --cpu-strict 0 --prio 2 --threads-batch 24 ^
  --kv-unified ^
  --jinja --temp 1.0 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.05 ^
  --host 127.0.0.1 --port 8080

Adapt to your rig:

• -ot pins whole expert layers into leftover VRAM (exps=CPU last as the catch-all). One expert layer ≈ 1.76 GiB here. More VRAM → pin more layers → faster. On some llama.cpp builds only the last -ot flag is honored — put all rules in ONE comma-separated flag.
• --no-warmup matters: warmup touches all 256 experts per layer and thrashes the page cache.
• Use all physical cores for decode once RAM-resident (P+E cores both help; this was disk-latency-bound before, where fewer threads won).
• Single-GPU or CPU-only works too — just drop -ts/-ot pins accordingly. Minimum ~112 GB free RAM for the CPU-side experts, plus VRAM/RAM for the rest.
• Quality guardrails recommended at 1-bit experts: --min-p 0.05 --repeat-penalty 1.05.

Quality expectations

Be realistic: the routed experts are 1.58-bit. Unsloth's GLM-5.2 measurements put 1-bit expert quants around ~76% of an aggressive-benchmark score vs ~82% for 2-bit — degraded, not lobotomized. Mitigations baked in here: Q8 attention path (better than typical uniform 1-bit quants), all 8 experts active, imatrix-guided. In practice: coherent multi-paragraph prose, solid instruction-following, working chat template. Use the biggest quant your hardware can hold resident — this one targets the 128 GB RAM + 48–64 GB VRAM class where the alternative is NVMe streaming at ~1 tok/s.

The abliteration is fully preserved — expert tensors were never ablated in the source, and the ablated non-expert tensors are carried at Q8_0.

Credits

• zai-org — GLM-5.2 base model
• huihui-ai — abliteration
• unsloth — imatrix and their excellent GLM-5.2 quant writeups
• llama.cpp — the whole engine