license: mit
language:
- en
- zh
base_model: - huihui-ai/Huihui-GLM-4.7-Flash-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
tags: - glm4_moe_lite
- fp8
- w8a8
- compressed-tensors
- vllm
- abliterated
- uncensored
GLM-4.7-Flash-abliterated-FP8-Dynamic
FP8 (W8A8, dynamic) quantization of huihui-ai/Huihui-GLM-4.7-Flash-abliterated,
itself an abliterated finetune of zai-org/GLM-4.7-Flash
(a 30B-A3B glm4_moe_lite MoE model).
Quantized by mlavkin.
What this is
- Weights: FP8 E4M3, per-channel scales.
- Activations: FP8 dynamic, per-token (W8A8).
- Kept in BF16 (not quantized): all MoE router gates (
*.mlp.gate) andlm_head— standard for MoE to preserve routing quality. - Format:
compressed-tensors(float-quantized). - Size: ~29 GB (vs ~59 GB BF16).
Why
FP8 halves weight memory traffic. On memory-bandwidth-bound hardware this yields a large decode speedup with negligible quality loss. Measured on an NVIDIA GB10 (DGX Spark), single-stream decode vs BF16:
| Context | BF16 | FP8 | Speedup |
|---|---|---|---|
| ~0.6k | 25.0 | 41.3 | 1.65× |
| ~13k | 21.7 | 33.6 | 1.55× |
| ~33k | 18.2 | 25.6 | 1.41× |
Quality spot-checks (arithmetic, factual, code, needle-in-haystack retrieval up to ~47k) matched BF16.
Usage (vLLM)
Requires a recent vLLM + Transformers with glm4_moe_lite support.
vllm serve operationrange/GLM-4.7-Flash-abliterated-FP8-Dynamic \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice \
--max-model-len 202752 \
--host 0.0.0.0 --port 8000
The model's native context is 202,752 tokens. RoPE scaling (e.g. YaRN) can extend it further but noticeably degrades coherence — native length is recommended.
Quantization recipe
Produced with llm-compressor:
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
model = AutoModelForCausalLM.from_pretrained(SRC, dtype="bfloat16", device_map="cpu")
recipe = QuantizationModifier(
targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*mlp.gate$", "re:.*eh_proj$"],
)
oneshot(model=model, recipe=recipe, output_dir=OUT)
FP8_DYNAMIC needs no calibration data (data-free).
License
MIT, inherited from the base models. Abliteration by huihui-ai; original model by Z.ai (zai-org).