license: apache-2.0
language:
- en
- zh
- multilingual
base_model: huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated
tags: - abliterated
- gemma
- gemma-4
- fp8
- fp8-dynamic
- compressed-tensors
- vllm
- multimodal
pipeline_tag: image-text-to-text
Huihui-gemma-4-26B-A4B-it-abliterated FP8-Dynamic
vLLM-compatible FP8-Dynamic quantization of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated.
The original BF16 release (~50 GB) is not directly servable by vLLM at production speed — --quantization fp8 runtime path is ~6× slower than pre-quantized FP8. This repo fills that gap: 27 GB FP8 weights, drop-in for vLLM, multimodal vision retained, abliteration profile preserved.
Smoke test (DGX Spark / GB10)
| Modality | Result |
|---|---|
| Text English (haiku) | ✅ coherent |
| Text 繁體中文 | ✅ humor + structure preserved |
| Vision (古風美女 portrait) | ✅ correctly described hanfu, bamboo, pose, expression |
| Audio | N/A — Gemma 4 26B-A4B-it has audio_config: null (vanilla design, not quant artifact) |
Speculative-decoding bench (DGX Spark, GB10, batch=1, T=0.0)
Tested pairing this model as MTP target with the vanilla google/gemma-4-26B-A4B-it-assistant draft model:
| Config | Acceptance | Throughput |
|---|---|---|
| Baseline (no spec) | n/a | 39.3 tok/s |
| MTP, num_speculative_tokens=4 | 40% token-level (per-pos: 65 / 43 / 29 / 21) | 52.4 tok/s (+33%) |
| MTP, num_speculative_tokens=1 | 69% | 52.6 tok/s (+34%) |
Verdict: MTP gives a real +33% gain even with vanilla draft head. num_speculative_tokens=1 is recommended — cleaner per-step acceptance (69% vs 40% token-level for n=4), same throughput. The per-position decay (65% → 21% over 4 positions) is consistent with huihui's abliteration shifting the body's prediction distribution; n=1 only uses the high-confidence first position.
For comparison, vanilla Gemma 4 + MTP achieves 108 tok/s on the same hardware (Part 27 reference). The ~50% gap is the "abliteration tax" — likely from a combination of FP8 calibration on shifted weights and MoE routing imbalance after abliteration. If you need raw speed without abliteration, use the vanilla google/gemma-4-26B-A4B-it directly.
vLLM serving
Recommended (with MTP, +34%)
vllm serve coolthor/Huihui-gemma-4-26B-A4B-it-abliterated-FP8-Dynamic \
--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":1}' \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.65 \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image":0,"audio":0,"video":0}' \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--trust-remote-code
Note: MTP requires vLLM with PR #41745's gemma4_mtp.py integration. Tested with the vllm/vllm-openai:gemma4-0505-arm64-cu130 image.
Usage (client side)
⚠️ This is an instruction-tuned model. Always use the chat completions endpoint.
curl -s -X POST http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "huihui-gemma4",
"messages": [{"role": "user", "content": "Write a haiku about an old API."}],
"max_tokens": 80,
"temperature": 0.7
}'
Do NOT use /v1/completions with a raw prompt — it bypasses the Gemma chat template (<start_of_turn>user ... <end_of_turn>\n<start_of_turn>model\n) and the model will fall off-distribution and loop garbage tokens. This is not abliteration / FP8 / MTP related; vanilla google/gemma-4-26B-A4B-it behaves the same way under /v1/completions. If you must use the completions endpoint, apply the chat template manually.
Quantization recipe
Used llmcompressor with FP8_DYNAMIC scheme. Critical ignore list — note re:.*router.* in particular: omitting this entry quantizes MoE router weights and breaks expert dispatch (verified by trial in our own testing):
ignore = [
"re:.*router.*", # MoE router — must NOT quantize
"lm_head",
"re:.*embed_tokens.*",
"re:.*norm.*", "re:.*layernorm.*", "re:.*layer_norm.*",
"re:.*rmsnorm.*", "re:.*rms_norm.*",
"re:.*conv1d.*", "re:.*linear_attn.*",
"re:visual.*", "re:model.visual.*",
"re:.*patch_embed.*", "re:.*vision.*", "re:.*image.*",
"re:.*video.*", "re:.*projector.*", "re:.*merger.*",
"re:.*mlp.gate$", "re:.*shared_expert_gate.*",
"re:.*embed_audio.*", "re:.*embed_vision.*",
"re:.*audio_tower.*", "re:.*audio_projector.*",
]
Quantization run: 2.9 minutes on GB10 (data-free pipeline, 30 MoE module calibration, FP8 scale derivation).
Toolchain notes
llmcompressor stable releases pin transformers <= 4.57.6, but Gemma4ForConditionalGeneration requires transformers 5.0+. Workaround: patch llmcompressor/entrypoints/utils.py to remove deprecated use_auth_token=... kwargs (8 occurrences) before reinstalling editable.
Credits
- Base abliteration: huihui-ai
- Original Gemma 4:
google/gemma-4-26B-A4B-it - MTP draft model:
google/gemma-4-26B-A4B-it-assistant - Quantization toolchain:
vllm-project/llm-compressor
License
Apache 2.0, inherited from base model and Gemma usage terms.
☕ If this saved you GPU hours, you can buy me a coffee.
📝 Quantized & benchmarked by ai-muninn — writeups on how it was built and how it actually runs.