Huihui-gemma-4-31B-it-abliterated-v2-MXFP4

MXFP4-quantized version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2

Quantization Details

Quantized Layers (~1,571 layers)

• Main LLM attention layers (q_proj, k_proj, v_proj, o_proj)
• MLP layers (gate_proj, up_proj, down_proj)
• Embed tokens

BF16 Layers (192 layers)

• Vision tower (26 encoder layers)
• Vision embeddings
• LM head

Comparison: Original vs Quantized

Memory Requirements

• Original (BF16): ~62 GB VRAM for inference
• Quantized (MXFP4): ~20 GB VRAM for inference (~3x less)

Inference Speed (Estimated)

With Blackwell GPU (GB10/B200) native FP4 tensor cores:

• Throughput: ~2-3x faster decoding vs BF16
• Pre-fill: Similar or slight improvement

Quality (PPL)

Note: Standard perplexity evaluation via HuggingFace transformers is not supported because MXFP4 decompression is not implemented in transformers. However, vLLM handles the decompression transparently during inference, so output quality should be equivalent to the original model.

The high PPL values seen in direct API perplexity tests are due to the model's thinking/reasoning mode interfering with token prediction - this is a model-specific behavior, not a quantization issue.

Original Model

Source: huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2

Base model: google/gemma-4-31B-it

Model Information

About This Model

This is an abliterated (uncensored) version of google/gemma-4-31B-it created using abliteration to remove refusals.

PPL (Perplexity) Comparison:

Lower PPL = better quality

Usage Warnings

• ⚠️ Significantly reduced safety filtering - may generate sensitive/inappropriate content
• Not suitable for minors or public-facing applications
• Users bear full responsibility for any consequences
• Recommended for research/testing only

Usage with vLLM

Basic Inference

NOTE: Do NOT specify --quantization mxfp4 - the quantization format is auto-detected from the model config.

vllm serve ./Huihui-gemma-4-31B-it-abliterated-v2-MXFP4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.85

Or in Python:

from vllm import LLM

llm = LLM(
    model="./Huihui-gemma-4-31B-it-abliterated-v2-MXFP4",
    # DO NOT specify quantization - auto-detected from compressed-tensors config
    trust_remote_code=True,
    gpu_memory_utilization=0.85,
    max_model_len=32768,
)

Enable Thinking/Reasoning Mode

To enable the model's built-in thinking/reasoning capability:

vllm serve ./Huihui-gemma-4-31B-it-abliterated-v2-MXFP4 \
  --trust-remote-code \
  --max-model-len 16384 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}'

Or in Python:

from vllm import LLM

llm = LLM(
    model="./Huihui-gemma-4-31B-it-abliterated-v2-MXFP4",
    trust_remote_code=True,
    max_model_len=16384,
    enable_auto_tool_choice=True,
    chat_template="examples/tool_chat_template_gemma4.jinja",
    default_chat_template_kwargs={"enable_thinking": True},
)

Troubleshooting

Error: "Quantization method specified in the model config (compressed-tensors) does not match..."

This means you're explicitly specifying --quantization mxfp4 when it's not needed. Simply remove the --quantization argument - vLLM auto-detects the compressed-tensors format from the model config.

License

Apache-2.0 (same as base model)