license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
base_model:

• google/gemma-4-31B-it
  base_model_relation: quantized
  pipeline_tag: text-generation
  library_name: vllm
  tags:
• gemma
• gemma-4
• abliterated
• uncensored
• nvfp4
• fp4
• quantized
• compressed-tensors
• vllm
• blackwell
• speculative-decoding
• mtp

Gemma-4-31B-it — Abliterated + NVFP4

An abliterated (refusal-direction removed) build of
google/gemma-4-31B-it,
quantized to plain NVFP4 (4-bit weights and activations) with
llm-compressor / compressed-tensors, for native FP4 inference on
NVIDIA Blackwell (sm_120) GPUs under vLLM.

⚠️ Safety notice. Abliteration removes the model's learned refusal
behavior. This model will attempt to answer prompts the original -it model
would decline. It is intended for research and use on your own hardware.
You are responsible for how you use it; see Intended use & limitations below.

Why this build exists

At the time it was made, no public checkpoint combined all three of:

1. abliterated / uncensored,
2. plain NVFP4 (not NVFP4_AWQ, which stock vLLM rejects), and
3. x86 Blackwell–runnable (not an ARM-only DGX Spark image).

Existing NVFP4 Gemma-4 builds were either the stock/censored model, the
NVFP4_AWQ variant, or shipped only in ARM-only images. This repo fills that
gap by self-quantizing a bf16 abliterate to plain NVFP4.

What was done (modifications from the base model)

This is a modified derivative of google/gemma-4-31B-it. Two changes:

1. Abliteration — directional ablation / weight orthogonalization
   (Arditi et al., 2024). The refusal direction is estimated from mean
   last-token residual activations on matched harmful vs. harmless prompts,
   then orthogonalized out of every residual-writing weight (embed_tokens,
   per-layer attention o_proj and MLP down_proj). The ablation is baked
   into the weights — no inference-time hooks required.
2. NVFP4 quantization — one-shot PTQ (E2M1 elements with FP8 block scales)
   on the text decoder's Linear layers only. The vision tower,
   multimodal projector, audio modules, token embeddings and the (tied)
   lm_head are left in higher precision. See recipe.yaml in this repo for
   the exact scheme and ignore list.

Hardware / software requirements

• GPU: NVIDIA Blackwell with native FP4 tensor cores (sm_120, e.g. RTX PRO
  Blackwell). NVFP4 activation quantization needs hardware FP4 support.
• Serving: a recent vLLM with compressed-tensors NVFP4 support.
• ~20 GB on disk; weights fit comfortably on a single 24 GB+ card (leave KV-cache
  headroom), or use tensor parallelism.

Serving with vLLM (+ optional MTP speculative decoding)

Gemma 4 ships an official EAGLE/MTP-style draft,
google/gemma-4-31B-it-assistant,
which vLLM drives as a native multi-token speculator. Use "method": "mtp"
(passing "draft_model" for a Gemma-4 assistant silently disables MTP):

vllm serve aday777/gemma-4-31B-it-abliterated-NVFP4 \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.88 \
    --speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 3, "method": "mtp"}' \
    --port 8003

Drop the --speculative-config line to serve without speculative decoding.

Intended use & limitations

• Intended use: local research, red-teaming, evaluation, and applications
  where you supply your own guardrails.
• No safety filtering: refusal behavior has been removed; this model can
  produce harmful, offensive, or otherwise objectionable content. It is not
  suitable for unsupervised or public-facing deployment without your own safety
  layer.
• Quantization: 4-bit weights and activations trade some quality for speed
  and memory; expect small accuracy differences from the bf16 model.
• All original capability limitations of gemma-4-31B-it still apply.

License & attribution

Derived from google/gemma-4-31B-it by Google DeepMind, licensed under the
Apache License 2.0 (see Gemma 4 license).
This derivative is distributed under the same license; a copy of the Apache 2.0
License is included as LICENSE. Per the license, note that these files have
been modified from the original (abliterated and NVFP4-quantized as described
above). Please also review Google's Gemma prohibited-use policy.

Reproduction

The model was produced with directional-ablation + llm-compressor NVFP4 PTQ.
The exact quantization recipe is in recipe.yaml. Abliteration calibration used
matched harmful/harmless instruction sets and selected the refusal direction by
the layer that most reduced refusals on a validation split.