license: gemma
base_model: Jiunsong/supergemma4-26b-abliterated-multimodal
tags:

• gemma4
• nvfp4
• quantized
• abliterated
• multimodal
• vllm
• compressed-tensors
• blackwell
• moe
  library_name: transformers
  pipeline_tag: image-text-to-text
  model_type: gemma4
  quantized_by: Lna-Lab

SuperGemma4-26B-Abliterated-Multimodal-NVFP4

NVFP4 quantized version of Jiunsong/supergemma4-26b-abliterated-multimodal — an aggressively abliterated, low-refusal multimodal Gemma 4 model with stronger coding, logic, and tool-use capabilities.

~50 GB → 16.5 GB with minimal quality loss. Runs on a single NVIDIA Blackwell GPU.

Why This Model

SuperGemma4 is not just abliterated — it's enhanced. Compared to the base Gemma 4:

• +8.2 on code benchmarks, +4.1 on logic
• 2x improvement in API tool-call success rate
• Multimodal (image + text) preserved at full precision
• Low-refusal for local agent workflows

This NVFP4 checkpoint brings all of that down to 16.5 GB — single-GPU inference at 145+ tok/s.

Key Specs

Quickstart

vLLM (recommended)

vllm serve Lna-Lab/SuperGemma4-26B-Abliterated-Multimodal-NVFP4 \
    --max-model-len 8192

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

Docker

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 8192

Python (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="Lna-Lab/SuperGemma4-26B-Abliterated-Multimodal-NVFP4",
    max_model_len=8192,
    gpu_memory_utilization=0.85,
)

output = llm.generate(
    ["Explain quantum entanglement in simple terms."],
    SamplingParams(max_tokens=256, temperature=0.7),
)
print(output[0].outputs[0].text)

Benchmark

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), CUDA Graph PIECEWISE mode.

Sustained throughput: ~145 tok/s (post-warmup, single GPU).

Quantization Details

Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*router', 're:.*vision_tower.*']
      scheme: NVFP4

What's quantized, what's not

• Quantized (NVFP4): All Linear layers in the language model, including MoE expert layers
• Kept in BF16: lm_head, all embedding layers, MoE routers, entire vision tower

Calibration

• Dataset: neuralmagic/calibration (LLM split)
• Samples: 20
• Max sequence length: 8192
• MoE expert calibration handled automatically by llm-compressor's SequentialGemma4TextExperts

Reproduction

from datasets import load_dataset
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = Gemma4ForConditionalGeneration.from_pretrained(
    "Jiunsong/supergemma4-26b-abliterated-multimodal", dtype="auto"
)
processor = AutoProcessor.from_pretrained(
    "Jiunsong/supergemma4-26b-abliterated-multimodal"
)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:20]")

def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, return_tensors="pt", padding=False, truncation=True,
        max_length=8192, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False,
    )

ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

import torch
def data_collator(batch):
    assert len(batch) == 1
    return {
        key: (torch.tensor(value) if key != "pixel_values"
              else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
        for key, value in batch[0].items()
    }

oneshot(
    model=model, recipe=recipe, dataset=ds,
    max_seq_length=8192, num_calibration_samples=20,
    data_collator=data_collator,
)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

Requirements

• GPU: NVIDIA Blackwell (RTX 5090, RTX PRO 6000, B200, etc.) — NVFP4 requires SM 120
• VRAM: ~16 GB minimum
• Software: vLLM nightly (cu130 build), or any framework supporting compressed-tensors NVFP4

Notes

• This is an abliterated (uncensored) model. Use responsibly.
• Vision tower is kept in BF16 — multimodal capabilities are preserved at full precision.
• NVFP4 is a Blackwell-specific format. This checkpoint will not work on Ampere/Hopper GPUs.

Credits

• Base model: Jiunsong (SuperGemma4 abliteration + enhancement)
• Original model: Google DeepMind (Gemma 4)
• Quantization tool: vllm-project/llm-compressor
• Quantization method follows RedHatAI/gemma-4-26B-A4B-it-NVFP4 (proven path)

Support the Base Model Author

If you find this model useful, please consider supporting Jiunsong — the creator of SuperGemma4:

• Ko-fi: https://ko-fi.com/jiunsong