language: [sv, en]
license: gemma
license_name: gemma
library_name: vllm
pipeline_tag: text-generation
tags:

• gemma
• gemma-4
• multimodal
• vision
• audio
• nvfp4
• w4a16
• modelopt
• vllm
• speculative-decoding
• mtp
• long-context
• 131k
• tool-calling
• uncensored
• blackwell
• rtx-5090
  base_model: HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive
  quantization_bit: 4

Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4

A 9.7 GB multimodal (vision + text + audio) checkpoint that fits on a single RTX 5090 and decodes at ~430 tok/s with MTP speculative decoding enabled.

Name breakdown: Gemma-4-E4B (base) → Uncensored-HauhauCS-Aggressive (base fine-tune) → MTP (native multi-token-prediction drafter support, --spec-tokens 5 recommended) → NVFP4 (W4A16 ModelOpt quantization of the text transformer).

This is a community-built, hand-assembled checkpoint: the HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive fine-tune (GGUF) was converted to HF format, quantized to NVFP4 (W4A16) with NVIDIA ModelOpt, and re-assembled with the base model's multimodal encoders (vision tower + audio encoder kept in BF16).

It is not an official Google or HauhauCS release. It exists because nobody had published a native multimodal NVFP4 build of this model that serves with stock vLLM on consumer Blackwell.

What's inside

Files: single model.safetensors (9.7 GB, 3,032 tensors) + tokenizer/processor/config.

Serve with vLLM

Verified on vLLM 0.28, Python 3.12, CUDA 13, RTX 5090 (32 GB, shared). Requires the nvidia/cu13 CUDA_HOME layout for FlashInfer:

export CUDA_HOME=$VENV/lib/python3.12/site-packages/nvidia/cu13
export FLASHINFER_EXTRA_CUDAFLAGS="-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK=1"

vllm serve Adolphsson/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4 \
  --quantization modelopt_fp4 \
  --port 8000 --host 0.0.0.0 \
  --served-model-name gemma4-e4b-nvfp4 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.72 \
  --max-num-seqs 8 \
  --limit-mm-per-prompt '{"image":4,"video":2,"audio":4}' \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --enable-prefix-caching \
  --mm-processor-cache-gb 2

With the av (PyAV) + soundfile packages installed in the venv, all three modalities are accepted at the API level (pip install av soundfile).

One patch required for audio input (vLLM 0.28 + transformers 5.6): the quantized audio encoder hits a non-contiguous tensor in Gemma4ClippedLinear. In transformers/models/gemma4/modeling_gemma4.py, change the Gemma4ClippedLinear.forward line:

hidden_states = self.linear(hidden_states.contiguous())   # was: self.linear(hidden_states)

This is the only quantized multimodal tower (vision/embeddings are excluded from NVFP4), which is why image input works unpatched but audio input crashes without it (engine death: marlin_gemm … A.stride(1) is not 1).

MTP speculative decoding (~2x decode speed)

This model ships with native MTP support in vLLM 0.28 (Gemma4MTP). Add:

  --spec-model google/gemma-4-e4b-it-assistant \
  --spec-tokens 5

Measured on this checkpoint (RTX 5090, vLLM 0.28, 3 reps):

• K=5 is the measured optimum — the drafter has 4 layers; at K=6 it re-runs the same layers per iteration (note the accepted tok/iter dropping despite one more draft position), with diminishing returns beyond.
• VRAM cost of the drafter: ~0.1–0.3 GB.
• For very short outputs (<~200 tokens) MTP can be a slight *loss* (drafter overhead). For typical assistant/chat workloads (>300 token answers) it is a clean ~2x.
• Table measured with thinking off (chat-template default). With enable_thinking: true the model produces a full reasoning channel and decodes it even faster (measured 624–700 tok/s with MTP K=5) — thought tokens are far more predictable, so drafter acceptance rises. Reasoning-on throughput is therefore higher, not lower, than the table.

Quick test

curl http://localhost:8000/v1/chat/completions -d '{
  "model": "gemma4-e4b-nvfp4",
  "messages": [{"role":"user","content":"Write a short poem about cats"}],
  "max_tokens": 300, "temperature": 0.7
}'

Multimodal (vision) via the OpenAI-compatible input_image content part; audio via input_audio (see known issue below).

Known issues (be honest section)

1. Audio input required a 1-line patch (fixed, documented above): Gemma4ClippedLinear.forward in transformers' gemma4 module — add .contiguous() before the quantized linear. Without it the engine dies on the first input_audio request. Verified: transcription, name extraction, and word counting all work with the patch (MTP on or off).
2. Uncensored fine-tune behavior: inherits the HauhauCS-Aggressive training profile (direct, less cautious phrasing). Same model, quantized — expect the same personality as the GGUF.
3. NVFP4 calibration: quantized with ModelOpt defaults on this checkpoint; text quality is excellent in practice, but per-linear weight_scale_2 divergence across fused QKV is present (vLLM logs a warning) — typical for NVFP4 Gemma-class models.
4. Reasoning channel is raw-text, not structured: with chat_template_kwargs: {"enable_thinking": true} the model does produce a full reasoning channel (verified, decoding at 624–700 tok/s under MTP — faster than non-thinking because thought tokens are highly predictable). How it is consumable per interface:
   • Chat API + streaming: the thought text arrives as plain content deltas (with a literal "thought\n" prefix to strip). Works in real time — the recommended path for voice/latency pipelines.
   • Chat API, non-streaming: both reasoning and content come back empty — the parser discards the unclosed thought channel.
   • /v1/completions: full raw text in text, fully consumable.
   • The reasoning field is never populated (the uncensored fine-tune emits thought-channel markers the vLLM 0.28 gemma4 parser doesn't map to structured fields). Clients that render thinking in a separate UI panel get an empty panel — consume content and strip the thought prefix instead.

Benchmarks & provenance

Full benchmark methodology, per-position acceptance curves, and raw JSON results: see the author's notes linked below.

• Built: August 2026, on an RTX 5090 LXC container
• Base fine-tune: HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive (GGUF Q8_K_P)
• Encoders/config: google/gemma-4-e4b
• Quantizer: NVIDIA ModelOpt 0.46.0 (W4A16_NVFP4, group size 16)
• Serving: vLLM 0.28 (--quantization modelopt_fp4)
• MTP drafter: google/gemma-4-e4b-it-assistant (separate, 159 MB)

License

gemma — derivative of the Gemma 4 E4B base model and the HauhauCS uncensored fine-tune. Subject to the Gemma Terms of Use.