license: apache-2.0
base_model:
- TrevorJS/gemma-4-12B-it-uncensored
tags: - gemma4
- nvfp4
- mxfp8
- mixed-precision
- unsloth-dynamic
- compressed-tensors
- vllm
- multimodal
- voice
- audio
- speech-recognition
- vision
- speculative-decoding
pipeline_tag: any-to-any
Gemma 4 12B Unified IT (Uncensored) — Grand Hybrid (NVFP4 + MXFP8, Audio & Vision Preserved)
Pareto-optimal mixed-precision compression of TrevorJS/gemma-4-12B-it-uncensored built for single 16 GB consumer GPUs (RTX 5060 Ti / sm_120, Thor sm_110, Ada/Hopper). Unlike naive uniform 4-bit quantizations that collapse continuous speech and acoustic features, the Grand Hybrid protects audio/vision towers and early transformer layers in BF16, achieving BF16-parity speech transcription & visual understanding at 10.84 GiB (11 GB on disk).
Companion Speculative Predictors (catplusplus/)
Pair this base model with either of our calibrated/tuned predictors depending on your latency, concurrency, and KV-cache budget:
| Predictor | Method & Tokens | Checkpoint Size | Free FP8 KV Cache (16 GB GPU) | Direct Chat (tech/arch/jp_en) |
Reason (think=True) |
Coding Decode (Code Accept) |
Tasuki Agent (ToolGuard / Clarifier) |
Concurrency C=1 / C=4 |
|---|---|---|---|---|---|---|---|---|
| None (Target Only) | — | — | 84,209 tok (0.96 @ 32K) |
33.8–33.9 tok/s | 34.0 tok/s | 33.7–33.9 tok/s (1.00) |
33.8 (1.62s) / 33.9 (1.23s) |
33.7 / 132.7 tok/s |
catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8 (Recommended All-Rounder) |
mtp, K=5 |
420.65 MB | 254,059 tok (0.98, auto) |
71.3–78.1 tok/s (54–85 ms TTFT) |
130.4 tok/s | 102.5–141.9 tok/s (4.188) |
98.1 (0.63s) / 73.8 (0.70s) |
74.3 / 285.8 tok/s |
catplusplus/dspark-gemma-4-12b-it-Grand-Hybrid-Unsloth-Dynamic-NVFP4-MXFP8 (Burst Code / Structured JSON) |
dspark, K=7 |
1,257.81 MB | 66,714 tok (0.98, auto) |
59.0–79.0 tok/s (89–175 ms TTFT) |
131.1 tok/s | 108.0–143.3 tok/s (4.926) |
156.0 (0.45s) / 60.0 (1.01s) |
70.7 / 227.6 tok/s |
Key Architectural Differences Between the Two Predictors
- KV-Cache Footprint (
254Kvs66.7Ktokens):- MTP (
gemma-4-12B-it-assistant) uses 4 lightweight Q-only decoder layers (hidden_size=1024) that share target KV caches from layers 46 & 47—allocating zero extra KV cache, leaving 254,059 FP8 KV tokens free on a 16 GB GPU. - DSpark Block-7 (
dspark-gemma-4-12b) runs 5 full-width (hidden_size=3840) decoder layers + cross-layerfcbridge + parallel Markov transition head (embed_tokens/lm_headshared viashare_target_embeddings: true, omitting 4.03 GB of duplicate weights), allocating its own 5-layer KV cache and leaving 66,714 KV tokens.
- MTP (
- Decoding Pattern:
- MTP (
K=5) runs 5 tiny sequential steps (~52–86 msdirect TTFT) and wins direct chat, Japanese↔English translation, multi-stream concurrency (285.8 tok/satC=4), and long context. - DSpark (
K=7) drafts 7 tokens in a single parallel block-diffusion pass followed by rank-256 Markov refinement, tuned on-policy with QAT on this exact base model (+0.51accept len /+11.9 pppos-0 on held-out autoregressive validation), excelling at low-entropy coding chains (4.93tokens/step) and structured JSON agent verdicts (156.0 tok/s,0.45s). (Requires vLLMGemma4DSparkModelwithshare_target_embeddings: truesupport.)
- MTP (
Architectural Precision Map (Audio & Vision Preserved)
| Layer Category | Modules | Precision Scheme | Rationale |
|---|---|---|---|
| Modality Encoders & Embeddings | Audio tower, Vision tower, embed_tokens, lm_head |
100% BF16 | Preserves acoustic mel/conformers, high-res visual features, and full 262K vocabulary logits. |
| Early Comprehension | Layers 0..3 (Attention + MLPs) |
100% BF16 | Safeguards continuous audio and visual projections as multimodal tokens enter the LLM backbone. |
| Global Full Attention | Layers 5, 11, 17, 23, 29, 35, 41, 47 (q, k, v, o_proj) |
100% BF16 | Protects long-range associative recall and cross-turn reasoning (attention_k_eq_v aligned for vLLM QKVParallelLinear). |
| Final Output Polish | Layers 46..47 (Attention + MLPs) |
100% BF16 | Preserves final logit calibration and draft KV-sharing source layers (46 sliding, 47 full). |
| Middle Sliding Attention | 35 middle sliding layers (4..45, non-global) |
OCP MXFP8 | 32-element micro-blocks (group_0) halving attention memory with ~31.5 dB SNR. |
| Middle Semantic MLPs | Layers 4..45 (gate_proj, up_proj, down_proj) |
Blackwell NVFP4 W4A4 | 16-element micro-blocks (group_1) for peak Tensor Core throughput and 10.84 GiB footprint. |
| Normalizations & Scalars | All RMSNorms, Q/K Norms, layer_scalar |
100% BF16 | Prevents activation variance drift across modalities. |
Multimodal Audio/STT Calibration & Verification
Quantized using llmcompressor oneshot calibration over 192 balanced multimodal samples:
- 64 Speech & Audio samples: Accented English, Japanese, and code-switching conversational speech.
- 48 Photographic Images + 16 Multi-Frame Video sequences.
- 64 Agentic Code & Bilingual Instruction texts.
72-sample bilingual speech recognition eval (eval_gemma4_stt.py):
- Overall Exact Match: 11.11% (identical to Base BF16
11.11%). - Native English Dataset (
n=10): 60.0% EM / 0.0937 CER / 0.2009 WER (vs Base BF1640.0% EM / 2.1444 CER). - Cloned Voice (
n=3): 0.4267 CER (vs Base BF161.0553 CER).
Quickstart with vLLM
vllm serve catplusplus/gemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8 \
--trust-remote-code \
--gpu-memory-utilization 0.98 \
--max-model-len -1 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-num-batched-tokens 2496 \
--max-num-seqs 4 \
--cudagraph-capture-sizes 1 2 4 \
--performance-mode interactivity \
--spec-method mtp \
--spec-model catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8 \
--spec-tokens 5 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--default-chat-template-kwargs '{"enable_thinking": true}'
(For pure burst coding / JSON classification using DSpark K=7 instead, pass --spec-method dspark --spec-model catplusplus/dspark-gemma-4-12b-it-Grand-Hybrid-Unsloth-Dynamic-NVFP4-MXFP8 --spec-tokens 7.)
Python Multimodal (Audio + Vision + Text) Example
import librosa
from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams
model_id = "catplusplus/gemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8"
processor = AutoProcessor.from_pretrained(model_id)
llm = LLM(
model=model_id,
trust_remote_code=True,
gpu_memory_utilization=0.98,
kv_cache_dtype="fp8",
max_num_batched_tokens=2496,
speculative_config={
"method": "mtp",
"model": "catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8",
"num_speculative_tokens": 5,
},
)
# Audio transcription
audio, sr = librosa.load("speech.wav", sr=16000, mono=True)
audio_prompt = processor.apply_chat_template(
[{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Transcribe this audio accurately."}]}],
add_generation_prompt=True,
)
print(llm.generate([{"prompt": audio_prompt, "multi_modal_data": {"audio": (audio, sr)}}], SamplingParams(max_tokens=128))[0].outputs[0].text)
# Vision understanding
img = Image.open("image.png").convert("RGB")
img_prompt = processor.apply_chat_template(
[{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this image concisely."}]}],
add_generation_prompt=True,
)
print(llm.generate([{"prompt": img_prompt, "multi_modal_data": {"image": img}}], SamplingParams(max_tokens=128))[0].outputs[0].text)