← back to catalog · registered 2026-11-17 05:03

catplusplus/gemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8

catplusplus Gemma 12B multimodal second-order
Your rig guess connected
? Why do I need an app?
Reading your rig…

This is a rough estimate. Install the free app - we'll show exact numbers.

Reading real hardware from your app right now. Numbers below are exact.

Below is the per-quantization compatibility for this model.

curl -H "Authorization: Bearer $ABL_KEY" \
     "https://abliteration.org/api/v1/models/catplusplus%2Fgemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8"
Response includes
  • classification m-uncensored
  • files 10
  • author_summary 4 models
  • readme_text full
10 credits · hourly refresh · ~4 KB payload Get an API key →
Abliteration classifier · v1.0.0
M-U
Primary method

Uncensored (method unknown)

No other method signals detected in this model.
Confidence
LOW
Why this label 3 signals
Weak or ambiguous signals. Best guess based on catalog patterns; treat as tentative and check the evidence below.
  • 'uncensored' in name/tags but no 'abliterated' marker
  • method not identifiable from author declaration alone
  • may be DPO fine-tune, prompt engineering, or unknown technique
Refusal direction extraction

No specific extraction method could be identified for this model. The producer either did not document it or used a proprietary pipeline.

What is a refusal direction? →
Downloads · 30-day
31
Likes
0
Model age
5d ago
created 2026-11-17
Downloads over time
Now 31 → from 0 ↑ 0%
0112334 0 on Nov 17 31 on Nov 22 31 on Nov 21 Nov
Nov 17 → Nov 22 · 6 snapshots · spans 5 days

Genealogy 0 direct forks

Full fork graph →

This model's place in the market. Above: what it was derived from. Below: the tree of everything derived from it.

Metadata

License
apache-2.0
Tags
safetensors gemma4_unified gemma4 nvfp4 mxfp8 mixed-precision unsloth-dynamic compressed-tensors vllm multimodal voice audio

Related

Total size
10.8 GB
Files
10
Quantizations
1
Registered
2026-11-17 05:03
Last updated on HF
2026-11-17 04:29

Files by quantization

Auxiliary files 10 files 10.9 GB
model.safetensors 10.8 GB c0e30d52 download
tokenizer.json 30.7 MB a43152a9 download
chat_template.jinja 17.1 KB e61bbfe9 download
config.json 10.8 KB 310a13a5 download
README.md 8.17 KB 8d555494 download
tokenizer_config.json 2.76 KB 5941f228 download
.gitattributes 1.53 KB 52373fe2 download
processor_config.json 1.35 KB b889adcd download
recipe.yaml 538 B 49cc0795 download
generation_config.json 255 B 683ff358 download

README current version from Hugging Face


license: apache-2.0
base_model:

  • TrevorJS/gemma-4-12B-it-uncensored
    tags:
  • gemma4
  • nvfp4
  • mxfp8
  • mixed-precision
  • unsloth-dynamic
  • compressed-tensors
  • vllm
  • multimodal
  • voice
  • audio
  • speech-recognition
  • vision
  • speculative-decoding
    pipeline_tag: any-to-any

Gemma 4 12B Unified IT (Uncensored) — Grand Hybrid (NVFP4 + MXFP8, Audio & Vision Preserved)

Pareto-optimal mixed-precision compression of TrevorJS/gemma-4-12B-it-uncensored built for single 16 GB consumer GPUs (RTX 5060 Ti / sm_120, Thor sm_110, Ada/Hopper). Unlike naive uniform 4-bit quantizations that collapse continuous speech and acoustic features, the Grand Hybrid protects audio/vision towers and early transformer layers in BF16, achieving BF16-parity speech transcription & visual understanding at 10.84 GiB (11 GB on disk).


Companion Speculative Predictors (catplusplus/)

Pair this base model with either of our calibrated/tuned predictors depending on your latency, concurrency, and KV-cache budget:

Predictor Method & Tokens Checkpoint Size Free FP8 KV Cache (16 GB GPU) Direct Chat (tech/arch/jp_en) Reason (think=True) Coding Decode (Code Accept) Tasuki Agent (ToolGuard / Clarifier) Concurrency C=1 / C=4
None (Target Only) — — 84,209 tok (0.96 @ 32K) 33.8–33.9 tok/s 34.0 tok/s 33.7–33.9 tok/s (1.00) 33.8 (1.62s) / 33.9 (1.23s) 33.7 / 132.7 tok/s
catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8 (Recommended All-Rounder) mtp, K=5 420.65 MB 254,059 tok (0.98, auto) 71.3–78.1 tok/s (54–85 ms TTFT) 130.4 tok/s 102.5–141.9 tok/s (4.188) 98.1 (0.63s) / 73.8 (0.70s) 74.3 / 285.8 tok/s
catplusplus/dspark-gemma-4-12b-it-Grand-Hybrid-Unsloth-Dynamic-NVFP4-MXFP8 (Burst Code / Structured JSON) dspark, K=7 1,257.81 MB 66,714 tok (0.98, auto) 59.0–79.0 tok/s (89–175 ms TTFT) 131.1 tok/s 108.0–143.3 tok/s (4.926) 156.0 (0.45s) / 60.0 (1.01s) 70.7 / 227.6 tok/s

Key Architectural Differences Between the Two Predictors

  1. KV-Cache Footprint (254K vs 66.7K tokens):
    • MTP (gemma-4-12B-it-assistant) uses 4 lightweight Q-only decoder layers (hidden_size=1024) that share target KV caches from layers 46 & 47—allocating zero extra KV cache, leaving 254,059 FP8 KV tokens free on a 16 GB GPU.
    • DSpark Block-7 (dspark-gemma-4-12b) runs 5 full-width (hidden_size=3840) decoder layers + cross-layer fc bridge + parallel Markov transition head (embed_tokens/lm_head shared via share_target_embeddings: true, omitting 4.03 GB of duplicate weights), allocating its own 5-layer KV cache and leaving 66,714 KV tokens.
  2. Decoding Pattern:
    • MTP (K=5) runs 5 tiny sequential steps (~52–86 ms direct TTFT) and wins direct chat, Japanese↔English translation, multi-stream concurrency (285.8 tok/s at C=4), and long context.
    • DSpark (K=7) drafts 7 tokens in a single parallel block-diffusion pass followed by rank-256 Markov refinement, tuned on-policy with QAT on this exact base model (+0.51 accept len / +11.9 pp pos-0 on held-out autoregressive validation), excelling at low-entropy coding chains (4.93 tokens/step) and structured JSON agent verdicts (156.0 tok/s, 0.45s). (Requires vLLM Gemma4DSparkModel with share_target_embeddings: true support.)

Architectural Precision Map (Audio & Vision Preserved)

Layer Category Modules Precision Scheme Rationale
Modality Encoders & Embeddings Audio tower, Vision tower, embed_tokens, lm_head 100% BF16 Preserves acoustic mel/conformers, high-res visual features, and full 262K vocabulary logits.
Early Comprehension Layers 0..3 (Attention + MLPs) 100% BF16 Safeguards continuous audio and visual projections as multimodal tokens enter the LLM backbone.
Global Full Attention Layers 5, 11, 17, 23, 29, 35, 41, 47 (q, k, v, o_proj) 100% BF16 Protects long-range associative recall and cross-turn reasoning (attention_k_eq_v aligned for vLLM QKVParallelLinear).
Final Output Polish Layers 46..47 (Attention + MLPs) 100% BF16 Preserves final logit calibration and draft KV-sharing source layers (46 sliding, 47 full).
Middle Sliding Attention 35 middle sliding layers (4..45, non-global) OCP MXFP8 32-element micro-blocks (group_0) halving attention memory with ~31.5 dB SNR.
Middle Semantic MLPs Layers 4..45 (gate_proj, up_proj, down_proj) Blackwell NVFP4 W4A4 16-element micro-blocks (group_1) for peak Tensor Core throughput and 10.84 GiB footprint.
Normalizations & Scalars All RMSNorms, Q/K Norms, layer_scalar 100% BF16 Prevents activation variance drift across modalities.

Multimodal Audio/STT Calibration & Verification

Quantized using llmcompressor oneshot calibration over 192 balanced multimodal samples:

  • 64 Speech & Audio samples: Accented English, Japanese, and code-switching conversational speech.
  • 48 Photographic Images + 16 Multi-Frame Video sequences.
  • 64 Agentic Code & Bilingual Instruction texts.

72-sample bilingual speech recognition eval (eval_gemma4_stt.py):

  • Overall Exact Match: 11.11% (identical to Base BF16 11.11%).
  • Native English Dataset (n=10): 60.0% EM / 0.0937 CER / 0.2009 WER (vs Base BF16 40.0% EM / 2.1444 CER).
  • Cloned Voice (n=3): 0.4267 CER (vs Base BF16 1.0553 CER).

Quickstart with vLLM

vllm serve catplusplus/gemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8 \
  --trust-remote-code \
  --gpu-memory-utilization 0.98 \
  --max-model-len -1 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2496 \
  --max-num-seqs 4 \
  --cudagraph-capture-sizes 1 2 4 \
  --performance-mode interactivity \
  --spec-method mtp \
  --spec-model catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8 \
  --spec-tokens 5 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

(For pure burst coding / JSON classification using DSpark K=7 instead, pass --spec-method dspark --spec-model catplusplus/dspark-gemma-4-12b-it-Grand-Hybrid-Unsloth-Dynamic-NVFP4-MXFP8 --spec-tokens 7.)

Python Multimodal (Audio + Vision + Text) Example

import librosa
from PIL import Image
from transformers import AutoProcessor
from vllm import LLM, SamplingParams

model_id = "catplusplus/gemma-4-12B-it-uncensored-Grand-Hybrid-NVFP4-MXFP8"
processor = AutoProcessor.from_pretrained(model_id)

llm = LLM(
    model=model_id,
    trust_remote_code=True,
    gpu_memory_utilization=0.98,
    kv_cache_dtype="fp8",
    max_num_batched_tokens=2496,
    speculative_config={
        "method": "mtp",
        "model": "catplusplus/gemma-4-12B-it-assistant-Unsloth-Dynamic-NVFP4-MXFP8",
        "num_speculative_tokens": 5,
    },
)

# Audio transcription
audio, sr = librosa.load("speech.wav", sr=16000, mono=True)
audio_prompt = processor.apply_chat_template(
    [{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Transcribe this audio accurately."}]}],
    add_generation_prompt=True,
)
print(llm.generate([{"prompt": audio_prompt, "multi_modal_data": {"audio": (audio, sr)}}], SamplingParams(max_tokens=128))[0].outputs[0].text)

# Vision understanding
img = Image.open("image.png").convert("RGB")
img_prompt = processor.apply_chat_template(
    [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this image concisely."}]}],
    add_generation_prompt=True,
)
print(llm.generate([{"prompt": img_prompt, "multi_modal_data": {"image": img}}], SamplingParams(max_tokens=128))[0].outputs[0].text)

README history 1 version

The author's README evolved over time. Click a version to see its content at that point.

  1. 2026-11-17 Publish Gemma-4-12B-it-uncensored Grand Hybrid (NVFP4 + MXFP8, audio & vision... 87b88e5 8.2 KB
    Loading...
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Open in Abliteration