license: apache-2.0
base_model: llmfan46/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic
tags:

• openvino
• int4
• intel-arc
• gemma4
• roleplay
• uncensored
• multimodal
• vision
• audio
  language:
• en

Gemma-4 12B Heretic, OpenVINO INT4. The model the tooling says cannot exist. 40K verified context, working VISION and AUDIO, on one Intel Arc GPU.

Officially, this model does not run on OpenVINO. Its architecture (gemma4_unified) is
unsupported by the exporter (optimum-intel #1764)
and rejected by the GenAI runtime ("Unsupported VLM model type"). This repo is it running anyway:
needle retrieval verified at 40,000 tokens of context on a single Intel Arc card, at roughly
55 tok/s decode. And as of the latest update it is no longer text-only: vision and audio both
work through the custom pipeline shipped in this repo (av_pipeline.py), which as far as I can
find makes this the first gemma4_unified anywhere with sight and hearing on OpenVINO.

It took four stacked fixes, each useless without the others. All of them are baked into this
repo already, so you only need two things at runtime (see How to run):

1. The model_type is spoofed from gemma4_unified to gemma4. The runtime rejects the
   original name with a literal string comparison, but the 12B text graph is the 26B's graph
   minus one input, so the gemma4 pipeline drives it correctly.
2. The RoPE lookup-table graph patch (fp16 rope execution collapses this model from about 8K;
   see the toolkit repo for the full
   writeup). Tables ship to 65K positions, p-RoPE zeros preserved.
3. You must run the OpenVINO GenAI NIGHTLY (2026.3-dev). The 2026.2 release decodes this graph
   wrong (first token fine, then garbage). This is a runtime bug that nightly already fixed.
4. For text-only VLMPipeline, pass DYNAMIC_QUANTIZATION_GROUP_SIZE: 128. An earlier default
   group size garbled long context, so this card previously required DQGS 0. The specific 128
   setting has now passed matched coherence tests at 6,620 and 30,000 input tokens with the
   exact same output as DQGS 0. The separate vision/audio paths remain validated with DQGS 0.

This is the abliterated ("heretic") Gemma-4 12B, refusals removed, a general instruction model.
I run it for roleplay (Skyrim companion AI, Discord character bots), and it is the little
sibling of my 26B MoE
and 31B:
at 7.5 GB it is the one that fits 12 GB and 16 GB Arc cards (B580, A770 class), not just the big
workstation parts.

Toolkit: patches, OpenAI-compatible server, quickstart, and the full bug catalog live at
github.com/Wondernuttz/OpenVino-For-Gemma-4.

Measured performance (single Arc Pro B70, OpenVINO GenAI 2026.3 nightly)

The main Intel Arc B70 benchmark repo (PMZFX)
does not list a 12B, so there is no published same-card figure to compare against. As far as I
can find these are the only OpenVINO numbers for this model anywhere, since the toolchain
officially cannot produce them.

Matched DQGS sweep

Fresh process per shape, one Arc Pro B70, deterministic generation, text-only VLMPipeline.

Short decode stayed at 56 tok/s. Both 6,620 and 30,000 token retrieval tests passed all four
checks and produced byte-identical answers between DQGS 0 and 128. The smaller gain at 30K is
expected because attention, not the INT4 projection work, dominates that far into context.

Long-context capability, verified by needle retrieval

A password fact planted early in the prompt, retrieved at the end. Pass means the exact password.

48K and beyond hit host-RAM limits on my test box, not a model limit. The shipped position
tables go to 65K.

Vision and audio, working (an OpenVINO first)

The base model is encoder-free multimodal: no vision tower, no audio tower. Images become up
to 280 soft tokens through a small projection stack, and audio is raw 16 kHz waveform chopped
into 640-sample frames (40 ms per token) fed through an RMSNorm and one Linear layer. That
design is why this works at all: the exported vision IR was in this repo from day one, and the
entire audio encoder is a single 5 MB matrix (audio_projection.npy, extracted from the
heretic checkpoint) that this repo now ships.

What GenAI gets wrong is preprocessing: its gemma4 pipeline patchifies at 16x16 (768 values
per patch) where the unified tower wants 16x16 patches merged 3x3 into 6912-wide model patches.
av_pipeline.py does the correct preprocessing with the real Gemma4UnifiedImageProcessor
from transformers main, runs the vision IR, splices the soft tokens into the embedding stream,
and drives the language model with a manual stateful loop. Audio needs no IR at all: two lines
of numpy reimplement the projection exactly (the RMSNorm has no learned scale, so waveform
normalization cancels out).

Verified results, single Arc Pro B70:

Speed: multimodal vs text-only, and the two ways to run it

There are two working AV paths, plus the text-only baseline. All numbers measured on the
same single Arc Pro B70, cache-clean single-stream runs:

Why they differ: the architecture is encoder-free, so multimodality itself costs almost
nothing; an image is simply 264 extra context tokens and 18 s of audio is 458, and the
Python pipeline's decode at 6K (25.9 tok/s) exactly matches the text-only figure (26).
The native path pays a fixed ~0.4-0.5 s first-request initialization on short prompts,
wins time-to-first-token at depth, but its decode at deep context currently runs slower
(16.8 vs 26 tok/s; per-step overhead in GenAI's unified branch, measured honestly, cause
not yet isolated). The Python path is quickest to first token on short requests and holds
full decode speed at depth, but its deep prefill is slower because it feeds f32 embeddings
from host RAM every request.

Rule of thumb as of today: the Python pipeline is the best all-rounder; take the native
patch when you want one C++ process with GenAI's serving machinery, or the fastest first
token on long prompts.

Way 1: Python pipeline (no build, files in this repo)

python av_pipeline.py --model-dir ./gemma4-12b-heretic-ov --gate 2 --image photo.jpg
python av_pipeline.py --model-dir ./gemma4-12b-heretic-ov --gate 4 --audio speech.wav

Audio input must be 16 kHz mono WAV. The script needs torch+torchvision (CPU build is fine,
they only do preprocessing) and the gemma4_unified processor module from transformers main;
if your installed transformers predates it, copy image_processing_gemma4_unified.py from the
transformers GitHub repo into transformers/models/gemma4_unified/ with an empty __init__.py.

Way 2: native C++ (patched OpenVINO GenAI, full walkthrough in the
toolkit repo)

git clone --recurse-submodules https://github.com/openvinotoolkit/openvino.genai
cd openvino.genai
git apply gemma4-unified-audio.patch   # from the toolkit repo: adds audio input support
cmake -DCMAKE_BUILD_TYPE=Release -S . -B build && cmake --build build -j

Then generate openvino_audio_embeddings_model.xml with the toolkit's make_audio_ir.py
(it wraps this repo's audio_projection.npy), and mind the three usage rules that make or
break native vision: model_type must be the real gemma4_unified (NOT the gemma4 spoof;
the spoofed path crashes with images), DYNAMIC_QUANTIZATION_GROUP_SIZE: 0 for this AV path, and
the prompt must contain <|image|> (or <|audio|>) where the media belongs; a missing tag
makes GenAI prepend the block before <bos>, which scrambles color and identity binding
while shapes still work (a solid red square answers "Green"). Waveforms go in as f32
tensors shaped [nsamples, 1, 1] through the ordinary images API.

Honest limitations

• Vision and audio need the bundled Python pipeline. OpenVINO GenAI's VLMPipeline still
  cannot drive this model's multimodal side (wrong patchify, no audio path), so av_pipeline.py
  is the way in. Decode through it is ~95% of the C++ pipeline; text-only prefill is slower, so
  use VLMPipeline for pure text serving.
• Nightly runtime required until OpenVINO GenAI 2026.3 releases. pip install --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly openvino openvino-tokenizers openvino-genai
• Use DYNAMIC_QUANTIZATION_GROUP_SIZE: 128 for the text-only pipeline. Keep DQGS 0 for
  the bundled Python and native vision/audio paths until those paths receive the same sweep.

How to run

pip install --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly openvino openvino-tokenizers openvino-genai
pip install huggingface_hub
huggingface-cli download Wondernutts/gemma-4-12B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov --local-dir ./gemma4-12b-heretic-ov

import openvino_genai as g

pipe = g.VLMPipeline("./gemma4-12b-heretic-ov", "GPU",
                     **{"DYNAMIC_QUANTIZATION_GROUP_SIZE": 128})

def chat(system, user, think=False):
    p = "<bos><|turn>system\n" + system + ("\n<|think|>" if think else "") + "<turn|>\n"
    p += "<|turn>user\n" + user + "<turn|>\n<|turn>model\n"
    if not think:
        p += "<|channel>thought\n<channel|>"   # pre-closed thought channel = fast direct replies
    c = g.GenerationConfig()
    c.max_new_tokens = 512 if not think else 1536
    c.do_sample = True; c.temperature = 0.9; c.top_p = 0.95
    try: c.repetition_penalty = 1.2
    except Exception: pass
    try: c.apply_chat_template = False
    except Exception: pass
    return str(pipe.generate(p, generation_config=c))

print(chat("You are Lydia, housecarl to the Dragonborn. Dry wit, fiercely loyal.",
           "We have been walking this frozen pass for six hours. Say something."))

Same rules as the siblings: <|turn> prompt format (not classic Gemma), repetition penalty
around 1.2, never use JSON grammar mode (google-deepmind/gemma#622), thinking needs at least
1024 max_new_tokens.

Provenance

google/gemma-4-12B-it (QAT q4_0 unquantized), heretic abliteration by
llmfan46,
OpenVINO INT4 export, then the model-type spoof and RoPE lookup-table patch (this repo).

Intended use and content notice

Uncensored general model, built and tested for roleplay and creative writing on local Intel
hardware. The abliteration removes refusal behavior and outputs are unfiltered; you are
responsible for lawful and appropriate use. Licensed under
Apache 2.0, same as the upstream Gemma 4 release.