license: apache-2.0
base_model: llmfan46/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic
tags:

• openvino
• int4
• intel-arc
• gemma4
• mixture-of-experts
• roleplay
• uncensored
  language:
• en

Gemma-4 26B-A4B Heretic, OpenVINO INT4. Coherent to 32K on ONE Intel Arc B70 (32 GB).

This is the abliterated ("heretic") Gemma-4 26B-A4B mixture-of-experts, exported to OpenVINO INT4
and graph-patched so it stays coherent at long context on Intel GPUs. Stock exports of this model
break down into word salad past about 16K tokens on Arc. This build does not.

It is a general instruction model with the refusal behavior removed, so you can use it for
anything you would use the base model for. I personally run it for uncensored roleplay
(Skyrim companion AI and Discord character bots), and the defaults below are tuned from
hundreds of hours of that use.

Everything below was measured on one Intel Arc Pro B70 with 32 GB VRAM. The current fast path
uses the Wondernuttz OpenVINO 2026.4 fork on Linux. The older stock OpenVINO 2026.2
VLMPipeline path remains documented because it is the path separately verified at 32K.
No llama.cpp and no CUDA were used for either set of results.

Toolkit: patches, OpenAI-compatible server, quickstart, and the full bug catalog live at github.com/Wondernuttz/OpenVino-For-Gemma-4.

Measured performance (single Arc Pro B70)

Fork branch: arc-xe2-gemma4-pa-2026.4, commit 2c82358676.

Benchmark history

The last patch is a full-model 31.0% PP gain over the previous accepted fork result and a 5.56x
gain over the coherent July 23 stock continuous-batching result. Prompt contents differed
between runs, so these are not prefix-cache hits. The accepted coherence output remained
bit-identical with SHA-256
b123146233af2aac9e725826d9011513ee4c3dc9ec4634fd892e6937f06afb58.

The full history, exact settings, profiles, build commands and rejected runs are in
BENCHMARK_HISTORY_GEMMA4_26B.md.

Current context curve

Release plugin, DQ128, second unique request at each shape:

At 6,622 tokens, two sustained DQ64 runs measured 5,821.9 and 5,832.5 tok/s. DQ128 tied within
measurement noise and stays in the deployment configuration.

Older stock single-stream context curve

These OpenVINO 2026.2 measurements use the compatibility path. They are kept because this is the
path that was actually exercised at 32K.

Fastest short-prompt prefill in my fleet (2.5x SYCL at matched pp512). Past roughly 5K my
Qwen3.6-35B MoE
overtakes it, and at 32K the Qwen ingests nearly 3x faster; full attention pays a quadratic tax
at depth that the Qwen's hybrid linear attention mostly avoids.

Long-context capability, verified by needle retrieval

A password fact was planted early in the prompt and the model was asked to retrieve it at the
end. Pass means it answered with the exact password.

Thinking mode reasoning correctly over 32,000 tokens of context is the standout result. Without
the rope patch in this build, thinking degrades within a few thousand tokens on Intel GPUs.
Past 32K the limit on my box is host RAM during prefill, not the model or the card; the
architecture is rated to 262K positions and this build ships position tables to 131K.

Those 32K results belong to the older stock single-stream path. The current 2026.4 fork path is
validated through 16K. Do not split a 16K continuous-batching prefill with an 8,192-token
scheduler limit; that test triggered an Xe GPU fault. A 30K continuous-batching test also faulted
after the larger scheduler shape exceeded the GPU maximum allocation size. The fast path is not
advertised beyond 16K yet.

How this compares to published numbers

I found no other public single-B70 result matching the current OpenVINO speed as of 2026-07-26.
The public PMZFX repository reports the same Gemma-4 26B-A4B class at 1,129 pp512 and 52.6 tg128
with llama.cpp SYCL.

That is 2.13x the public B70 decode result. The current nearest short-prefill point is 5,158.9
tok/s at 966 tokens, so it must not be called a matched pp512 comparison. The older stock path
did measure about 2,900 tok/s at pp512.

For an NVIDIA reference, a public single-RTX-3090 replication of this model reports 129 to 131
tok/s with llama.cpp and n-gram speculative decoding. The current B70 result is 112.2 tok/s
without speculative decoding, so RTX 3090-class decode is a fair description. A public RTX 5090
vLLM result reports about 228 tok/s without DFlash and 578 tok/s with DFlash. This B70 is not at
RTX 5090 decode speed.

NVIDIA sources: single RTX 3090 Gemma-4 replication and the single RTX 5090 DFlash run.

What was fixed (why this is not just another export)

1. The MoE router is kept unquantized (AWQ INT4, group size 64, ignored_scope on the router,
   matching Intel's own recipe). Quantizing the router breaks GPU MoE fusion and the model will
   not load at all.
2. RoPE lookup-table patch. Intel GPUs execute the rope angle math in fp16, which cannot
   represent the rotation angles past roughly 16K positions, so long context collapses into
   gibberish. This build replaces the runtime sin/cos computation with precomputed tables and a
   Gather, which fp16 execution cannot corrupt. Note for anyone patching other Gemma-4 exports:
   the global attention uses proportional (partial) RoPE, so three quarters of the frequency
   values are zero BY DESIGN. Do not "repair" them; preserve the zeros.
3. The matching July 23 OpenVINO and GenAI commits fix Gemma-4 PagedAttention conversion,
   token_type_ids layout and the sliding-window graph path. Earlier continuous-batching builds
   either aborted or returned garbage at long prompt lengths.
4. max_num_batched_tokens=8192 keeps the 6,622-token benchmark in one scheduler pass. This was
   the jump from roughly 1,014 to 4,143 PP tok/s.
5. Grouped MoE prefill, DQ128 and the N128 tile selection raised the accepted result to 4,448 PP
   tok/s.
6. Commit 2c82358676 raises Xe2 micro-SDPA eligibility from 256 to 512 head size. Gemma's five
   global-attention layers then use the existing XMX route, raising the sustained result to
   5,827 PP tok/s.

How to run

The stock single-stream path is the easiest compatibility route:

pip install openvino-genai==2026.2.0 huggingface_hub
hf download Wondernutts/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic-int4-ov --local-dir ./gemma4-26b-heretic-ov

import openvino_genai as g

# "GPU" is your Arc card. On multi-GPU systems check
# openvino.Core().get_property("GPU.N", "DEVICE_PCI_INFO") first.
pipe = g.VLMPipeline("./gemma4-26b-heretic-ov", "GPU")

def chat(system, user, think=False):
    p = "<bos><|turn>system\n" + system + ("\n<|think|>" if think else "") + "<turn|>\n"
    p += "<|turn>user\n" + user + "<turn|>\n<|turn>model\n"
    if not think:
        p += "<|channel>thought\n<channel|>"   # pre-closed thought channel = fast direct replies
    c = g.GenerationConfig()
    c.max_new_tokens = 512 if not think else 1536
    c.do_sample = True; c.temperature = 0.9; c.top_p = 0.95
    try: c.repetition_penalty = 1.2            # recommended, see notes
    except Exception: pass
    try: c.apply_chat_template = False
    except Exception: pass
    return str(pipe.generate(p, generation_config=c))

print(chat("You are a witty tavern keeper in Whiterun.", "Rough night?"))

Headline-performance path

The 5,827 PP and 112 decode results require the Linux fork branch above, the matching OpenVINO
2026.4 ABI and the matching GenAI build. A 2026.4 GPU plugin cannot be dropped into a 2026.2
runtime. Build instructions and pinned commits are in the benchmark history.

The runtime configuration is:

import os
import openvino_genai as g

os.environ["MOE_USE_GROUPED_GEMM_PREFILL"] = "1"
os.environ["MOE_MICRO_GEMM_N_HINT"] = "128"

scheduler = g.SchedulerConfig()
scheduler.cache_size = 8
scheduler.enable_prefix_caching = True
scheduler.max_num_batched_tokens = 8192

pipe = g.VLMPipeline(
    "./gemma4-26b-heretic-ov",
    "GPU",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
    scheduler_config=scheduler,
)

Use max_num_batched_tokens=16384 for the separately validated 16K shape. Do not use 8,192 to
split that prompt. The optimized branch is Linux-tested. Windows is untested for this fork path.

Notes that will save you pain:

• The prompt format is <|turn>role ... <turn|> (this model's native template, check
  chat_template.jinja), not classic Gemma <start_of_turn>. The classic format "works" but
  leaks a stray thought prefix and runs about 20% slower.
• Thinking is binary. Pre-close the thought channel as above for fast direct replies. Put
  <|think|> in the system turn and do not pre-close for reasoning-first answers. When thinking,
  give it at least 1024 max_new_tokens or the answer gets truncated.
• Do not use grammar-constrained or JSON mode (response_format). Gemma-4 has a documented
  repetition-collapse bug (google-deepmind/gemma#622) that fires most reliably under constrained
  JSON decoding. Asking for JSON in the prompt is fine.
• Repetition penalty around 1.2 holds INT4 coherence at temperature 1.0 in long roleplay
  sessions. Values near 1.05 were observed to degenerate.
• On the stock 2026.2 continuous-batching path, use DYNAMIC_QUANTIZATION_GROUP_SIZE=0.
  The corrected 2026.4 fork path is coherent at DQ128 and uses it for deployment.

Provenance

google/gemma-4-26B-A4B-it (QAT q4_0 unquantized), heretic abliteration by
llmfan46
(directional ablation, norm-preserving), then OpenVINO INT4 AWQ export with the router excluded,
then the RoPE lookup-table graph patch (this repo).

The 31B dense sibling is also published: richer prose, slower replies, honest single-card limits on its own card.

Intended use and content notice

This is an uncensored general model. Built and tested for roleplay and creative writing on local
Intel hardware; The abliteration removes refusal behavior and outputs are unfiltered; you are responsible for lawful and appropriate use. Licensed under
Apache 2.0, same as the upstream Gemma 4 release.