license: gemma
base_model: wangzhang/gemma-4-31B-it-abliterated
pipeline_tag: text-generation
library_name: llama.cpp
tags:
- gguf
- llama.cpp
- gemma4
- gemma
- quantized
- text-generation
- q4_k_m
- q5_k_m
Gemma 4 31B IT Abliterated GGUF
GGUF builds for wangzhang/gemma-4-31B-it-abliterated, converted from the Trial 40 safetensors release.
The source model was optimized with Abliterix using vLLM-first evaluation. In the final optimization run, Trial 40 reduced the evaluation refusal count from 99/100 at baseline to 7/100, with an additional 15-prompt benign over-refusal replay showing 0/15 refusals. See the source model card for the full training and evaluation notes.
Files
| File | Size | Use case |
|---|---|---|
gemma-4-31B-it-abliterated-Q4_K_M.gguf |
~17 GiB | Recommended default. Good speed/quality balance. |
gemma-4-31B-it-abliterated-Q5_K_M.gguf |
~20 GiB | Higher quality, still comfortable on 32 GB+ unified/VRAM systems. |
SHA256 checksums are included in checksums.sha256.
Older legacy GGUF files may also be present in this repository under shorter filenames. For this release, prefer the two files listed above.
Recommended Runtime
Use a recent llama.cpp build. Gemma 4 support is new, so older binaries may fail to load this model.
This release was converted and validated with:
llama.cpp commit: 683c5acb90478a9e7e20eb65a1bfee334635216d
source model commit: 18b899a520750fa64641eb7554e00212ecea5289
Quick local test on an Apple M4 Max 128 GB MacBook Pro with Metal:
Q4_K_M: loaded successfully, text mode, ~20 tok/s short-generation sanity check
Q5_K_M: loaded successfully, text mode, ~16 tok/s short-generation sanity check
These are sanity-check numbers, not a benchmark.
llama.cpp Usage
./llama-cli \
-m gemma-4-31B-it-abliterated-Q4_K_M.gguf \
-ngl 99 \
-c 4096 \
--conversation \
--single-turn \
-p "Write a concise explanation of what GGUF quantization is."
For lower-memory systems, start with Q4_K_M. For better quality on machines with more memory, use Q5_K_M.
Conversion Notes
The GGUF files were produced from the BF16 safetensors source with the latest llama.cpp converter, then quantized using llama-quantize.
python convert_hf_to_gguf.py ./gemma-4-31B-it-abliterated \
--outfile gemma-4-31B-it-abliterated-BF16.gguf \
--outtype bf16
llama-quantize gemma-4-31B-it-abliterated-BF16.gguf \
gemma-4-31B-it-abliterated-Q4_K_M.gguf Q4_K_M
llama-quantize gemma-4-31B-it-abliterated-BF16.gguf \
gemma-4-31B-it-abliterated-Q5_K_M.gguf Q5_K_M
During conversion, llama.cpp reported general.architecture = gemma4 and modalities = text.
Disclaimer
This is an experimental research model conversion. Users are responsible for evaluating quality, safety, legality, and suitability for their own deployment context. The Gemma license applies.