license: gemma
base_model: mlabonne/gemma-3-27b-it-abliterated
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- quantized
- exllamav2
- exl2
Gemma 3 27B IT Abliterated - EXL2 Quantized
Exllamav2 quantized versions of mlabonne/gemma-3-27b-it-abliterated.
Hardware Requirements
4.0 bpw version fits on a 24GB GPU with 8192 context window
Vision
Vision works with ExllamaV2 0.2.9
Confirmed with exllamav2s examples/multimodal.py
Direct Download
huggingface-cli download Kooten/gemma-3-27b-it-abliterated-exl2 --revision 4.0bpw --local-dir gemma-3-27b-it-abliterated-4.0bpw --local-dir-use-symlinks False
huggingface-cli download Kooten/gemma-3-27b-it-abliterated-exl2 --revision 5.0bpw --local-dir gemma-3-27b-it-abliterated-5.0bpw --local-dir-use-symlinks False
💎 Gemma 3 27B IT Abliterated
This is an uncensored version of google/gemma-3-27b-it created with a new abliteration technique.
See this article to know more about abliteration.
I was playing with model weights and noticed that Gemma 3 was much more resilient to abliteration than other models like Qwen 2.5.
I experimented with a few recipes to remove refusals while preserving most of the model capabilities.
Note that this is fairly experimental, so it might not turn out as well as expected.
I recommend using these generation parameters: temperature=1.0, top_k=64, top_p=0.95.
⚡️ Quantization
✂️ Layerwise abliteration
In the original technique, a refusal direction is computed by comparing the residual streams between target (harmful) and baseline (harmless) samples.
Here, the model was abliterated by computing a refusal direction based on hidden states (inspired by Sumandora's repo) for each layer, independently.
This is combined with a refusal weight of 1.5 to upscale the importance of this refusal direction in each layer.
This created a very high acceptance rate (>90%) and still produced coherent outputs.