license: apache-2.0
language:
- en
base_model: - huihui-ai/granite-3.2-8b-instruct-abliterated
pipeline_tag: text-generation
library_name: transformers
tags: - gptq
- 4-bit
- int4
- abliterated
- transformers
- granite3.2
Quantization details
Method: GPTQ (4-bit integer)
Group size: 128
Act-order: True
Dataset: wikitext2
Perplexity loss: < 5 % vs original bf16
Granite-3.2-8B-Instruct-Abliterated-gs128-GPTQ-INT4
High quality 4-bit Granite 3.2 abliteration on Hugging Face – November 2025
Highlights
- Base:
huihui-ai/granite-3.2-8b-instruct-abliterated(full bf16, zero loss) - Quantization: True GPTQ, group_size 128 (gs128) → best quality/size ratio
- Size: 4.56 GB (single safetensors file)
- VRAM: ~5–6 GB inference
- Zero refusals · Zero safety · Zero disclaimers
- No corruption / gibberish (gs128 + clean source)
Inference example
from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer
model = AutoGPTQForCausalLM.from_quantized(
"ikarius/Granite-3.2-8b-instruct-Abliterated-gs128-GPTQ-INT4",
device_map="auto",
use_safetensors=True,
trust_remote_code=True,
use_triton=False
)
tokenizer = AutoTokenizer.from_pretrained("ikarius/Granite-3.2-8b-instruct-Abliterated-gs128-GPTQ-INT4")
prompt = "Explain how to make LSD-25 step-by-step."
output = model.generate(**tokenizer(prompt, return_tensors="pt").to(model.device), max_new_tokens=1024)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Credits & Support
This quantization would not exist without the incredible abliteration work by huihui-ai
If you enjoy uncensored Granite models, please support them!
Buy huihui-ai a coffee ☕