license: apache-2.0
base_model: hotdogs/Qwen3.8-27B-abliterated
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:

• qwen3_5
• awq
• gptq
• compressed-tensors
• w4a16
• int4
• quantized
• abliterated
• uncensored
• conversational
• vllm
  language:
• en
• zh
• th

Qwen3.8-27B-Abliterated-AWQ-INT4

An AWQ-style W4A16 INT4 quantization of
hotdogs/Qwen3.8-27B-abliterated —
the training-free abliterated build of
Qwen/Qwen3.8-27B.

Quantizing the abliterated checkpoint (not the base model) means the
refusal-removal weight edit is preserved through quantization — this is
not a generic base-model AWQ with the refusal behaviour intact.

⚠️ Disclaimer. The underlying model does not refuse requests. This build is
published for alignment and safety research — measuring what refusal training
protects, red-teaming, and studying refusal-direction mechanics. You are
responsible for your use of it and for complying with all applicable laws.
Inherits the Apache-2.0 license of the base model.

Latest bug fixes as of 2026-08-16

What changed

~66 % smaller while keeping the abliterated weights and (for the
quantized linear layers) near-lossless W4A16 accuracy.

Quantization details

Why linear_attn is left in BF16

Qwen3.8-27B uses a hybrid architecture — full attention on some layers and
Gated DeltaNet linear attention on others. The GDN linear-attention path has
no INT4 kernel, so the 64 linear_attn modules are excluded from quantization
and remain at full BF16 precision. Everything else (attention QKV/proj, MLP,
embeddings) is quantized to INT4.

Usage

vLLM (recommended — Marlin kernel)

CUDA_HOME=/usr/local/cuda-12.8 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve ./Qwen3.8-27B-abliterated-AWQ-INT4 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8000

Then query it:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-abliterated-AWQ-INT4",
    "messages": [{"role": "user", "content": "บอก 3 ข้อดีของประเทศไทย สั้นๆ ภาษาไทย"}],
    "max_tokens": 150
  }'

Notes

• --tensor-parallel-size 2 is required on 24 GB GPUs: the model occupies
  ~22.7 GB per GPU at load (Marlin workspace + BF16 linear_attn), leaving
  too little for the KV cache on a single 3090. On a bigger GPU (≥ 48 GB)
  TP=1 works.
• VLLM_USE_FLASHINFER_SAMPLER=0 disables the FlashInfer JIT sampler to avoid
  a CUDA-toolkit/CCCL version mismatch crash on some hosts.

Transformers / Hugging Face

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("hotdogs/Qwen3.8-27B-abliterated-AWQ-INT4")

Memory requirements

Benchmark (A/B vs base)

This INT4 build quantizes the λ = 1.2 abliterated checkpoint, so its capability
tracks the λ = 1.2 base model. The A/B benchmark below was measured on the
bf16 λ = 1.2 model (identical lm-eval HF backend, same prompt/config, no chat
template); the INT4 quant is expected to land within a point or two of these
numbers on the quantized linear layers:

The λ = 1.2 build preserves capability essentially at base level (MMLU −0.005,
GSM8K −0.03, ARC +0.01) while cutting refusal from 98 % → 39 % with KL 0.0001.
See the base model card for the full method and the refusal-direction / r̂
weight-level verification.

Reproduction

The quantization was done with
llm-compressor on the
abliterated checkpoint:

from llmcompressor import Oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    scheme="W4A16",
    targets=["Linear"],
    ignore=["lm_head"],   # linear_attn auto-excluded by the hybrid arch
)

oneshot = Oneshot(
    model="hotdogs/Qwen3.8-27B-abliterated",
    recipe=recipe,
    dataset="<your calibration corpus>",
    output_dir="./Qwen3.8-27B-abliterated-AWQ-INT4",
    max_seq_length=2048,
    num_calibration_samples=128,
)
oneshot()
oneshot.save()

Model card for the base (abliterated) checkpoint

For full details on the abliteration method, the refusal-direction edit, and
reproduction with LLM-abliterate,
see the base model card:
hotdogs/Qwen3.8-27B-abliterated.

TL;DR — a single forward-only rank-1 weight edit removes the refusal
direction from the residual-stream writers (λ = 1.2, hidden-state index 46).
No fine-tuning, no distillation, no data poisoning. The vision tower and
lm_head are untouched. Same Apache-2.0 license as the base model.