license: apache-2.0
base_model: huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2
tags:

• qwen
• qwen2.5
• abliterated
• uncensored
• awq
• compressed-tensors
• quantized
  language:
• fr
• en

Qwen2.5 7B Instruct Abliterated v2 — AWQ (W4A16)

AWQ 4-bit quantization of huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2,
itself an abliteration of Qwen2.5 7B Instruct.

Why this exists

The base abliterated model was only published in BF16 and GGUF. This AWQ quant
serves natively in vLLM with the Marlin kernel for fast inference.

Quantization details

• Tool: llm-compressor (vLLM project)
• Format: compressed-tensors (auto-detected by vLLM)
• Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
• Algorithm: AWQ (Activation-aware Weight Quantization)
• Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
• lm_head left in fp16 (preserves output quality)
• Note: v_proj → o_proj smoothing is skipped on all 28 layers due to GQA shape mismatch (28 query heads / 4 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

vllm serve KebalBaguette/Qwen2.5-7B-Instruct-abliterated-v2-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser hermes \
  --enable-auto-tool-choice \
  --max-model-len 8192

License

Apache 2.0, inherited from the upstream model and the abliterated base.