base_model: rajaykumar12959/qwen3.8-27b-abliterated
license: apache-2.0
pipeline_tag: text-generation
library_name: gguf
tags:
- gguf
- imatrix
- llama.cpp
- qwen3.8
- abliteration
- uncensored
Qwen3.8-27B-abliterated GGUF
imatrix GGUF quants of rajaykumar12959/qwen3.8-27b-abliterated, a refusal-ablated Qwen3.8-27B, sized to run on a 16 GB GPU with llama.cpp. Each quant was evaluated against the BF16 original. The scripts, raw outputs and full report are at github.com/vedevpatel/qwen3.8-quantized.
This model has had its refusal behaviour removed. It will answer requests the original Qwen3.8 declines. You are responsible for how you use it. See the source model card for how the ablation was done and what it costs.
Files
| File | Size | MMLU-Pro (Δ vs BF16) | Mean KLD vs BF16 | Pick it if |
|---|---|---|---|---|
Qwen3.8-27B-abliterated-IQ3_M.gguf |
12.6 GB | 79.00% (−0.60 pp) | 0.0642 | You have 16 GB of VRAM and want room for long context |
Qwen3.8-27B-abliterated-IQ4_XS.gguf |
15.1 GB | 79.64% (+0.05 pp) | 0.0195 | You have 20 GB or more of VRAM (it doesn't fit fully on 16 GB) |
Qwen3.8-27B-abliterated-Q4_K_M.gguf |
16.5 GB | 78.83% (−0.76 pp) | 0.0167 | You have 24 GB, or accept partial CPU offload |
imatrix.gguf |
27 MB | You want to make your own quants |
BF16 scores 79.60% on the same questions. No quant differs significantly from BF16 on MMLU-Pro (all 95% intervals include zero). KL divergence on WikiText-2 is the more sensitive measure: IQ3_M drifts 3.3–3.8× more than the 4-bit quants, so prefer a 4-bit file for open-ended generation if it fits.
MMLU-Pro here is 4,200 questions (300 per category), 5-shot chain of thought, greedy, thinking off. These absolute scores aren't comparable to the 61.8% on the source model card, which used a different setup; only the comparisons between files are like-for-like. Full report.
Speed
IQ3_M on an RTX 4070 Ti Super 16 GB, llama-bench -ngl 99, default f16 KV cache:
| Context already filled | Prompt processing (tok/s) | Generation (tok/s) |
|---|---|---|
| 0 | 1,772 | 39.9 |
| 16,384 | 1,478 | 37.2 |
| 32,768 | 1,152 | 34.9 |
| 49,152 | 40 | 19.5 |
With the default cache, up to about 32K tokens of context fit in 16 GB. Beyond that the cache spills out of VRAM and speed collapses.
With an 8-bit KV cache (-fa on -ctk q8_0 -ctv q8_0), 64K of context fits at 29.9 tok/s generation (770 tok/s prompt processing) and speed at zero context is unchanged. The quality effect of the 8-bit cache wasn't measured.
On an A100 80 GB, generation runs at 47.6 tok/s (IQ3_M), 59.5 (IQ4_XS) and 48.1 (Q4_K_M).
Running it
llama-server -m Qwen3.8-27B-abliterated-IQ3_M.gguf -ngl 99 -c 32768 --jinja
Then open http://127.0.0.1:8080, or use the OpenAI-compatible API at /v1. For up to 64K of context on 16 GB, add -fa on -ctk q8_0 -ctv q8_0 and set -c 65536.
The chat template puts the model in thinking mode by default, so it reasons at length before every answer. For direct answers, start the server with --reasoning off, or send "chat_template_kwargs": {"enable_thinking": false} with a request. llama-cli/llama-completion don't accept template kwargs, so thinking stays on there.
Text input only: the source checkpoint has no vision encoder.
How these were made
- Source:
rajaykumar12959/qwen3.8-27b-abliterated@9f85b10, converted with llama.cpp @23f7aab. - Converted with
--no-mtp. The source config declares an MTP layer, but the checkpoint ships no MTP weights. tokenizer_config.jsonwas replaced with the one fromQwen/Qwen3.8-27B@1d4bf0f, because the source's copy can't be loaded by the converter. The tokenizer itself is identical to upstream.- The importance matrix was computed on a Q8_0 copy using bartowski's
calibration_datav3.
License
Apache-2.0, following the source model and Qwen3.8-27B.