license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
tags:
- exl3
- exllamav3
- qwen3.8
- abliterated
- uncensored
- mtp
- quantized
language: - en
- fr
- zh
pipeline_tag: text-generation
Qwen3.8-27B-Uncensored — EXL3 6.0 bpw
EXL3 quantization of
orcarouter/Qwen3.8-27B-Uncensored,
an abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.
Quantized from the BF16 source with
ExLlamaV3 1.4.6.
The MTP head is preserved and works. GGUF conversions of this model
family drop the mtp.* tensors, so they have no self-drafter. This build
keeps the head at 8 bpw, and TabbyAPI logs Using main model MTP component for drafting on load.
Files
| Size on disk | 22 GB (BF16 source: 52 GB; FP8: 29 GB) |
| Shards | 3 |
| Language model | 6.00 bpw |
lm_head |
8 bpw |
| MTP layers | 8 bpw |
| Vision tower | unquantized (16-bit) |
| Embeddings | unquantized (16-bit) |
Conversion
python convert.py \
-i Qwen3.8-27B-Uncensored \
-o Qwen3.8-27B-Uncensored-exl3-6bpw \
-w /tmp/exl3-work \
-b 6.0 -hb 8 -mb 8 -vb 16 \
-d 0,1,2,3 -v
Default bundled calibration corpus (weighted mix: wiki 50, C4 20, code 20,
random tokens 20, technical 10, multilingual 10, tiny 5), 250 rows x 2048
columns. Roughly 1 h on 4x RTX 4000 Ada.
Usage — TabbyAPI
model:
model_dir: /path/to/models
model_name: Qwen3.8-27B-Uncensored-exl3-6bpw
cache_size: 32768
cache_mode: "8,8"
tensor_parallel: true
draft_model:
draft_mode: mtp
Vision is present in the weights but TabbyAPI disables it unlessvision: true is set.
Benchmarks
Everything below was measured in one session on the same box, same model
weights, aligned settings. Not vendor estimates.
Hardware: 4x RTX 4000 Ada Generation (20 GiB, sm89), PCIe, no NVLink,
2x Xeon Gold 5318Y, 247 GB RAM, Ubuntu 26.04, driver 595.71.05.
Tool: NVIDIA AIPerf, streaming, ignore_eos, fixed seed 42, with warmup.
Both engines: MTP speculative decoding on, 8-bit KV cache.
EXL3 side: ExLlamaV3 1.4.6 + TabbyAPI, tensor-parallel, cache_mode: "8,8".
vLLM side: 0.28.0, the FP8 build of the same abliterated model
(orcarouter/Qwen3.8-27B-Uncensored-FP8), --kv-cache-dtype fp8_e4m3.
Short prompts, 4 concurrent (ISL 1000 / OSL 500 / conc. 4 / 20 req)
| Metric | vLLM 0.28.0 FP8 | EXL3 6.0 bpw |
|---|---|---|
| Request latency | 14,158 ms | 20,816 ms |
| Inter-token latency | 26.3 ms | 35.0 ms |
| Time to first token | 1,028 ms | 3,515 ms |
| Output throughput | 139 tok/s | 91 tok/s |
Long prompts, 2 concurrent (ISL 8000 / OSL 200 / conc. 2 / 12 req)
| Metric | vLLM 0.28.0 FP8 | EXL3 6.0 bpw |
|---|---|---|
| Time to first token | 5,508 ms | 9,481 ms |
| Active prefill | 1,733 tok/s | 1,277 tok/s |
| Inter-token latency | 43.9 ms | 40.6 ms |
| Request latency | 14,244 ms | 17,362 ms |
Single stream, short prompt (ExLlamaV3 chat.py, no server)
| EXL3 6.0 bpw | |
|---|---|
| Generation | 63 tok/s |
| MTP acceptance | 62.5% on code, 41.7% on prose |
For reference, vLLM serves this model at ~45 tok/s single-stream.
Settings that mattered
Two knobs changed the EXL3 numbers a lot on this hardware. Both are worth
checking before quoting any EXL3 benchmark:
| Setting | Effect |
|---|---|
tensor_parallel: true |
Prefill TTFT 17,644 ms → 9,577 ms (-46%). But single-stream generation dropped 63 → 34.6 tok/s. It helps compute-bound prefill, hurts memory-bound decode. |
cache_mode: "8,8" vs FP16 |
1–4% only. Needed for a fair comparison against vLLM's fp8 KV cache, but not a performance lever here. |
Honest reading
On this hardware, vLLM is faster in every served scenario — 47% lower
latency on short prompts, 72% lower TTFT on long ones. EXL3 wins on two
things: VRAM (22 GB vs 29 GB, so ~7 GB more available for KV cache) and
single-stream generation with short prompts (63 vs ~45 tok/s).
Caveat on the hardware: these are PCIe-connected cards with no NVLink, andSymmMemCommunicator is unavailable on sm89. Tensor-parallel results are
likely better on NVLink systems. Treat the EXL3 numbers as a floor, not a
ceiling.
Quality
Not benchmarked at 6.0 bpw. For reference on the same abliterated weights,
the FP8 build scores 88.0% on MMLU-Pro (business subset, 100 questions, 2
unparsed) vs 89.0% for the official Qwen3.8-27B-FP8 — a one-question gap,
within noise, so the abliteration itself costs nothing measurable there.
Whether 6.0 bpw EXL3 matches FP8 has not been verified.
Safety
Inherited from the base model: safety alignment has been substantially
removed via abliteration. It will comply with requests the original
Qwen3.8-27B refuses, and has no meaningful built-in guardrails. Released
for research and controlled experimentation. Add your own moderation layer
before any deployment. You are responsible for what you do with it.
License
Apache 2.0, inherited from Qwen/Qwen3.8-27B. Quantization does not change
the license obligations.
Credits
- Qwen for the base model
- orcarouter for the abliterated build
- turboderp for ExLlamaV3 and
the EXL3 format - theroyallab for TabbyAPI